2023-11-20 15:43:22,497 INFO [train_asr.py:1289] (2/4) Training started
2023-11-20 15:43:22,497 INFO [train_asr.py:1299] (2/4) Device: cuda:2
2023-11-20 15:43:22,503 INFO [train_asr.py:1311] (2/4) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 50, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'warm_step': 2000, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '2b2ac14b326d61d79d04e53fbd69b1ff6d630411', 'k2-git-date': 'Thu Aug 24 05:58:26 2023', 'lhotse-version': '1.16.0', 'torch-version': '2.0.1+cu117', 'torch-cuda-available': True, 'torch-cuda-version': '11.7', 'python-version': '3.1', 'icefall-git-branch': 'multi_KD', 'icefall-git-sha1': '16e77b48-dirty', 'icefall-git-date': 'Mon Nov 20 11:32:19 2023', 'icefall-path': '/star-xy/softwares/icefall_development/icefall_multi_KD', 'k2-path': '/star-xy/softwares/k2_development/k2/k2/python/k2/__init__.py', 'lhotse-path': '/star-xy/softwares/anaconda3/envs/multi_KD/lib/python3.10/site-packages/lhotse/__init__.py', 'hostname': 'de-74279-k2-train-6-0423201309-7c68fd68fb-qfn6b', 'IP address': '10.177.58.19'}, 'world_size': 4, 'master_port': 13490, 'tensorboard': True, 'num_epochs': 40, 'start_epoch': 15, 'start_batch': 0, 'exp_dir': PosixPath('multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0'), 'bpe_model': 'data/lang_bpe_500/bpe.model', 'base_lr': 0.045, 'lr_batches': 7500, 'lr_epochs': 3.5, 'ref_duration': 600, 'context_size': 2, 'prune_range': 5, 'lm_scale': 0.25, 'am_scale': 0.0, 'simple_loss_scale': 0.5, 'ctc_loss_scale': 0.2, 'audio_tagging_loss_scale': 1.0, 'seed': 42, 'print_diagnostics': False, 'inf_check': False, 'save_every_n': 4000, 'keep_last_k': 30, 'average_period': 200, 'use_fp16': True, 'do_finetune': False, 'init_modules': None, 'freeze_modules': None, 'finetune_ckpt': None, 'num_encoder_layers': '2,2,3,4,3,2', 'downsampling_factor': '1,2,4,8,4,2', 'feedforward_dim': '512,768,1024,1536,1024,768', 'num_heads': '4,4,4,8,4,4', 'encoder_dim': '192,256,384,512,384,256', 'query_head_dim': '32', 'value_head_dim': '12', 'pos_head_dim': '4', 'pos_dim': 48, 'encoder_unmasked_dim': '192,192,256,256,256,192', 'cnn_module_kernel': '31,31,15,15,15,31', 'decoder_dim': 512, 'joiner_dim': 512, 'causal': False, 'chunk_size': '16,32,64,-1', 'left_context_frames': '64,128,256,-1', 'use_transducer': True, 'use_ctc': False, 'do_audio_tagging': True, 'use_encoder_projection': False, 'encoder_projection_dim': -1, 'freeze_encoder': False, 'freezing_encoder_layer_index': '-1', 'freeze_encoder_steps': -1, 'encoder_lr_scale': 1.0, 'full_libri': True, 'mini_libri': False, 'use_vox2': False, 'use_libriheavy': False, 'libriheavy_subset': 'small', 'use_audioset': True, 'audioset_subset': 'unbalanced', 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 1000, 'bucketing_sampler': False, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'enable_audioset': False, 'use_musan_separately': False, 'input_strategy': 'PrecomputedFeatures', 'drop_features': False, 'return_audio': False, 'use_beats': True, 'use_ecapa': True, 'use_whisper': True, 'whisper_mvq': False, 'beats_ckpt': 'data/models/BEATs/BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt', 'whisper_version': 'small.en', 'blank_id': 0, 'vocab_size': 500}
2023-11-20 15:43:22,503 INFO [train_asr.py:1320] (2/4) About to create model
2023-11-20 15:43:23,552 INFO [train_asr.py:1324] (2/4) Number of model parameters: 65819362
2023-11-20 15:43:23,553 INFO [checkpoint.py:112] (2/4) Loading checkpoint from multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-14.pt
2023-11-20 15:43:27,120 INFO [train_asr.py:1352] (2/4) Setting the lr scale of parameters in encoder and encoder_embed to 1.0
2023-11-20 15:43:31,933 INFO [train_asr.py:1361] (2/4) Using DDP
2023-11-20 15:43:32,389 INFO [train_asr.py:1384] (2/4) Loading optimizer state dict
2023-11-20 15:43:33,458 INFO [train_asr.py:1392] (2/4) Loading scheduler state dict
2023-11-20 15:43:33,462 INFO [train_asr.py:1414] (2/4) Getting audioset cuts
2023-11-20 15:43:33,462 INFO [kd_datamodule.py:796] (2/4) About to get the audioset cuts.
2023-11-20 15:43:33,465 INFO [train_asr.py:1420] (2/4) Using mux to combine Librispeech with audioset
2023-11-20 15:43:33,465 INFO [train_asr.py:1430] (2/4) CutSet(len=2748469) [underlying data type: <class 'lhotse.lazy.LazyIteratorMultiplexer'>]
2023-11-20 15:43:48,911 INFO [kd_datamodule.py:396] (2/4) Enable MUSAN
2023-11-20 15:43:48,911 INFO [kd_datamodule.py:397] (2/4) About to get Musan cuts
2023-11-20 15:43:52,481 INFO [kd_datamodule.py:427] (2/4) Enable SpecAugment
2023-11-20 15:43:52,481 INFO [kd_datamodule.py:428] (2/4) Time warp factor: 80
2023-11-20 15:43:52,482 INFO [kd_datamodule.py:438] (2/4) Num frame mask: 10
2023-11-20 15:43:52,482 INFO [kd_datamodule.py:451] (2/4) About to create train dataset
2023-11-20 15:43:52,483 INFO [kd_datamodule.py:487] (2/4) Using SimpleCutSampler
2023-11-20 15:43:52,484 INFO [kd_datamodule.py:495] (2/4) About to create train dataloader
2023-11-20 15:43:52,487 INFO [kd_datamodule.py:814] (2/4) About to get the audioset eval cuts.
2023-11-20 15:43:52,488 INFO [train_asr.py:1494] (2/4) CutSet(len=20681) [underlying data type: <class 'lhotse.lazy.LazyMapper'>]
2023-11-20 15:43:52,579 INFO [kd_datamodule.py:529] (2/4) About to create dev dataset
2023-11-20 15:43:53,362 INFO [kd_datamodule.py:550] (2/4) About to create dev dataloader
2023-11-20 15:43:53,362 INFO [train_asr.py:1508] (2/4) Loading grad scaler state dict
2023-11-20 15:44:29,644 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.63 vs. limit=15.0
2023-11-20 15:44:29,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.68 vs. limit=22.5
2023-11-20 15:44:30,186 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 0, loss[loss=0.08922, simple_loss=0.09392, pruned_loss=0.01872, audio_tagging_loss=0.02353, over 16005.00 frames. ], tot_loss[loss=0.08922, simple_loss=0.09392, pruned_loss=0.01872, audio_tagging_loss=0.02353, over 16005.00 frames. ], batch size: 60, lr: 4.68e-03, grad_scale: 32.0
2023-11-20 15:44:30,188 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 15:45:06,719 INFO [train_asr.py:1253] (2/4) Epoch 15, validation: loss=0.06153, simple_loss=0.05347, pruned_loss=0.005654, audio_tagging_loss=0.02914, over 4681554.00 frames. 
2023-11-20 15:45:06,720 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 15:45:11,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1122200.0, ans=0.0
2023-11-20 15:45:12,401 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.292e+01 9.006e+01 9.945e+01 1.226e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-20 15:45:15,784 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.48 vs. limit=12.0
2023-11-20 15:45:25,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.76 vs. limit=15.0
2023-11-20 15:45:32,150 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168350
2023-11-20 15:45:32,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1122266.6666666667, ans=0.125
2023-11-20 15:45:39,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1122333.3333333333, ans=0.09899494936611666
2023-11-20 15:45:44,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1122333.3333333333, ans=0.125
2023-11-20 15:45:59,122 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.67 vs. limit=22.5
2023-11-20 15:46:12,513 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 50, loss[loss=0.07646, simple_loss=0.0895, pruned_loss=0.01227, audio_tagging_loss=0.01943, over 15095.00 frames. ], tot_loss[loss=0.08743, simple_loss=0.09849, pruned_loss=0.01884, audio_tagging_loss=0.01935, over 681732.90 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 32.0
2023-11-20 15:46:36,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1122600.0, ans=0.125
2023-11-20 15:46:38,025 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168400
2023-11-20 15:46:50,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1122666.6666666667, ans=0.0
2023-11-20 15:47:00,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1122733.3333333333, ans=0.0
2023-11-20 15:47:11,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1122800.0, ans=0.0
2023-11-20 15:47:14,337 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.86 vs. limit=15.0
2023-11-20 15:47:20,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1122866.6666666667, ans=0.035
2023-11-20 15:47:21,457 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 100, loss[loss=0.0621, simple_loss=0.07133, pruned_loss=0.008747, audio_tagging_loss=0.01769, over 15750.00 frames. ], tot_loss[loss=0.08676, simple_loss=0.0982, pruned_loss=0.0189, audio_tagging_loss=0.01876, over 1198527.59 frames. ], batch size: 62, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:47:27,306 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.21 vs. limit=15.0
2023-11-20 15:47:27,773 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.479e+01 8.856e+01 9.417e+01 9.999e+01 1.434e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-20 15:47:44,488 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168450
2023-11-20 15:48:07,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1123066.6666666667, ans=0.125
2023-11-20 15:48:14,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1123133.3333333333, ans=0.1
2023-11-20 15:48:26,578 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=12.0
2023-11-20 15:48:27,322 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 150, loss[loss=0.08967, simple_loss=0.1141, pruned_loss=0.02122, audio_tagging_loss=0.01141, over 15322.00 frames. ], tot_loss[loss=0.08581, simple_loss=0.1007, pruned_loss=0.0191, audio_tagging_loss=0.01635, over 1611629.59 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:48:32,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1123200.0, ans=0.2
2023-11-20 15:48:36,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1123200.0, ans=0.1
2023-11-20 15:48:40,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1123266.6666666667, ans=0.125
2023-11-20 15:48:44,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.13 vs. limit=15.0
2023-11-20 15:48:46,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1123266.6666666667, ans=0.125
2023-11-20 15:48:50,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168500
2023-11-20 15:49:04,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1123333.3333333333, ans=0.025
2023-11-20 15:49:14,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1123400.0, ans=0.0
2023-11-20 15:49:21,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1123466.6666666667, ans=0.2
2023-11-20 15:49:27,805 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.36 vs. limit=15.0
2023-11-20 15:49:28,817 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.96 vs. limit=15.0
2023-11-20 15:49:31,867 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 200, loss[loss=0.06673, simple_loss=0.0801, pruned_loss=0.01416, audio_tagging_loss=0.01252, over 15065.00 frames. ], tot_loss[loss=0.08272, simple_loss=0.09958, pruned_loss=0.01846, audio_tagging_loss=0.01447, over 1932252.67 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:49:32,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1123533.3333333333, ans=0.1
2023-11-20 15:49:32,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.62 vs. limit=15.0
2023-11-20 15:49:38,092 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.283e+01 8.917e+01 9.795e+01 1.407e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-20 15:49:56,463 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168550
2023-11-20 15:50:12,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1123733.3333333333, ans=0.125
2023-11-20 15:50:38,418 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 250, loss[loss=0.07873, simple_loss=0.104, pruned_loss=0.01695, audio_tagging_loss=0.009802, over 15005.00 frames. ], tot_loss[loss=0.08201, simple_loss=0.09975, pruned_loss=0.01894, audio_tagging_loss=0.0132, over 2179970.17 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:50:44,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1123866.6666666667, ans=0.1
2023-11-20 15:50:48,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1123866.6666666667, ans=0.125
2023-11-20 15:51:00,749 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168600
2023-11-20 15:51:18,354 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:51:20,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1124066.6666666667, ans=0.125
2023-11-20 15:51:28,958 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:51:34,184 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-20 15:51:43,530 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 300, loss[loss=0.05693, simple_loss=0.07116, pruned_loss=0.01023, audio_tagging_loss=0.01112, over 15578.00 frames. ], tot_loss[loss=0.08163, simple_loss=0.1004, pruned_loss=0.01919, audio_tagging_loss=0.01224, over 2369243.82 frames. ], batch size: 59, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:51:49,070 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.94 vs. limit=15.0
2023-11-20 15:51:49,689 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.496e+01 9.198e+01 1.000e+02 1.340e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-20 15:51:59,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1124266.6666666667, ans=0.2
2023-11-20 15:52:05,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168650
2023-11-20 15:52:30,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1124400.0, ans=0.07
2023-11-20 15:52:47,378 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 350, loss[loss=0.09309, simple_loss=0.1148, pruned_loss=0.0263, audio_tagging_loss=0.009398, over 14572.00 frames. ], tot_loss[loss=0.08161, simple_loss=0.1014, pruned_loss=0.01932, audio_tagging_loss=0.01159, over 2518824.88 frames. ], batch size: 58, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:53:00,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1124600.0, ans=0.1
2023-11-20 15:53:07,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1124600.0, ans=0.125
2023-11-20 15:53:09,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1124600.0, ans=0.125
2023-11-20 15:53:11,744 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168700
2023-11-20 15:53:12,495 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.08 vs. limit=15.0
2023-11-20 15:53:18,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1124666.6666666667, ans=0.0
2023-11-20 15:53:21,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1124666.6666666667, ans=0.0
2023-11-20 15:53:43,413 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.49 vs. limit=10.0
2023-11-20 15:53:50,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.11 vs. limit=15.0
2023-11-20 15:53:52,668 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 400, loss[loss=0.06665, simple_loss=0.08997, pruned_loss=0.01141, audio_tagging_loss=0.01026, over 15314.00 frames. ], tot_loss[loss=0.08066, simple_loss=0.1007, pruned_loss=0.01919, audio_tagging_loss=0.01112, over 2641794.94 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 32.0
2023-11-20 15:53:52,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1124866.6666666667, ans=0.2
2023-11-20 15:53:55,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1124866.6666666667, ans=22.5
2023-11-20 15:53:59,453 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.311e+01 9.076e+01 1.027e+02 1.296e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-20 15:54:03,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1124866.6666666667, ans=0.125
2023-11-20 15:54:03,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.48 vs. limit=22.5
2023-11-20 15:54:15,658 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168750
2023-11-20 15:54:25,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1125000.0, ans=0.125
2023-11-20 15:54:56,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1125200.0, ans=0.1
2023-11-20 15:54:57,577 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 450, loss[loss=0.07696, simple_loss=0.09904, pruned_loss=0.01859, audio_tagging_loss=0.008846, over 16150.00 frames. ], tot_loss[loss=0.08044, simple_loss=0.1007, pruned_loss=0.01924, audio_tagging_loss=0.01083, over 2734537.68 frames. ], batch size: 59, lr: 4.67e-03, grad_scale: 32.0
2023-11-20 15:54:58,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1125200.0, ans=0.04949747468305833
2023-11-20 15:55:07,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1125200.0, ans=0.125
2023-11-20 15:55:19,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168800
2023-11-20 15:55:55,145 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.42 vs. limit=15.0
2023-11-20 15:56:03,099 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 500, loss[loss=0.07167, simple_loss=0.09367, pruned_loss=0.0144, audio_tagging_loss=0.01044, over 15086.00 frames. ], tot_loss[loss=0.07961, simple_loss=0.1, pruned_loss=0.01903, audio_tagging_loss=0.01057, over 2793123.70 frames. ], batch size: 55, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:56:10,664 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.084e+01 8.651e+01 9.631e+01 1.244e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 15:56:17,417 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:56:27,745 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168850
2023-11-20 15:56:32,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1125666.6666666667, ans=0.1
2023-11-20 15:56:37,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1125666.6666666667, ans=0.125
2023-11-20 15:56:41,475 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.23 vs. limit=10.0
2023-11-20 15:56:43,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1125733.3333333333, ans=0.125
2023-11-20 15:56:43,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1125733.3333333333, ans=0.05
2023-11-20 15:56:59,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1125800.0, ans=10.0
2023-11-20 15:57:07,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1125800.0, ans=0.2
2023-11-20 15:57:09,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1125866.6666666667, ans=0.07
2023-11-20 15:57:10,346 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 550, loss[loss=0.07489, simple_loss=0.08574, pruned_loss=0.02169, audio_tagging_loss=0.01033, over 15307.00 frames. ], tot_loss[loss=0.07895, simple_loss=0.09897, pruned_loss=0.0189, audio_tagging_loss=0.01057, over 2845392.14 frames. ], batch size: 60, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:57:28,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1125933.3333333333, ans=0.125
2023-11-20 15:57:30,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1125933.3333333333, ans=0.0
2023-11-20 15:57:34,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168900
2023-11-20 15:57:52,006 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.70 vs. limit=15.0
2023-11-20 15:58:15,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0
2023-11-20 15:58:16,243 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 600, loss[loss=0.07821, simple_loss=0.1004, pruned_loss=0.01894, audio_tagging_loss=0.009054, over 15248.00 frames. ], tot_loss[loss=0.07882, simple_loss=0.09918, pruned_loss=0.01885, audio_tagging_loss=0.01038, over 2894867.35 frames. ], batch size: 55, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:58:23,541 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 7.985e+01 8.751e+01 9.752e+01 1.592e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 15:58:24,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1126200.0, ans=0.05
2023-11-20 15:58:30,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1126266.6666666667, ans=0.0
2023-11-20 15:58:36,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1126266.6666666667, ans=0.5
2023-11-20 15:58:38,506 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 168950
2023-11-20 15:58:52,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.85 vs. limit=15.0
2023-11-20 15:58:56,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1126400.0, ans=0.2
2023-11-20 15:59:00,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1126400.0, ans=0.05
2023-11-20 15:59:01,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1126400.0, ans=0.2
2023-11-20 15:59:20,619 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 650, loss[loss=0.08663, simple_loss=0.111, pruned_loss=0.02046, audio_tagging_loss=0.01067, over 15234.00 frames. ], tot_loss[loss=0.07836, simple_loss=0.09875, pruned_loss=0.01869, audio_tagging_loss=0.0103, over 2927330.32 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:59:21,209 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.41 vs. limit=12.0
2023-11-20 15:59:29,028 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.90 vs. limit=15.0
2023-11-20 15:59:44,702 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169000
2023-11-20 15:59:48,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1126666.6666666667, ans=0.2
2023-11-20 15:59:54,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1126666.6666666667, ans=0.2
2023-11-20 15:59:54,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1126666.6666666667, ans=0.0
2023-11-20 16:00:02,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1126733.3333333333, ans=0.125
2023-11-20 16:00:08,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1126733.3333333333, ans=0.0
2023-11-20 16:00:10,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.48 vs. limit=15.0
2023-11-20 16:00:22,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.97 vs. limit=15.0
2023-11-20 16:00:26,791 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 700, loss[loss=0.06249, simple_loss=0.07926, pruned_loss=0.01459, audio_tagging_loss=0.00827, over 14938.00 frames. ], tot_loss[loss=0.07906, simple_loss=0.1001, pruned_loss=0.0189, audio_tagging_loss=0.01012, over 2958643.84 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 16:00:31,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1126866.6666666667, ans=0.0
2023-11-20 16:00:34,823 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.436e+01 8.027e+01 8.656e+01 9.309e+01 1.974e+02, threshold=1.731e+02, percent-clipped=1.0
2023-11-20 16:00:42,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1126933.3333333333, ans=0.0
2023-11-20 16:00:50,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169050
2023-11-20 16:00:59,719 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.24 vs. limit=15.0
2023-11-20 16:01:01,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1127000.0, ans=0.125
2023-11-20 16:01:14,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1127066.6666666667, ans=0.2
2023-11-20 16:01:18,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1127133.3333333333, ans=0.0
2023-11-20 16:01:23,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1127133.3333333333, ans=0.5
2023-11-20 16:01:31,580 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 750, loss[loss=0.06715, simple_loss=0.08245, pruned_loss=0.01687, audio_tagging_loss=0.009064, over 13799.00 frames. ], tot_loss[loss=0.079, simple_loss=0.0998, pruned_loss=0.01892, audio_tagging_loss=0.01018, over 2973528.99 frames. ], batch size: 52, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 16:01:37,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1127200.0, ans=0.125
2023-11-20 16:01:54,448 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169100
2023-11-20 16:01:58,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1127333.3333333333, ans=0.125
2023-11-20 16:02:00,919 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=4.506e-01
2023-11-20 16:02:14,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1127400.0, ans=0.125
2023-11-20 16:02:15,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1127400.0, ans=0.0
2023-11-20 16:02:32,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1127466.6666666667, ans=0.0
2023-11-20 16:02:36,485 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 800, loss[loss=0.09083, simple_loss=0.122, pruned_loss=0.02152, audio_tagging_loss=0.00831, over 15570.00 frames. ], tot_loss[loss=0.07927, simple_loss=0.1001, pruned_loss=0.01898, audio_tagging_loss=0.01026, over 2997099.15 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:02:43,907 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.081e+01 8.651e+01 9.218e+01 1.161e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 16:02:55,480 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:03:00,015 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:03:00,984 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169150
2023-11-20 16:03:13,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1127666.6666666667, ans=0.2
2023-11-20 16:03:31,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1127800.0, ans=0.95
2023-11-20 16:03:32,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.54 vs. limit=22.5
2023-11-20 16:03:41,745 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 850, loss[loss=0.08685, simple_loss=0.1066, pruned_loss=0.02119, audio_tagging_loss=0.01238, over 15441.00 frames. ], tot_loss[loss=0.07923, simple_loss=0.1, pruned_loss=0.01891, audio_tagging_loss=0.01031, over 3007751.55 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:03:45,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1127866.6666666667, ans=0.125
2023-11-20 16:03:56,733 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:04:05,118 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169200
2023-11-20 16:04:33,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1128133.3333333333, ans=0.07
2023-11-20 16:04:33,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1128133.3333333333, ans=0.09899494936611666
2023-11-20 16:04:48,006 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 900, loss[loss=0.09295, simple_loss=0.1149, pruned_loss=0.027, audio_tagging_loss=0.008484, over 15140.00 frames. ], tot_loss[loss=0.07974, simple_loss=0.1005, pruned_loss=0.01912, audio_tagging_loss=0.01036, over 3015072.63 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:04:55,433 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.050e+01 8.717e+01 9.449e+01 1.348e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 16:05:03,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1128266.6666666667, ans=0.0
2023-11-20 16:05:11,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169250
2023-11-20 16:05:49,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1128466.6666666667, ans=0.0
2023-11-20 16:05:52,999 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 950, loss[loss=0.0829, simple_loss=0.1061, pruned_loss=0.02, audio_tagging_loss=0.009833, over 15604.00 frames. ], tot_loss[loss=0.07906, simple_loss=0.1001, pruned_loss=0.01882, audio_tagging_loss=0.0102, over 3025250.91 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:05:53,779 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.10 vs. limit=10.0
2023-11-20 16:05:58,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1128533.3333333333, ans=0.0
2023-11-20 16:06:03,885 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=22.5
2023-11-20 16:06:17,456 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169300
2023-11-20 16:06:28,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1128666.6666666667, ans=0.125
2023-11-20 16:06:57,911 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1000, loss[loss=0.06899, simple_loss=0.0874, pruned_loss=0.01488, audio_tagging_loss=0.01041, over 15518.00 frames. ], tot_loss[loss=0.0792, simple_loss=0.1006, pruned_loss=0.01904, audio_tagging_loss=0.009842, over 3031449.99 frames. ], batch size: 58, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:07:05,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1128866.6666666667, ans=0.125
2023-11-20 16:07:06,526 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.366e+01 7.987e+01 8.758e+01 9.641e+01 1.276e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 16:07:11,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0
2023-11-20 16:07:22,245 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169350
2023-11-20 16:07:25,940 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:07:34,913 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:07:40,360 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.32 vs. limit=15.0
2023-11-20 16:07:45,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1129066.6666666667, ans=0.04949747468305833
2023-11-20 16:08:04,575 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1050, loss[loss=0.07036, simple_loss=0.08732, pruned_loss=0.01778, audio_tagging_loss=0.008918, over 14955.00 frames. ], tot_loss[loss=0.08041, simple_loss=0.1024, pruned_loss=0.01951, audio_tagging_loss=0.009697, over 3034410.62 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:08:08,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1129200.0, ans=0.125
2023-11-20 16:08:09,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1129200.0, ans=0.0
2023-11-20 16:08:17,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1129266.6666666667, ans=0.125
2023-11-20 16:08:19,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1129266.6666666667, ans=0.07
2023-11-20 16:08:27,021 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169400
2023-11-20 16:08:34,993 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.72 vs. limit=15.0
2023-11-20 16:08:38,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.21 vs. limit=10.0
2023-11-20 16:08:47,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.03 vs. limit=22.5
2023-11-20 16:09:00,160 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.53 vs. limit=6.0
2023-11-20 16:09:03,854 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-20 16:09:09,318 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1100, loss[loss=0.05992, simple_loss=0.07838, pruned_loss=0.01071, audio_tagging_loss=0.01003, over 15410.00 frames. ], tot_loss[loss=0.07913, simple_loss=0.1008, pruned_loss=0.01901, audio_tagging_loss=0.009739, over 3029723.94 frames. ], batch size: 61, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:09:11,868 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:09:14,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1129533.3333333333, ans=0.0
2023-11-20 16:09:15,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1129533.3333333333, ans=0.125
2023-11-20 16:09:16,779 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.137e+01 8.767e+01 9.510e+01 1.319e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-20 16:09:18,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1129533.3333333333, ans=0.125
2023-11-20 16:09:19,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1129533.3333333333, ans=0.1
2023-11-20 16:09:22,624 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.14 vs. limit=12.0
2023-11-20 16:09:32,981 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169450
2023-11-20 16:09:46,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1129666.6666666667, ans=0.2
2023-11-20 16:09:50,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1129733.3333333333, ans=0.125
2023-11-20 16:09:59,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.32 vs. limit=15.0
2023-11-20 16:10:00,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1129800.0, ans=0.0
2023-11-20 16:10:11,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1129800.0, ans=0.1
2023-11-20 16:10:13,840 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1150, loss[loss=0.06483, simple_loss=0.08543, pruned_loss=0.01155, audio_tagging_loss=0.01057, over 15377.00 frames. ], tot_loss[loss=0.07915, simple_loss=0.1009, pruned_loss=0.01909, audio_tagging_loss=0.009609, over 3032844.44 frames. ], batch size: 56, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:10:21,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1129866.6666666667, ans=0.04949747468305833
2023-11-20 16:10:38,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169500
2023-11-20 16:10:57,088 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.98 vs. limit=15.0
2023-11-20 16:11:05,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1130133.3333333333, ans=0.125
2023-11-20 16:11:05,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1130133.3333333333, ans=0.0
2023-11-20 16:11:11,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1130133.3333333333, ans=0.125
2023-11-20 16:11:14,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1130133.3333333333, ans=0.0
2023-11-20 16:11:21,447 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1200, loss[loss=0.07951, simple_loss=0.0942, pruned_loss=0.02005, audio_tagging_loss=0.01235, over 14021.00 frames. ], tot_loss[loss=0.07879, simple_loss=0.1004, pruned_loss=0.01904, audio_tagging_loss=0.009567, over 3030993.66 frames. ], batch size: 53, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:11:30,085 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.141e+01 8.800e+01 9.449e+01 1.223e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-20 16:11:43,984 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169550
2023-11-20 16:12:26,136 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1250, loss[loss=0.09668, simple_loss=0.1223, pruned_loss=0.02765, audio_tagging_loss=0.007858, over 15390.00 frames. ], tot_loss[loss=0.07966, simple_loss=0.1013, pruned_loss=0.01945, audio_tagging_loss=0.009582, over 3029795.00 frames. ], batch size: 56, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:12:46,359 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:12:49,051 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169600
2023-11-20 16:12:49,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1130600.0, ans=0.2
2023-11-20 16:12:59,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1130666.6666666667, ans=0.0
2023-11-20 16:13:21,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1130800.0, ans=0.1
2023-11-20 16:13:30,640 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1300, loss[loss=0.08725, simple_loss=0.1023, pruned_loss=0.0247, audio_tagging_loss=0.0114, over 14901.00 frames. ], tot_loss[loss=0.07947, simple_loss=0.101, pruned_loss=0.01933, audio_tagging_loss=0.009623, over 3030313.84 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:13:41,676 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.174e+01 8.827e+01 9.605e+01 1.804e+02, threshold=1.765e+02, percent-clipped=1.0
2023-11-20 16:13:42,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1130866.6666666667, ans=0.0
2023-11-20 16:13:48,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1130933.3333333333, ans=0.2
2023-11-20 16:13:48,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1130933.3333333333, ans=0.125
2023-11-20 16:13:55,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169650
2023-11-20 16:13:57,031 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:14:21,179 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.53 vs. limit=22.5
2023-11-20 16:14:26,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.69 vs. limit=15.0
2023-11-20 16:14:28,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1131133.3333333333, ans=0.05
2023-11-20 16:14:30,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1131133.3333333333, ans=0.2
2023-11-20 16:14:36,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1131200.0, ans=0.0
2023-11-20 16:14:37,369 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1350, loss[loss=0.07327, simple_loss=0.08907, pruned_loss=0.01807, audio_tagging_loss=0.01066, over 15612.00 frames. ], tot_loss[loss=0.07929, simple_loss=0.1008, pruned_loss=0.01924, audio_tagging_loss=0.009644, over 3037144.37 frames. ], batch size: 58, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:15:00,489 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169700
2023-11-20 16:15:17,026 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.56 vs. limit=15.0
2023-11-20 16:15:18,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1131400.0, ans=0.125
2023-11-20 16:15:23,963 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:15:37,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1131466.6666666667, ans=0.0
2023-11-20 16:15:42,461 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1400, loss[loss=0.06939, simple_loss=0.08879, pruned_loss=0.01525, audio_tagging_loss=0.009751, over 15330.00 frames. ], tot_loss[loss=0.07978, simple_loss=0.1017, pruned_loss=0.01934, audio_tagging_loss=0.009588, over 3040876.87 frames. ], batch size: 61, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:15:42,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1131533.3333333333, ans=0.0
2023-11-20 16:15:47,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1131533.3333333333, ans=0.0
2023-11-20 16:15:52,435 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 7.986e+01 8.672e+01 9.769e+01 1.289e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-20 16:15:53,296 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.62 vs. limit=15.0
2023-11-20 16:16:05,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169750
2023-11-20 16:16:16,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1131666.6666666667, ans=0.125
2023-11-20 16:16:23,228 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.84 vs. limit=10.0
2023-11-20 16:16:27,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1131733.3333333333, ans=0.125
2023-11-20 16:16:29,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=15.0
2023-11-20 16:16:42,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1131800.0, ans=0.1
2023-11-20 16:16:47,114 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1450, loss[loss=0.06616, simple_loss=0.07804, pruned_loss=0.01419, audio_tagging_loss=0.01294, over 15412.00 frames. ], tot_loss[loss=0.07991, simple_loss=0.1017, pruned_loss=0.01931, audio_tagging_loss=0.009766, over 3036447.57 frames. ], batch size: 60, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:17:11,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169800
2023-11-20 16:17:21,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1132000.0, ans=0.1
2023-11-20 16:17:47,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1132133.3333333333, ans=0.0
2023-11-20 16:17:52,585 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1500, loss[loss=0.07811, simple_loss=0.0901, pruned_loss=0.02165, audio_tagging_loss=0.0114, over 14422.00 frames. ], tot_loss[loss=0.07956, simple_loss=0.1007, pruned_loss=0.01932, audio_tagging_loss=0.009915, over 3036870.27 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 8.0
2023-11-20 16:18:00,714 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.32 vs. limit=12.0
2023-11-20 16:18:04,950 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.236e+01 9.090e+01 9.607e+01 1.235e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-20 16:18:16,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169850
2023-11-20 16:18:19,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.84 vs. limit=15.0
2023-11-20 16:18:41,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1132400.0, ans=0.125
2023-11-20 16:18:49,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1132466.6666666667, ans=0.0
2023-11-20 16:18:58,740 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1550, loss[loss=0.07174, simple_loss=0.08453, pruned_loss=0.01645, audio_tagging_loss=0.01303, over 15566.00 frames. ], tot_loss[loss=0.07883, simple_loss=0.09949, pruned_loss=0.01899, audio_tagging_loss=0.01009, over 3039282.21 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 8.0
2023-11-20 16:19:01,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1132533.3333333333, ans=0.1
2023-11-20 16:19:01,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1132533.3333333333, ans=0.125
2023-11-20 16:19:14,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1132600.0, ans=0.125
2023-11-20 16:19:21,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169900
2023-11-20 16:19:23,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1132666.6666666667, ans=0.125
2023-11-20 16:19:25,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1132666.6666666667, ans=10.0
2023-11-20 16:19:42,266 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.17 vs. limit=10.0
2023-11-20 16:20:03,522 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1600, loss[loss=0.08096, simple_loss=0.1113, pruned_loss=0.01797, audio_tagging_loss=0.007341, over 16731.00 frames. ], tot_loss[loss=0.07845, simple_loss=0.09891, pruned_loss=0.01886, audio_tagging_loss=0.01014, over 3035974.05 frames. ], batch size: 61, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:20:14,714 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.371e+01 8.069e+01 8.651e+01 9.390e+01 1.565e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 16:20:26,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1132933.3333333333, ans=0.07
2023-11-20 16:20:27,743 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 169950
2023-11-20 16:20:31,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1133000.0, ans=0.0
2023-11-20 16:20:36,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.12 vs. limit=15.0
2023-11-20 16:20:48,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1133066.6666666667, ans=0.025
2023-11-20 16:20:49,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1133066.6666666667, ans=0.5
2023-11-20 16:20:51,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1133066.6666666667, ans=0.1
2023-11-20 16:20:52,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1133066.6666666667, ans=0.07
2023-11-20 16:20:56,396 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:21:10,160 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1650, loss[loss=0.07812, simple_loss=0.09809, pruned_loss=0.02129, audio_tagging_loss=0.007785, over 14832.00 frames. ], tot_loss[loss=0.07904, simple_loss=0.09984, pruned_loss=0.01909, audio_tagging_loss=0.01003, over 3036000.23 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:21:33,769 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170000
2023-11-20 16:21:38,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1133333.3333333333, ans=0.0
2023-11-20 16:21:39,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1133333.3333333333, ans=0.0
2023-11-20 16:22:06,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1133466.6666666667, ans=0.0
2023-11-20 16:22:07,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1133466.6666666667, ans=0.125
2023-11-20 16:22:15,868 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1700, loss[loss=0.09402, simple_loss=0.1195, pruned_loss=0.02364, audio_tagging_loss=0.01062, over 14109.00 frames. ], tot_loss[loss=0.07915, simple_loss=0.1001, pruned_loss=0.01901, audio_tagging_loss=0.01008, over 3040108.04 frames. ], batch size: 53, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:22:27,589 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.019e+01 8.780e+01 9.587e+01 1.403e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-20 16:22:39,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170050
2023-11-20 16:22:39,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1133600.0, ans=0.125
2023-11-20 16:22:47,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1133666.6666666667, ans=0.125
2023-11-20 16:23:10,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1133800.0, ans=0.2
2023-11-20 16:23:15,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.10 vs. limit=22.5
2023-11-20 16:23:21,143 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1750, loss[loss=0.06984, simple_loss=0.07871, pruned_loss=0.01715, audio_tagging_loss=0.01333, over 15179.00 frames. ], tot_loss[loss=0.07876, simple_loss=0.09933, pruned_loss=0.01894, audio_tagging_loss=0.01015, over 3033265.48 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:23:44,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170100
2023-11-20 16:23:50,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1134000.0, ans=0.1
2023-11-20 16:23:52,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1134000.0, ans=0.125
2023-11-20 16:23:55,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1134000.0, ans=0.0
2023-11-20 16:23:55,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1134000.0, ans=0.125
2023-11-20 16:24:01,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1134066.6666666667, ans=0.125
2023-11-20 16:24:08,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1134066.6666666667, ans=0.0
2023-11-20 16:24:11,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=1134066.6666666667, ans=0.02
2023-11-20 16:24:15,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1134133.3333333333, ans=0.1
2023-11-20 16:24:17,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1134133.3333333333, ans=0.125
2023-11-20 16:24:26,747 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1800, loss[loss=0.07268, simple_loss=0.09578, pruned_loss=0.01713, audio_tagging_loss=0.007661, over 14557.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09893, pruned_loss=0.01882, audio_tagging_loss=0.009997, over 3039930.33 frames. ], batch size: 53, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:24:39,070 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.001e+01 8.710e+01 9.346e+01 1.351e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-20 16:24:41,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1134266.6666666667, ans=0.125
2023-11-20 16:24:42,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1134266.6666666667, ans=0.0
2023-11-20 16:24:43,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1134266.6666666667, ans=0.0
2023-11-20 16:24:51,045 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170150
2023-11-20 16:24:57,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.77 vs. limit=12.0
2023-11-20 16:25:08,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1134400.0, ans=0.125
2023-11-20 16:25:23,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.89 vs. limit=15.0
2023-11-20 16:25:27,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1134466.6666666667, ans=0.0
2023-11-20 16:25:32,388 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1850, loss[loss=0.1048, simple_loss=0.1356, pruned_loss=0.02715, audio_tagging_loss=0.009845, over 16106.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.09851, pruned_loss=0.01858, audio_tagging_loss=0.009967, over 3044905.50 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:25:38,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1134533.3333333333, ans=0.125
2023-11-20 16:25:54,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1134600.0, ans=0.125
2023-11-20 16:25:55,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170200
2023-11-20 16:26:10,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1134666.6666666667, ans=0.125
2023-11-20 16:26:38,602 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1900, loss[loss=0.08239, simple_loss=0.115, pruned_loss=0.01793, audio_tagging_loss=0.006967, over 14672.00 frames. ], tot_loss[loss=0.07838, simple_loss=0.09953, pruned_loss=0.01884, audio_tagging_loss=0.009775, over 3053636.93 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:26:38,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1134866.6666666667, ans=0.2
2023-11-20 16:26:49,718 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.240e+01 7.901e+01 8.698e+01 9.524e+01 1.257e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-20 16:26:56,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1134933.3333333333, ans=0.125
2023-11-20 16:27:02,078 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170250
2023-11-20 16:27:16,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1135066.6666666667, ans=15.0
2023-11-20 16:27:37,230 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:27:41,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1135133.3333333333, ans=0.2
2023-11-20 16:27:43,332 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 1950, loss[loss=0.05326, simple_loss=0.05572, pruned_loss=0.01157, audio_tagging_loss=0.01383, over 14078.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09864, pruned_loss=0.01861, audio_tagging_loss=0.009792, over 3049938.22 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:27:43,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1135200.0, ans=0.125
2023-11-20 16:27:54,005 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.35 vs. limit=22.5
2023-11-20 16:28:07,837 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170300
2023-11-20 16:28:12,208 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=12.0
2023-11-20 16:28:12,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1135333.3333333333, ans=0.07
2023-11-20 16:28:26,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1135400.0, ans=0.125
2023-11-20 16:28:48,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1135533.3333333333, ans=0.015
2023-11-20 16:28:49,866 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2000, loss[loss=0.08972, simple_loss=0.1253, pruned_loss=0.01809, audio_tagging_loss=0.008982, over 14320.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.0991, pruned_loss=0.01875, audio_tagging_loss=0.009852, over 3042280.36 frames. ], batch size: 52, lr: 4.65e-03, grad_scale: 32.0
2023-11-20 16:29:00,827 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 7.997e+01 8.751e+01 9.649e+01 1.208e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 16:29:12,902 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170350
2023-11-20 16:29:20,889 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.88 vs. limit=15.0
2023-11-20 16:29:38,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1135733.3333333333, ans=0.125
2023-11-20 16:29:40,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1135800.0, ans=0.0
2023-11-20 16:29:54,345 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2050, loss[loss=0.08151, simple_loss=0.1027, pruned_loss=0.02001, audio_tagging_loss=0.01017, over 15060.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09861, pruned_loss=0.01862, audio_tagging_loss=0.00989, over 3044805.17 frames. ], batch size: 57, lr: 4.65e-03, grad_scale: 32.0
2023-11-20 16:30:01,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1135866.6666666667, ans=0.125
2023-11-20 16:30:18,779 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170400
2023-11-20 16:30:56,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.09 vs. limit=6.0
2023-11-20 16:31:00,392 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2100, loss[loss=0.109, simple_loss=0.1413, pruned_loss=0.03004, audio_tagging_loss=0.008331, over 14683.00 frames. ], tot_loss[loss=0.07825, simple_loss=0.09914, pruned_loss=0.0188, audio_tagging_loss=0.009883, over 3038505.54 frames. ], batch size: 53, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:31:14,451 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.060e+01 8.301e+01 9.019e+01 9.806e+01 1.324e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-20 16:31:25,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170450
2023-11-20 16:31:26,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1136333.3333333333, ans=0.125
2023-11-20 16:31:35,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1136333.3333333333, ans=0.125
2023-11-20 16:32:07,323 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2150, loss[loss=0.08326, simple_loss=0.114, pruned_loss=0.01618, audio_tagging_loss=0.01006, over 15835.00 frames. ], tot_loss[loss=0.07848, simple_loss=0.09939, pruned_loss=0.01901, audio_tagging_loss=0.009786, over 3035203.28 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:32:18,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1136533.3333333333, ans=0.1
2023-11-20 16:32:30,339 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170500
2023-11-20 16:32:32,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.46 vs. limit=15.0
2023-11-20 16:32:36,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1136666.6666666667, ans=0.125
2023-11-20 16:32:46,667 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:32:51,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1136733.3333333333, ans=0.07
2023-11-20 16:32:53,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1136733.3333333333, ans=0.125
2023-11-20 16:33:12,831 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2200, loss[loss=0.07163, simple_loss=0.0932, pruned_loss=0.01682, audio_tagging_loss=0.008212, over 15044.00 frames. ], tot_loss[loss=0.07806, simple_loss=0.09897, pruned_loss=0.01875, audio_tagging_loss=0.009825, over 3034230.35 frames. ], batch size: 57, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:33:17,327 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.36 vs. limit=15.0
2023-11-20 16:33:25,933 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.367e+01 8.465e+01 9.006e+01 9.663e+01 1.304e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-20 16:33:36,751 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170550
2023-11-20 16:33:38,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1137000.0, ans=0.5
2023-11-20 16:33:38,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1137000.0, ans=0.0
2023-11-20 16:33:46,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1137000.0, ans=0.2
2023-11-20 16:33:55,848 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=2.657e-03
2023-11-20 16:33:56,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1137066.6666666667, ans=0.125
2023-11-20 16:34:18,610 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2250, loss[loss=0.09058, simple_loss=0.1274, pruned_loss=0.01897, audio_tagging_loss=0.007896, over 15197.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09917, pruned_loss=0.0187, audio_tagging_loss=0.009848, over 3033113.45 frames. ], batch size: 55, lr: 4.64e-03, grad_scale: 16.0
2023-11-20 16:34:25,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1137200.0, ans=0.125
2023-11-20 16:34:33,080 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.56 vs. limit=12.0
2023-11-20 16:34:38,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.87 vs. limit=15.0
2023-11-20 16:34:43,194 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170600
2023-11-20 16:34:53,182 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.05 vs. limit=15.0
2023-11-20 16:35:25,855 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2300, loss[loss=0.093, simple_loss=0.1241, pruned_loss=0.02303, audio_tagging_loss=0.007898, over 15713.00 frames. ], tot_loss[loss=0.0791, simple_loss=0.1007, pruned_loss=0.01895, audio_tagging_loss=0.009811, over 3026507.63 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 16.0
2023-11-20 16:35:38,747 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.000e+01 8.643e+01 9.323e+01 1.269e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-20 16:35:48,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170650
2023-11-20 16:36:14,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1137733.3333333333, ans=0.5
2023-11-20 16:36:23,768 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:36:31,355 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2350, loss[loss=0.06211, simple_loss=0.07112, pruned_loss=0.01418, audio_tagging_loss=0.01237, over 15194.00 frames. ], tot_loss[loss=0.08, simple_loss=0.1015, pruned_loss=0.01934, audio_tagging_loss=0.009937, over 3037130.75 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 16.0
2023-11-20 16:36:36,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1137866.6666666667, ans=0.125
2023-11-20 16:36:36,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1137866.6666666667, ans=0.125
2023-11-20 16:36:43,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1137933.3333333333, ans=0.05
2023-11-20 16:36:48,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1137933.3333333333, ans=0.125
2023-11-20 16:36:54,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170700
2023-11-20 16:36:56,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1138000.0, ans=0.125
2023-11-20 16:37:02,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1138000.0, ans=0.125
2023-11-20 16:37:03,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1138000.0, ans=0.2
2023-11-20 16:37:19,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1138066.6666666667, ans=10.0
2023-11-20 16:37:23,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff3.min_abs, batch_count=1138133.3333333333, ans=0.2
2023-11-20 16:37:33,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1138133.3333333333, ans=0.125
2023-11-20 16:37:36,636 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2400, loss[loss=0.09599, simple_loss=0.1214, pruned_loss=0.02406, audio_tagging_loss=0.01124, over 15817.00 frames. ], tot_loss[loss=0.08047, simple_loss=0.1018, pruned_loss=0.01945, audio_tagging_loss=0.01012, over 3039006.22 frames. ], batch size: 56, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:37:37,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.54 vs. limit=15.0
2023-11-20 16:37:50,906 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.259e+01 8.944e+01 9.702e+01 1.216e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-20 16:37:52,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1138266.6666666667, ans=0.2
2023-11-20 16:38:01,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170750
2023-11-20 16:38:19,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1138400.0, ans=0.07
2023-11-20 16:38:32,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1138466.6666666667, ans=0.125
2023-11-20 16:38:37,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1138466.6666666667, ans=0.125
2023-11-20 16:38:41,437 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.42 vs. limit=15.0
2023-11-20 16:38:43,349 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2450, loss[loss=0.07754, simple_loss=0.102, pruned_loss=0.0175, audio_tagging_loss=0.009061, over 15414.00 frames. ], tot_loss[loss=0.08004, simple_loss=0.1013, pruned_loss=0.01918, audio_tagging_loss=0.01021, over 3041405.66 frames. ], batch size: 60, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:38:49,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1138533.3333333333, ans=0.125
2023-11-20 16:38:58,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1138600.0, ans=0.125
2023-11-20 16:38:59,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1138600.0, ans=0.1
2023-11-20 16:39:06,329 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170800
2023-11-20 16:39:22,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1138733.3333333333, ans=0.1
2023-11-20 16:39:23,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1138733.3333333333, ans=0.0
2023-11-20 16:39:35,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1138800.0, ans=0.2
2023-11-20 16:39:36,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1138800.0, ans=0.125
2023-11-20 16:39:38,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1138800.0, ans=0.0
2023-11-20 16:39:48,745 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2500, loss[loss=0.08718, simple_loss=0.1098, pruned_loss=0.0217, audio_tagging_loss=0.01056, over 15933.00 frames. ], tot_loss[loss=0.07923, simple_loss=0.1, pruned_loss=0.01887, audio_tagging_loss=0.01034, over 3043295.81 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:39:48,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1138866.6666666667, ans=0.0
2023-11-20 16:39:56,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1138866.6666666667, ans=0.125
2023-11-20 16:40:00,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1138933.3333333333, ans=0.0
2023-11-20 16:40:01,217 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.113e+01 8.897e+01 9.641e+01 1.351e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 16:40:11,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170850
2023-11-20 16:40:15,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1139000.0, ans=0.0
2023-11-20 16:40:53,585 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2550, loss[loss=0.07223, simple_loss=0.08306, pruned_loss=0.01983, audio_tagging_loss=0.01088, over 15124.00 frames. ], tot_loss[loss=0.07823, simple_loss=0.09894, pruned_loss=0.01859, audio_tagging_loss=0.01017, over 3044166.61 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:41:05,059 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:41:15,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1139266.6666666667, ans=0.125
2023-11-20 16:41:18,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170900
2023-11-20 16:41:30,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1139333.3333333333, ans=0.125
2023-11-20 16:41:36,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1139400.0, ans=0.04949747468305833
2023-11-20 16:41:40,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1139400.0, ans=0.125
2023-11-20 16:41:40,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1139400.0, ans=0.125
2023-11-20 16:41:41,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1139400.0, ans=0.125
2023-11-20 16:41:59,392 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2600, loss[loss=0.07047, simple_loss=0.08718, pruned_loss=0.01413, audio_tagging_loss=0.01274, over 15303.00 frames. ], tot_loss[loss=0.07794, simple_loss=0.09884, pruned_loss=0.01849, audio_tagging_loss=0.01003, over 3041078.84 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:41:59,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1139533.3333333333, ans=0.125
2023-11-20 16:42:02,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1139533.3333333333, ans=0.125
2023-11-20 16:42:02,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1139533.3333333333, ans=0.0
2023-11-20 16:42:08,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.38 vs. limit=15.0
2023-11-20 16:42:10,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1139533.3333333333, ans=0.125
2023-11-20 16:42:13,172 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.062e+01 8.877e+01 9.575e+01 1.292e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-20 16:42:17,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1139600.0, ans=0.0
2023-11-20 16:42:23,171 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 170950
2023-11-20 16:42:27,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1139666.6666666667, ans=0.2
2023-11-20 16:43:01,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1139800.0, ans=0.1
2023-11-20 16:43:05,752 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2650, loss[loss=0.08797, simple_loss=0.1189, pruned_loss=0.021, audio_tagging_loss=0.007503, over 14969.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.09907, pruned_loss=0.0187, audio_tagging_loss=0.009945, over 3036558.70 frames. ], batch size: 56, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:43:09,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.47 vs. limit=22.5
2023-11-20 16:43:28,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171000
2023-11-20 16:44:10,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1140200.0, ans=0.04949747468305833
2023-11-20 16:44:11,713 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2700, loss[loss=0.09579, simple_loss=0.1132, pruned_loss=0.02762, audio_tagging_loss=0.01158, over 16678.00 frames. ], tot_loss[loss=0.07839, simple_loss=0.09951, pruned_loss=0.01873, audio_tagging_loss=0.009896, over 3041115.56 frames. ], batch size: 63, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:44:13,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1140200.0, ans=0.125
2023-11-20 16:44:24,010 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.274e+01 8.361e+01 8.930e+01 9.610e+01 1.277e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-20 16:44:28,882 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.50 vs. limit=22.5
2023-11-20 16:44:35,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171050
2023-11-20 16:44:42,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1140333.3333333333, ans=0.1
2023-11-20 16:44:48,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1140333.3333333333, ans=0.125
2023-11-20 16:45:11,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1140466.6666666667, ans=15.0
2023-11-20 16:45:15,551 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2750, loss[loss=0.05096, simple_loss=0.05154, pruned_loss=0.009751, audio_tagging_loss=0.01544, over 15380.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09806, pruned_loss=0.01843, audio_tagging_loss=0.01003, over 3040665.69 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:45:40,314 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171100
2023-11-20 16:45:42,089 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.81 vs. limit=15.0
2023-11-20 16:45:46,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1140666.6666666667, ans=0.0
2023-11-20 16:46:12,571 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:46:22,416 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2800, loss[loss=0.06858, simple_loss=0.08502, pruned_loss=0.01467, audio_tagging_loss=0.01141, over 16456.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09781, pruned_loss=0.01833, audio_tagging_loss=0.01007, over 3049909.23 frames. ], batch size: 62, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:46:26,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1140866.6666666667, ans=0.07
2023-11-20 16:46:30,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.05 vs. limit=15.0
2023-11-20 16:46:34,707 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 7.973e+01 8.545e+01 9.454e+01 1.250e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-20 16:46:35,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1140933.3333333333, ans=0.125
2023-11-20 16:46:44,791 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171150
2023-11-20 16:46:51,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1141000.0, ans=0.125
2023-11-20 16:47:00,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1141066.6666666667, ans=0.0
2023-11-20 16:47:01,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1141066.6666666667, ans=0.125
2023-11-20 16:47:02,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1141066.6666666667, ans=22.5
2023-11-20 16:47:22,535 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.70 vs. limit=22.5
2023-11-20 16:47:23,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1141133.3333333333, ans=0.1
2023-11-20 16:47:26,728 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2850, loss[loss=0.06897, simple_loss=0.09233, pruned_loss=0.01331, audio_tagging_loss=0.009497, over 14691.00 frames. ], tot_loss[loss=0.07724, simple_loss=0.0981, pruned_loss=0.01834, audio_tagging_loss=0.009851, over 3038864.73 frames. ], batch size: 54, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:47:50,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171200
2023-11-20 16:47:58,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1141333.3333333333, ans=0.0
2023-11-20 16:48:25,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1141466.6666666667, ans=0.0
2023-11-20 16:48:31,845 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2900, loss[loss=0.07777, simple_loss=0.09614, pruned_loss=0.01903, audio_tagging_loss=0.01067, over 15059.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09806, pruned_loss=0.01853, audio_tagging_loss=0.009897, over 3030730.18 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:48:32,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1141533.3333333333, ans=0.0
2023-11-20 16:48:39,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1141533.3333333333, ans=0.1
2023-11-20 16:48:41,248 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.60 vs. limit=15.0
2023-11-20 16:48:45,412 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.270e+01 7.954e+01 8.776e+01 9.382e+01 1.409e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-20 16:48:56,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171250
2023-11-20 16:48:56,746 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:49:11,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.32 vs. limit=15.0
2023-11-20 16:49:37,931 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 2950, loss[loss=0.09155, simple_loss=0.1259, pruned_loss=0.02135, audio_tagging_loss=0.007237, over 14776.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09814, pruned_loss=0.01859, audio_tagging_loss=0.009971, over 3034163.74 frames. ], batch size: 53, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:50:01,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171300
2023-11-20 16:50:15,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1142066.6666666667, ans=0.125
2023-11-20 16:50:17,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1142066.6666666667, ans=0.125
2023-11-20 16:50:28,682 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.45 vs. limit=6.0
2023-11-20 16:50:32,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1142133.3333333333, ans=0.0
2023-11-20 16:50:41,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1142133.3333333333, ans=0.125
2023-11-20 16:50:43,601 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3000, loss[loss=0.08091, simple_loss=0.09303, pruned_loss=0.02437, audio_tagging_loss=0.01002, over 14667.00 frames. ], tot_loss[loss=0.07819, simple_loss=0.09884, pruned_loss=0.01876, audio_tagging_loss=0.01002, over 3042278.40 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:50:43,602 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 16:51:23,431 INFO [train_asr.py:1253] (2/4) Epoch 15, validation: loss=0.06163, simple_loss=0.05329, pruned_loss=0.005569, audio_tagging_loss=0.02942, over 4681554.00 frames. 
2023-11-20 16:51:23,432 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 16:51:34,498 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.06 vs. limit=15.0
2023-11-20 16:51:36,341 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.037e+01 8.379e+01 9.112e+01 1.041e+02 3.133e+02, threshold=1.822e+02, percent-clipped=1.0
2023-11-20 16:51:37,009 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.65 vs. limit=6.0
2023-11-20 16:51:37,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1142266.6666666667, ans=6.0
2023-11-20 16:51:46,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171350
2023-11-20 16:51:49,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1142333.3333333333, ans=0.025
2023-11-20 16:52:28,812 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3050, loss[loss=0.09969, simple_loss=0.1209, pruned_loss=0.02797, audio_tagging_loss=0.01124, over 15459.00 frames. ], tot_loss[loss=0.07901, simple_loss=0.09977, pruned_loss=0.01907, audio_tagging_loss=0.01006, over 3044148.24 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:52:34,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1142533.3333333333, ans=0.125
2023-11-20 16:52:41,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.69 vs. limit=10.0
2023-11-20 16:52:51,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171400
2023-11-20 16:52:59,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1142666.6666666667, ans=0.125
2023-11-20 16:53:01,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1142666.6666666667, ans=0.125
2023-11-20 16:53:07,708 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:53:08,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1142733.3333333333, ans=0.0
2023-11-20 16:53:29,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1142800.0, ans=0.125
2023-11-20 16:53:33,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1142866.6666666667, ans=0.05
2023-11-20 16:53:34,118 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3100, loss[loss=0.0996, simple_loss=0.1349, pruned_loss=0.02181, audio_tagging_loss=0.01031, over 14904.00 frames. ], tot_loss[loss=0.07941, simple_loss=0.1005, pruned_loss=0.01909, audio_tagging_loss=0.01009, over 3043136.35 frames. ], batch size: 53, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:53:44,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1142866.6666666667, ans=0.0
2023-11-20 16:53:46,515 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.278e+01 8.696e+01 9.346e+01 1.301e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-20 16:53:51,236 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.77 vs. limit=22.5
2023-11-20 16:53:57,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171450
2023-11-20 16:53:58,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.55 vs. limit=15.0
2023-11-20 16:54:05,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1143000.0, ans=0.125
2023-11-20 16:54:27,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1143133.3333333333, ans=0.0
2023-11-20 16:54:36,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1143133.3333333333, ans=0.125
2023-11-20 16:54:39,661 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3150, loss[loss=0.08111, simple_loss=0.104, pruned_loss=0.01894, audio_tagging_loss=0.01018, over 14233.00 frames. ], tot_loss[loss=0.0794, simple_loss=0.1006, pruned_loss=0.0191, audio_tagging_loss=0.01002, over 3047344.06 frames. ], batch size: 53, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:54:46,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1143200.0, ans=0.125
2023-11-20 16:54:52,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1143266.6666666667, ans=0.0
2023-11-20 16:54:55,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1143266.6666666667, ans=0.1
2023-11-20 16:55:03,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171500
2023-11-20 16:55:39,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1143466.6666666667, ans=0.125
2023-11-20 16:55:45,982 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3200, loss[loss=0.09258, simple_loss=0.1173, pruned_loss=0.02442, audio_tagging_loss=0.009523, over 15984.00 frames. ], tot_loss[loss=0.07992, simple_loss=0.1012, pruned_loss=0.0192, audio_tagging_loss=0.01013, over 3054519.80 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:55:58,481 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.865e+01 8.146e+01 8.709e+01 9.815e+01 1.794e+02, threshold=1.742e+02, percent-clipped=1.0
2023-11-20 16:56:08,449 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.35 vs. limit=15.0
2023-11-20 16:56:09,261 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171550
2023-11-20 16:56:21,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1143666.6666666667, ans=0.0
2023-11-20 16:56:41,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1143800.0, ans=0.0
2023-11-20 16:56:49,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1143866.6666666667, ans=0.95
2023-11-20 16:56:50,354 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3250, loss[loss=0.05401, simple_loss=0.06157, pruned_loss=0.0101, audio_tagging_loss=0.01313, over 13649.00 frames. ], tot_loss[loss=0.07932, simple_loss=0.1, pruned_loss=0.01903, audio_tagging_loss=0.01027, over 3050916.62 frames. ], batch size: 55, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:56:56,586 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0
2023-11-20 16:57:14,009 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171600
2023-11-20 16:57:21,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1144000.0, ans=0.1
2023-11-20 16:57:28,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1144066.6666666667, ans=0.1
2023-11-20 16:57:40,493 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.54 vs. limit=12.0
2023-11-20 16:57:42,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1144133.3333333333, ans=0.125
2023-11-20 16:57:45,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1144133.3333333333, ans=0.0
2023-11-20 16:57:55,268 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3300, loss[loss=0.1019, simple_loss=0.1423, pruned_loss=0.02336, audio_tagging_loss=0.007434, over 17029.00 frames. ], tot_loss[loss=0.0789, simple_loss=0.09963, pruned_loss=0.01885, audio_tagging_loss=0.01023, over 3053802.95 frames. ], batch size: 61, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:58:07,393 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.32 vs. limit=15.0
2023-11-20 16:58:09,268 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.004e+01 8.604e+01 9.340e+01 1.280e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-20 16:58:20,151 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171650
2023-11-20 16:58:21,872 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.82 vs. limit=15.0
2023-11-20 16:58:32,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1144333.3333333333, ans=0.125
2023-11-20 16:59:02,013 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3350, loss[loss=0.08255, simple_loss=0.101, pruned_loss=0.0229, audio_tagging_loss=0.009168, over 15458.00 frames. ], tot_loss[loss=0.07864, simple_loss=0.09949, pruned_loss=0.01877, audio_tagging_loss=0.01012, over 3056817.97 frames. ], batch size: 59, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:59:18,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1144600.0, ans=0.2
2023-11-20 16:59:24,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171700
2023-11-20 16:59:28,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1144666.6666666667, ans=0.125
2023-11-20 17:00:05,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1144866.6666666667, ans=0.0
2023-11-20 17:00:06,505 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3400, loss[loss=0.0753, simple_loss=0.09394, pruned_loss=0.01781, audio_tagging_loss=0.01052, over 16982.00 frames. ], tot_loss[loss=0.07886, simple_loss=0.09994, pruned_loss=0.01893, audio_tagging_loss=0.009961, over 3050072.76 frames. ], batch size: 65, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:00:12,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.80 vs. limit=22.5
2023-11-20 17:00:16,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1144866.6666666667, ans=0.125
2023-11-20 17:00:20,872 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.042e+01 8.596e+01 9.172e+01 1.141e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-20 17:00:25,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.63 vs. limit=10.0
2023-11-20 17:00:30,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171750
2023-11-20 17:00:32,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.56 vs. limit=15.0
2023-11-20 17:00:52,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1145066.6666666667, ans=0.125
2023-11-20 17:00:58,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1145133.3333333333, ans=0.025
2023-11-20 17:01:12,060 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3450, loss[loss=0.0674, simple_loss=0.08071, pruned_loss=0.01558, audio_tagging_loss=0.01147, over 15517.00 frames. ], tot_loss[loss=0.0789, simple_loss=0.1002, pruned_loss=0.01891, audio_tagging_loss=0.0099, over 3051523.19 frames. ], batch size: 59, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:01:23,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1145200.0, ans=0.0
2023-11-20 17:01:36,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171800
2023-11-20 17:01:36,472 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:01:39,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1145333.3333333333, ans=0.1
2023-11-20 17:01:49,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1145333.3333333333, ans=0.125
2023-11-20 17:01:54,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1145400.0, ans=0.1
2023-11-20 17:02:09,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0
2023-11-20 17:02:16,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1145466.6666666667, ans=0.125
2023-11-20 17:02:18,781 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3500, loss[loss=0.06697, simple_loss=0.08068, pruned_loss=0.01649, audio_tagging_loss=0.01014, over 14529.00 frames. ], tot_loss[loss=0.08007, simple_loss=0.1018, pruned_loss=0.01948, audio_tagging_loss=0.009687, over 3055133.10 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:02:27,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1145533.3333333333, ans=0.1
2023-11-20 17:02:33,117 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.582e+01 8.137e+01 8.586e+01 9.265e+01 1.254e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-20 17:02:41,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171850
2023-11-20 17:02:47,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1145666.6666666667, ans=0.125
2023-11-20 17:02:51,703 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:03:01,178 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.94 vs. limit=15.0
2023-11-20 17:03:08,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1145733.3333333333, ans=0.125
2023-11-20 17:03:10,534 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:03:14,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1145800.0, ans=0.04949747468305833
2023-11-20 17:03:14,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1145800.0, ans=0.0
2023-11-20 17:03:24,244 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3550, loss[loss=0.04639, simple_loss=0.05298, pruned_loss=0.01112, audio_tagging_loss=0.00878, over 14799.00 frames. ], tot_loss[loss=0.07908, simple_loss=0.1002, pruned_loss=0.01916, audio_tagging_loss=0.009802, over 3045658.55 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:03:34,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1145866.6666666667, ans=0.0
2023-11-20 17:03:40,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1145933.3333333333, ans=0.125
2023-11-20 17:03:47,513 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171900
2023-11-20 17:04:00,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1146000.0, ans=0.0
2023-11-20 17:04:29,147 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3600, loss[loss=0.07551, simple_loss=0.08378, pruned_loss=0.02325, audio_tagging_loss=0.01037, over 14624.00 frames. ], tot_loss[loss=0.07918, simple_loss=0.1003, pruned_loss=0.01921, audio_tagging_loss=0.009823, over 3043867.49 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 17:04:43,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1146266.6666666667, ans=0.125
2023-11-20 17:04:44,611 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.710e+01 8.052e+01 8.613e+01 9.290e+01 1.173e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-20 17:04:51,986 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=12.0
2023-11-20 17:04:53,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 171950
2023-11-20 17:05:24,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1146466.6666666667, ans=0.125
2023-11-20 17:05:34,597 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3650, loss[loss=0.07499, simple_loss=0.1019, pruned_loss=0.0162, audio_tagging_loss=0.007819, over 15323.00 frames. ], tot_loss[loss=0.07941, simple_loss=0.1005, pruned_loss=0.01941, audio_tagging_loss=0.009729, over 3043327.18 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 17:05:51,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1146600.0, ans=0.0
2023-11-20 17:05:54,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1146600.0, ans=15.0
2023-11-20 17:05:57,530 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172000
2023-11-20 17:06:11,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1146666.6666666667, ans=0.125
2023-11-20 17:06:27,767 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.69 vs. limit=15.0
2023-11-20 17:06:30,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1146800.0, ans=0.125
2023-11-20 17:06:38,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.37 vs. limit=15.0
2023-11-20 17:06:42,353 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3700, loss[loss=0.08812, simple_loss=0.1141, pruned_loss=0.02193, audio_tagging_loss=0.009135, over 15425.00 frames. ], tot_loss[loss=0.07876, simple_loss=0.09988, pruned_loss=0.01914, audio_tagging_loss=0.009682, over 3048900.61 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 17:06:47,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1146866.6666666667, ans=0.0
2023-11-20 17:06:48,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1146866.6666666667, ans=0.2
2023-11-20 17:06:56,085 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.121e+01 8.690e+01 9.493e+01 1.302e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 17:06:57,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1146933.3333333333, ans=0.0
2023-11-20 17:07:03,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1146933.3333333333, ans=0.0
2023-11-20 17:07:05,258 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172050
2023-11-20 17:07:10,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1147000.0, ans=0.5
2023-11-20 17:07:46,849 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3750, loss[loss=0.08139, simple_loss=0.101, pruned_loss=0.01806, audio_tagging_loss=0.01284, over 15321.00 frames. ], tot_loss[loss=0.07897, simple_loss=0.09996, pruned_loss=0.01919, audio_tagging_loss=0.009798, over 3048647.30 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:07:50,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1147200.0, ans=0.125
2023-11-20 17:08:05,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1147266.6666666667, ans=0.1
2023-11-20 17:08:11,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172100
2023-11-20 17:08:28,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.29 vs. limit=5.0
2023-11-20 17:08:32,660 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:08:44,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1147466.6666666667, ans=0.125
2023-11-20 17:08:52,189 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3800, loss[loss=0.07546, simple_loss=0.0953, pruned_loss=0.01683, audio_tagging_loss=0.01098, over 16457.00 frames. ], tot_loss[loss=0.07926, simple_loss=0.1004, pruned_loss=0.01921, audio_tagging_loss=0.009842, over 3055016.05 frames. ], batch size: 60, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:09:02,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.49 vs. limit=15.0
2023-11-20 17:09:08,172 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.643e+01 8.118e+01 8.690e+01 9.561e+01 1.262e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 17:09:10,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1147600.0, ans=0.125
2023-11-20 17:09:15,862 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172150
2023-11-20 17:09:19,120 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.99 vs. limit=15.0
2023-11-20 17:09:32,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1147733.3333333333, ans=0.125
2023-11-20 17:09:58,586 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3850, loss[loss=0.06272, simple_loss=0.07934, pruned_loss=0.01086, audio_tagging_loss=0.01219, over 14060.00 frames. ], tot_loss[loss=0.07849, simple_loss=0.09969, pruned_loss=0.01877, audio_tagging_loss=0.00988, over 3055055.87 frames. ], batch size: 52, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:10:15,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1147933.3333333333, ans=0.125
2023-11-20 17:10:21,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172200
2023-11-20 17:10:38,237 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.84 vs. limit=15.0
2023-11-20 17:10:55,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1148133.3333333333, ans=0.2
2023-11-20 17:11:04,020 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3900, loss[loss=0.08393, simple_loss=0.1097, pruned_loss=0.0213, audio_tagging_loss=0.007757, over 14995.00 frames. ], tot_loss[loss=0.079, simple_loss=0.1004, pruned_loss=0.01891, audio_tagging_loss=0.009904, over 3049157.55 frames. ], batch size: 55, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:11:19,574 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.094e+01 8.942e+01 9.624e+01 1.235e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 17:11:22,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1148266.6666666667, ans=0.0
2023-11-20 17:11:25,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1148266.6666666667, ans=0.0
2023-11-20 17:11:29,188 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172250
2023-11-20 17:11:50,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1148400.0, ans=0.0
2023-11-20 17:12:00,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1148466.6666666667, ans=0.2
2023-11-20 17:12:08,857 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 3950, loss[loss=0.07188, simple_loss=0.08921, pruned_loss=0.01719, audio_tagging_loss=0.01008, over 14630.00 frames. ], tot_loss[loss=0.0794, simple_loss=0.1007, pruned_loss=0.019, audio_tagging_loss=0.01004, over 3046484.95 frames. ], batch size: 55, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:12:13,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1148533.3333333333, ans=0.0
2023-11-20 17:12:31,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1148600.0, ans=0.125
2023-11-20 17:12:32,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172300
2023-11-20 17:12:34,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1148666.6666666667, ans=0.125
2023-11-20 17:12:37,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1148666.6666666667, ans=0.125
2023-11-20 17:12:43,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1148666.6666666667, ans=0.2
2023-11-20 17:12:47,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1148733.3333333333, ans=0.125
2023-11-20 17:13:12,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1148800.0, ans=0.0
2023-11-20 17:13:14,339 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4000, loss[loss=0.08354, simple_loss=0.1085, pruned_loss=0.01899, audio_tagging_loss=0.01032, over 14461.00 frames. ], tot_loss[loss=0.07903, simple_loss=0.09993, pruned_loss=0.0189, audio_tagging_loss=0.01017, over 3052629.73 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:13:15,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1148866.6666666667, ans=0.125
2023-11-20 17:13:19,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1148866.6666666667, ans=0.0
2023-11-20 17:13:29,466 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 7.998e+01 8.708e+01 9.550e+01 1.131e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-20 17:13:37,146 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172350
2023-11-20 17:13:59,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1149066.6666666667, ans=0.0
2023-11-20 17:14:02,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1149066.6666666667, ans=0.125
2023-11-20 17:14:08,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1149133.3333333333, ans=0.1
2023-11-20 17:14:10,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1149133.3333333333, ans=0.125
2023-11-20 17:14:18,633 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4050, loss[loss=0.05529, simple_loss=0.06048, pruned_loss=0.01459, audio_tagging_loss=0.01046, over 14722.00 frames. ], tot_loss[loss=0.07889, simple_loss=0.09938, pruned_loss=0.01892, audio_tagging_loss=0.01027, over 3043865.07 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:14:20,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1149200.0, ans=0.0
2023-11-20 17:14:22,398 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:14:23,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.50 vs. limit=15.0
2023-11-20 17:14:27,633 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:14:42,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172400
2023-11-20 17:14:43,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.70 vs. limit=22.5
2023-11-20 17:14:49,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1149333.3333333333, ans=0.125
2023-11-20 17:14:55,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1149333.3333333333, ans=0.1
2023-11-20 17:15:10,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1149466.6666666667, ans=0.1
2023-11-20 17:15:10,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1149466.6666666667, ans=0.125
2023-11-20 17:15:24,176 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4100, loss[loss=0.08501, simple_loss=0.1018, pruned_loss=0.02345, audio_tagging_loss=0.01065, over 13178.00 frames. ], tot_loss[loss=0.07911, simple_loss=0.1001, pruned_loss=0.0189, audio_tagging_loss=0.01014, over 3040972.34 frames. ], batch size: 50, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:15:29,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1149533.3333333333, ans=0.125
2023-11-20 17:15:42,058 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.166e+01 8.151e+01 8.717e+01 9.713e+01 1.333e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 17:15:42,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1149600.0, ans=0.0
2023-11-20 17:15:48,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172450
2023-11-20 17:16:14,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.62 vs. limit=15.0
2023-11-20 17:16:17,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1149800.0, ans=0.2
2023-11-20 17:16:30,764 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4150, loss[loss=0.0773, simple_loss=0.09158, pruned_loss=0.02071, audio_tagging_loss=0.0108, over 16512.00 frames. ], tot_loss[loss=0.07923, simple_loss=0.1003, pruned_loss=0.01902, audio_tagging_loss=0.01005, over 3047604.36 frames. ], batch size: 62, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:16:32,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1149866.6666666667, ans=0.1
2023-11-20 17:16:53,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172500
2023-11-20 17:17:00,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1150000.0, ans=0.125
2023-11-20 17:17:04,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1150000.0, ans=0.2
2023-11-20 17:17:18,181 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:17:20,954 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:17:36,194 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4200, loss[loss=0.07421, simple_loss=0.09508, pruned_loss=0.01898, audio_tagging_loss=0.007693, over 15504.00 frames. ], tot_loss[loss=0.07884, simple_loss=0.09987, pruned_loss=0.01895, audio_tagging_loss=0.009946, over 3055004.01 frames. ], batch size: 60, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:17:52,265 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.093e+01 8.811e+01 9.474e+01 1.183e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-20 17:17:59,887 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172550
2023-11-20 17:18:00,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1150266.6666666667, ans=0.1
2023-11-20 17:18:40,995 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4250, loss[loss=0.09442, simple_loss=0.1164, pruned_loss=0.02525, audio_tagging_loss=0.01096, over 15889.00 frames. ], tot_loss[loss=0.07953, simple_loss=0.1009, pruned_loss=0.01917, audio_tagging_loss=0.009892, over 3052599.54 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:19:00,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1150600.0, ans=0.125
2023-11-20 17:19:05,235 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172600
2023-11-20 17:19:07,048 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.26 vs. limit=10.0
2023-11-20 17:19:16,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1150666.6666666667, ans=0.0
2023-11-20 17:19:19,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1150733.3333333333, ans=0.125
2023-11-20 17:19:23,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1150733.3333333333, ans=0.125
2023-11-20 17:19:29,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1150733.3333333333, ans=0.025
2023-11-20 17:19:32,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1150800.0, ans=0.125
2023-11-20 17:19:40,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1150800.0, ans=0.125
2023-11-20 17:19:47,490 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4300, loss[loss=0.06662, simple_loss=0.07407, pruned_loss=0.01818, audio_tagging_loss=0.01139, over 16190.00 frames. ], tot_loss[loss=0.07955, simple_loss=0.1011, pruned_loss=0.01926, audio_tagging_loss=0.009756, over 3050542.42 frames. ], batch size: 63, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:20:00,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.97 vs. limit=15.0
2023-11-20 17:20:04,272 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.964e+01 7.979e+01 8.573e+01 9.478e+01 1.236e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-20 17:20:10,713 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172650
2023-11-20 17:20:40,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1151133.3333333333, ans=0.125
2023-11-20 17:20:52,666 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4350, loss[loss=0.07682, simple_loss=0.09771, pruned_loss=0.0158, audio_tagging_loss=0.01216, over 15345.00 frames. ], tot_loss[loss=0.07966, simple_loss=0.1014, pruned_loss=0.01928, audio_tagging_loss=0.009695, over 3052085.88 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:20:55,933 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.34 vs. limit=15.0
2023-11-20 17:20:56,810 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:20:58,087 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.64 vs. limit=22.5
2023-11-20 17:21:04,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1151266.6666666667, ans=0.0
2023-11-20 17:21:04,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.98 vs. limit=15.0
2023-11-20 17:21:15,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172700
2023-11-20 17:21:17,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1151333.3333333333, ans=0.125
2023-11-20 17:21:33,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1151400.0, ans=0.125
2023-11-20 17:21:40,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1151400.0, ans=0.0
2023-11-20 17:21:57,281 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4400, loss[loss=0.0742, simple_loss=0.09573, pruned_loss=0.01637, audio_tagging_loss=0.009962, over 14194.00 frames. ], tot_loss[loss=0.07963, simple_loss=0.1013, pruned_loss=0.0193, audio_tagging_loss=0.009709, over 3047228.40 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:22:16,306 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.328e+01 8.168e+01 8.779e+01 9.674e+01 1.363e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-20 17:22:21,387 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172750
2023-11-20 17:23:02,741 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4450, loss[loss=0.07889, simple_loss=0.1036, pruned_loss=0.01828, audio_tagging_loss=0.008801, over 16861.00 frames. ], tot_loss[loss=0.07888, simple_loss=0.1003, pruned_loss=0.01904, audio_tagging_loss=0.009712, over 3054344.94 frames. ], batch size: 62, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:23:03,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1151866.6666666667, ans=0.0
2023-11-20 17:23:26,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172800
2023-11-20 17:23:37,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.47 vs. limit=6.0
2023-11-20 17:24:08,395 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4500, loss[loss=0.06572, simple_loss=0.08367, pruned_loss=0.01365, audio_tagging_loss=0.01023, over 15196.00 frames. ], tot_loss[loss=0.07807, simple_loss=0.09922, pruned_loss=0.01873, audio_tagging_loss=0.009736, over 3057200.29 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:24:13,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1152200.0, ans=0.125
2023-11-20 17:24:19,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1152200.0, ans=0.1
2023-11-20 17:24:20,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1152266.6666666667, ans=0.0
2023-11-20 17:24:23,975 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:24:26,178 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.191e+01 8.215e+01 8.732e+01 9.444e+01 1.886e+02, threshold=1.746e+02, percent-clipped=1.0
2023-11-20 17:24:31,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172850
2023-11-20 17:24:37,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1152333.3333333333, ans=0.0
2023-11-20 17:24:45,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1152333.3333333333, ans=0.125
2023-11-20 17:24:56,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1152400.0, ans=0.1
2023-11-20 17:24:57,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1152400.0, ans=0.125
2023-11-20 17:25:13,101 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4550, loss[loss=0.08631, simple_loss=0.1114, pruned_loss=0.0201, audio_tagging_loss=0.01054, over 15222.00 frames. ], tot_loss[loss=0.07808, simple_loss=0.09921, pruned_loss=0.01877, audio_tagging_loss=0.009711, over 3059433.95 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:25:13,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1152533.3333333333, ans=0.125
2023-11-20 17:25:37,530 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172900
2023-11-20 17:25:55,496 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.20 vs. limit=22.5
2023-11-20 17:26:02,829 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:26:13,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1152800.0, ans=0.0
2023-11-20 17:26:15,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1152800.0, ans=0.125
2023-11-20 17:26:18,993 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4600, loss[loss=0.06957, simple_loss=0.09512, pruned_loss=0.01185, audio_tagging_loss=0.01017, over 14679.00 frames. ], tot_loss[loss=0.07884, simple_loss=0.09995, pruned_loss=0.019, audio_tagging_loss=0.009869, over 3056606.67 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:26:28,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1152866.6666666667, ans=0.0
2023-11-20 17:26:36,938 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 8.346e+01 9.151e+01 1.010e+02 1.307e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-20 17:26:42,010 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 172950
2023-11-20 17:27:06,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1153066.6666666667, ans=0.125
2023-11-20 17:27:16,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=15.0
2023-11-20 17:27:24,114 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4650, loss[loss=0.05213, simple_loss=0.0627, pruned_loss=0.01286, audio_tagging_loss=0.007926, over 14020.00 frames. ], tot_loss[loss=0.07814, simple_loss=0.09868, pruned_loss=0.01881, audio_tagging_loss=0.009987, over 3048838.85 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:27:29,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1153200.0, ans=0.0
2023-11-20 17:27:37,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1153266.6666666667, ans=0.125
2023-11-20 17:27:46,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173000
2023-11-20 17:27:48,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1153333.3333333333, ans=0.0
2023-11-20 17:28:04,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1153400.0, ans=0.2
2023-11-20 17:28:17,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.51 vs. limit=22.5
2023-11-20 17:28:29,538 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4700, loss[loss=0.07172, simple_loss=0.08592, pruned_loss=0.017, audio_tagging_loss=0.01176, over 15348.00 frames. ], tot_loss[loss=0.07854, simple_loss=0.09914, pruned_loss=0.01891, audio_tagging_loss=0.01006, over 3054481.55 frames. ], batch size: 58, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:28:33,934 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.45 vs. limit=15.0
2023-11-20 17:28:43,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1153600.0, ans=0.035
2023-11-20 17:28:49,105 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 7.947e+01 8.690e+01 9.286e+01 1.416e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 17:28:50,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1153600.0, ans=0.0
2023-11-20 17:28:54,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173050
2023-11-20 17:29:01,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1153666.6666666667, ans=0.125
2023-11-20 17:29:35,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1153866.6666666667, ans=0.0
2023-11-20 17:29:36,591 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4750, loss[loss=0.07638, simple_loss=0.08517, pruned_loss=0.02095, audio_tagging_loss=0.01284, over 14595.00 frames. ], tot_loss[loss=0.07906, simple_loss=0.09973, pruned_loss=0.01901, audio_tagging_loss=0.01018, over 3054904.34 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:29:58,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.02 vs. limit=15.0
2023-11-20 17:29:59,527 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173100
2023-11-20 17:30:11,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1154000.0, ans=0.125
2023-11-20 17:30:19,809 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.52 vs. limit=10.0
2023-11-20 17:30:28,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1154133.3333333333, ans=0.0
2023-11-20 17:30:34,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1154133.3333333333, ans=0.125
2023-11-20 17:30:42,281 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4800, loss[loss=0.054, simple_loss=0.06139, pruned_loss=0.01323, audio_tagging_loss=0.01007, over 15281.00 frames. ], tot_loss[loss=0.0789, simple_loss=0.09952, pruned_loss=0.01894, audio_tagging_loss=0.0102, over 3057004.84 frames. ], batch size: 60, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:30:47,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1154200.0, ans=0.125
2023-11-20 17:30:55,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1154266.6666666667, ans=0.125
2023-11-20 17:30:59,594 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.494e+01 7.777e+01 8.468e+01 9.403e+01 1.279e+02, threshold=1.694e+02, percent-clipped=0.0
2023-11-20 17:31:04,467 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173150
2023-11-20 17:31:08,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1154333.3333333333, ans=0.125
2023-11-20 17:31:38,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1154466.6666666667, ans=0.125
2023-11-20 17:31:47,286 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4850, loss[loss=0.08771, simple_loss=0.1058, pruned_loss=0.02291, audio_tagging_loss=0.01189, over 16134.00 frames. ], tot_loss[loss=0.07848, simple_loss=0.09899, pruned_loss=0.0187, audio_tagging_loss=0.01029, over 3056819.53 frames. ], batch size: 60, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:31:57,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1154533.3333333333, ans=0.0
2023-11-20 17:31:58,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1154600.0, ans=0.1
2023-11-20 17:32:07,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1154600.0, ans=0.125
2023-11-20 17:32:11,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173200
2023-11-20 17:32:44,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1154800.0, ans=0.0
2023-11-20 17:32:44,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.18 vs. limit=10.0
2023-11-20 17:32:51,428 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4900, loss[loss=0.08915, simple_loss=0.1092, pruned_loss=0.02668, audio_tagging_loss=0.007882, over 15742.00 frames. ], tot_loss[loss=0.07898, simple_loss=0.1, pruned_loss=0.01881, audio_tagging_loss=0.01014, over 3060586.52 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:33:04,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1154933.3333333333, ans=0.125
2023-11-20 17:33:09,827 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.588e+01 8.105e+01 8.801e+01 9.548e+01 1.347e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-20 17:33:14,807 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173250
2023-11-20 17:33:19,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1155000.0, ans=0.1
2023-11-20 17:33:55,031 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 4950, loss[loss=0.1013, simple_loss=0.1363, pruned_loss=0.02836, audio_tagging_loss=0.004788, over 15759.00 frames. ], tot_loss[loss=0.07897, simple_loss=0.1004, pruned_loss=0.01885, audio_tagging_loss=0.009918, over 3055417.18 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:34:02,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1155200.0, ans=0.125
2023-11-20 17:34:02,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1155200.0, ans=0.2
2023-11-20 17:34:16,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173300
2023-11-20 17:34:28,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1155333.3333333333, ans=0.125
2023-11-20 17:34:39,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1155400.0, ans=0.07
2023-11-20 17:34:46,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1155466.6666666667, ans=0.0
2023-11-20 17:34:46,931 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.925e-02
2023-11-20 17:34:57,663 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5000, loss[loss=0.0629, simple_loss=0.07992, pruned_loss=0.0122, audio_tagging_loss=0.01074, over 15842.00 frames. ], tot_loss[loss=0.07816, simple_loss=0.09943, pruned_loss=0.01858, audio_tagging_loss=0.009868, over 3055013.50 frames. ], batch size: 60, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:34:59,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1155533.3333333333, ans=0.0
2023-11-20 17:35:05,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1155533.3333333333, ans=0.2
2023-11-20 17:35:16,434 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.798e+01 8.227e+01 8.824e+01 9.880e+01 1.350e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 17:35:20,296 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173350
2023-11-20 17:35:38,542 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=7.935e-03
2023-11-20 17:35:38,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1155733.3333333333, ans=0.125
2023-11-20 17:35:57,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.01 vs. limit=15.0
2023-11-20 17:35:59,936 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5050, loss[loss=0.08412, simple_loss=0.1067, pruned_loss=0.022, audio_tagging_loss=0.008765, over 15157.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.1001, pruned_loss=0.01865, audio_tagging_loss=0.009772, over 3056074.58 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:36:01,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1155866.6666666667, ans=0.125
2023-11-20 17:36:21,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1155933.3333333333, ans=0.1
2023-11-20 17:36:23,936 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173400
2023-11-20 17:36:29,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1156000.0, ans=0.0
2023-11-20 17:37:04,834 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5100, loss[loss=0.09232, simple_loss=0.1191, pruned_loss=0.02117, audio_tagging_loss=0.01161, over 14500.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.0999, pruned_loss=0.01863, audio_tagging_loss=0.009644, over 3053815.95 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:37:19,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1156266.6666666667, ans=0.2
2023-11-20 17:37:23,037 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.188e+01 8.649e+01 9.367e+01 2.807e+02, threshold=1.730e+02, percent-clipped=1.0
2023-11-20 17:37:26,795 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173450
2023-11-20 17:37:31,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1156333.3333333333, ans=0.125
2023-11-20 17:37:32,262 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.24 vs. limit=15.0
2023-11-20 17:37:59,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1156466.6666666667, ans=0.1
2023-11-20 17:38:01,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1156466.6666666667, ans=0.125
2023-11-20 17:38:07,894 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5150, loss[loss=0.08269, simple_loss=0.1018, pruned_loss=0.02145, audio_tagging_loss=0.01036, over 15554.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09991, pruned_loss=0.01859, audio_tagging_loss=0.009731, over 3053511.79 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:38:15,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1156533.3333333333, ans=0.0
2023-11-20 17:38:30,377 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173500
2023-11-20 17:38:35,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.14 vs. limit=15.0
2023-11-20 17:38:55,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1156733.3333333333, ans=0.05
2023-11-20 17:38:57,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1156800.0, ans=0.1
2023-11-20 17:38:58,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1156800.0, ans=0.125
2023-11-20 17:39:10,953 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5200, loss[loss=0.08228, simple_loss=0.1008, pruned_loss=0.01935, audio_tagging_loss=0.01252, over 14522.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09998, pruned_loss=0.01864, audio_tagging_loss=0.009644, over 3048684.57 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:39:13,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1156866.6666666667, ans=0.125
2023-11-20 17:39:15,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1156866.6666666667, ans=10.0
2023-11-20 17:39:16,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.27 vs. limit=6.0
2023-11-20 17:39:26,163 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.48 vs. limit=15.0
2023-11-20 17:39:31,491 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.205e+01 8.715e+01 9.461e+01 1.258e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 17:39:35,254 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173550
2023-11-20 17:39:38,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1157000.0, ans=0.1
2023-11-20 17:39:39,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1157000.0, ans=0.125
2023-11-20 17:39:41,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1157000.0, ans=0.125
2023-11-20 17:39:48,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1157066.6666666667, ans=0.125
2023-11-20 17:39:52,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1157066.6666666667, ans=0.125
2023-11-20 17:40:04,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1157133.3333333333, ans=0.125
2023-11-20 17:40:14,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1157200.0, ans=0.1
2023-11-20 17:40:15,238 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5250, loss[loss=0.09964, simple_loss=0.1303, pruned_loss=0.02639, audio_tagging_loss=0.008126, over 14843.00 frames. ], tot_loss[loss=0.07858, simple_loss=0.09996, pruned_loss=0.01887, audio_tagging_loss=0.009727, over 3056561.77 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:40:25,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1157200.0, ans=0.125
2023-11-20 17:40:36,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1157266.6666666667, ans=0.125
2023-11-20 17:40:38,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173600
2023-11-20 17:40:54,036 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.39 vs. limit=15.0
2023-11-20 17:41:16,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=1157466.6666666667, ans=0.1
2023-11-20 17:41:17,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.61 vs. limit=10.0
2023-11-20 17:41:19,890 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5300, loss[loss=0.06421, simple_loss=0.0795, pruned_loss=0.01363, audio_tagging_loss=0.01082, over 14833.00 frames. ], tot_loss[loss=0.07947, simple_loss=0.1009, pruned_loss=0.01931, audio_tagging_loss=0.009696, over 3050580.75 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:41:33,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1157600.0, ans=0.125
2023-11-20 17:41:37,919 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.138e+01 8.869e+01 9.897e+01 2.566e+02, threshold=1.774e+02, percent-clipped=1.0
2023-11-20 17:41:41,710 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173650
2023-11-20 17:41:44,197 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.06 vs. limit=15.0
2023-11-20 17:41:49,676 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.96 vs. limit=22.5
2023-11-20 17:41:51,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.58 vs. limit=22.5
2023-11-20 17:41:55,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1157666.6666666667, ans=0.0
2023-11-20 17:42:22,886 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5350, loss[loss=0.1085, simple_loss=0.1332, pruned_loss=0.03173, audio_tagging_loss=0.01022, over 15505.00 frames. ], tot_loss[loss=0.08023, simple_loss=0.1021, pruned_loss=0.01951, audio_tagging_loss=0.009691, over 3051396.94 frames. ], batch size: 54, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:42:46,548 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173700
2023-11-20 17:42:50,639 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.08 vs. limit=12.0
2023-11-20 17:43:11,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1158066.6666666667, ans=0.04949747468305833
2023-11-20 17:43:13,239 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.51 vs. limit=10.0
2023-11-20 17:43:26,669 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5400, loss[loss=0.1171, simple_loss=0.1502, pruned_loss=0.0336, audio_tagging_loss=0.008397, over 16483.00 frames. ], tot_loss[loss=0.08053, simple_loss=0.1025, pruned_loss=0.01959, audio_tagging_loss=0.009709, over 3054949.57 frames. ], batch size: 58, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:43:29,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1158200.0, ans=0.0
2023-11-20 17:43:45,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1158266.6666666667, ans=0.125
2023-11-20 17:43:46,148 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 7.982e+01 8.687e+01 9.392e+01 1.840e+02, threshold=1.737e+02, percent-clipped=1.0
2023-11-20 17:43:49,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173750
2023-11-20 17:43:56,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1158333.3333333333, ans=0.125
2023-11-20 17:44:19,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1158466.6666666667, ans=0.125
2023-11-20 17:44:30,517 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5450, loss[loss=0.1097, simple_loss=0.1508, pruned_loss=0.0272, audio_tagging_loss=0.007122, over 15726.00 frames. ], tot_loss[loss=0.08108, simple_loss=0.1033, pruned_loss=0.01977, audio_tagging_loss=0.009671, over 3059043.11 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:44:30,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1158533.3333333333, ans=0.125
2023-11-20 17:44:51,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=12.61 vs. limit=15.0
2023-11-20 17:44:53,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173800
2023-11-20 17:45:08,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.65 vs. limit=6.0
2023-11-20 17:45:24,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1158800.0, ans=0.0
2023-11-20 17:45:30,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1158800.0, ans=0.0
2023-11-20 17:45:34,279 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5500, loss[loss=0.07204, simple_loss=0.08793, pruned_loss=0.01668, audio_tagging_loss=0.0114, over 13765.00 frames. ], tot_loss[loss=0.08036, simple_loss=0.1022, pruned_loss=0.01941, audio_tagging_loss=0.00983, over 3048755.15 frames. ], batch size: 54, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:45:37,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1158866.6666666667, ans=0.125
2023-11-20 17:45:54,019 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.155e+01 8.582e+01 9.481e+01 1.342e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-20 17:45:57,155 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173850
2023-11-20 17:46:30,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1159133.3333333333, ans=0.125
2023-11-20 17:46:37,035 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5550, loss[loss=0.0821, simple_loss=0.1102, pruned_loss=0.01784, audio_tagging_loss=0.009153, over 14690.00 frames. ], tot_loss[loss=0.0804, simple_loss=0.1022, pruned_loss=0.0194, audio_tagging_loss=0.009908, over 3040501.19 frames. ], batch size: 58, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:46:46,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1159200.0, ans=0.125
2023-11-20 17:46:59,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173900
2023-11-20 17:47:04,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=12.0
2023-11-20 17:47:05,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1159333.3333333333, ans=0.125
2023-11-20 17:47:07,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1159333.3333333333, ans=0.125
2023-11-20 17:47:40,116 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5600, loss[loss=0.06541, simple_loss=0.08706, pruned_loss=0.01266, audio_tagging_loss=0.009222, over 14509.00 frames. ], tot_loss[loss=0.07991, simple_loss=0.1013, pruned_loss=0.01918, audio_tagging_loss=0.01006, over 3038888.68 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:47:40,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1159533.3333333333, ans=0.125
2023-11-20 17:47:49,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1159533.3333333333, ans=0.125
2023-11-20 17:47:54,887 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=14.76 vs. limit=15.0
2023-11-20 17:48:00,192 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 7.988e+01 8.538e+01 9.609e+01 1.592e+02, threshold=1.708e+02, percent-clipped=0.0
2023-11-20 17:48:00,735 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=12.0
2023-11-20 17:48:02,723 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 173950
2023-11-20 17:48:23,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.58 vs. limit=12.0
2023-11-20 17:48:24,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1159733.3333333333, ans=0.0
2023-11-20 17:48:26,232 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:48:32,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1159800.0, ans=0.0
2023-11-20 17:48:36,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1159800.0, ans=0.1
2023-11-20 17:48:37,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1159800.0, ans=0.0
2023-11-20 17:48:39,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1159800.0, ans=0.125
2023-11-20 17:48:43,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1159866.6666666667, ans=0.025
2023-11-20 17:48:44,018 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5650, loss[loss=0.07486, simple_loss=0.0855, pruned_loss=0.02067, audio_tagging_loss=0.01144, over 14804.00 frames. ], tot_loss[loss=0.07976, simple_loss=0.1008, pruned_loss=0.01916, audio_tagging_loss=0.01019, over 3040702.03 frames. ], batch size: 59, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:48:44,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1159866.6666666667, ans=0.125
2023-11-20 17:48:45,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1159866.6666666667, ans=0.125
2023-11-20 17:49:07,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174000
2023-11-20 17:49:08,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1159933.3333333333, ans=10.0
2023-11-20 17:49:20,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1160000.0, ans=0.2
2023-11-20 17:49:48,576 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5700, loss[loss=0.13, simple_loss=0.1633, pruned_loss=0.04091, audio_tagging_loss=0.007452, over 16064.00 frames. ], tot_loss[loss=0.0803, simple_loss=0.1012, pruned_loss=0.01954, audio_tagging_loss=0.01018, over 3040689.52 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:49:51,658 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:50:03,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1160266.6666666667, ans=0.0
2023-11-20 17:50:10,030 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.368e+01 9.255e+01 1.006e+02 1.511e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-20 17:50:11,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174050
2023-11-20 17:50:24,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1160333.3333333333, ans=0.125
2023-11-20 17:50:28,483 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.16 vs. limit=15.0
2023-11-20 17:50:31,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.97 vs. limit=12.0
2023-11-20 17:50:33,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1160400.0, ans=0.0
2023-11-20 17:50:34,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1160400.0, ans=0.125
2023-11-20 17:50:52,581 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5750, loss[loss=0.0781, simple_loss=0.1025, pruned_loss=0.01927, audio_tagging_loss=0.007573, over 14846.00 frames. ], tot_loss[loss=0.08041, simple_loss=0.1016, pruned_loss=0.01962, audio_tagging_loss=0.009991, over 3042258.23 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:51:15,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174100
2023-11-20 17:51:21,437 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:51:35,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.00 vs. limit=15.0
2023-11-20 17:51:55,630 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5800, loss[loss=0.1221, simple_loss=0.1506, pruned_loss=0.03799, audio_tagging_loss=0.008813, over 15493.00 frames. ], tot_loss[loss=0.07954, simple_loss=0.1005, pruned_loss=0.01935, audio_tagging_loss=0.009935, over 3041107.63 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:52:01,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.44 vs. limit=6.0
2023-11-20 17:52:07,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1160933.3333333333, ans=0.2
2023-11-20 17:52:17,735 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.484e+01 8.088e+01 8.646e+01 9.359e+01 1.315e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-20 17:52:19,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174150
2023-11-20 17:52:58,989 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5850, loss[loss=0.07376, simple_loss=0.0867, pruned_loss=0.0198, audio_tagging_loss=0.01061, over 15134.00 frames. ], tot_loss[loss=0.07908, simple_loss=0.1001, pruned_loss=0.01917, audio_tagging_loss=0.009841, over 3037751.32 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:53:15,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1161266.6666666667, ans=0.07
2023-11-20 17:53:16,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1161266.6666666667, ans=0.1
2023-11-20 17:53:22,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174200
2023-11-20 17:53:30,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1161333.3333333333, ans=0.0
2023-11-20 17:53:32,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1161333.3333333333, ans=0.125
2023-11-20 17:54:03,770 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5900, loss[loss=0.08401, simple_loss=0.108, pruned_loss=0.02029, audio_tagging_loss=0.009698, over 14842.00 frames. ], tot_loss[loss=0.07941, simple_loss=0.1009, pruned_loss=0.01927, audio_tagging_loss=0.009713, over 3035981.23 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:54:08,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1161533.3333333333, ans=0.0
2023-11-20 17:54:18,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1161600.0, ans=0.0
2023-11-20 17:54:24,231 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.636e+01 8.417e+01 8.980e+01 9.978e+01 1.286e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-20 17:54:25,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174250
2023-11-20 17:54:32,793 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.22 vs. limit=15.0
2023-11-20 17:54:34,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1161666.6666666667, ans=0.04949747468305833
2023-11-20 17:54:48,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1161733.3333333333, ans=0.2
2023-11-20 17:54:52,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1161733.3333333333, ans=0.125
2023-11-20 17:54:52,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1161733.3333333333, ans=0.125
2023-11-20 17:55:06,756 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 5950, loss[loss=0.06556, simple_loss=0.08253, pruned_loss=0.01452, audio_tagging_loss=0.009785, over 15672.00 frames. ], tot_loss[loss=0.07936, simple_loss=0.101, pruned_loss=0.01924, audio_tagging_loss=0.00964, over 3038608.72 frames. ], batch size: 60, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:55:17,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1161866.6666666667, ans=0.125
2023-11-20 17:55:23,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1161933.3333333333, ans=0.125
2023-11-20 17:55:30,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174300
2023-11-20 17:55:33,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1162000.0, ans=0.1
2023-11-20 17:56:00,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1162133.3333333333, ans=0.125
2023-11-20 17:56:02,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1162133.3333333333, ans=0.0
2023-11-20 17:56:03,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1162133.3333333333, ans=0.1
2023-11-20 17:56:07,542 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.44 vs. limit=6.0
2023-11-20 17:56:10,527 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6000, loss[loss=0.1202, simple_loss=0.1624, pruned_loss=0.03409, audio_tagging_loss=0.004925, over 16219.00 frames. ], tot_loss[loss=0.07961, simple_loss=0.1017, pruned_loss=0.01928, audio_tagging_loss=0.00947, over 3044304.82 frames. ], batch size: 56, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 17:56:10,527 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 17:56:51,559 INFO [train_asr.py:1253] (2/4) Epoch 15, validation: loss=0.06114, simple_loss=0.05327, pruned_loss=0.005599, audio_tagging_loss=0.02891, over 4681554.00 frames. 
2023-11-20 17:56:51,560 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 17:56:54,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1162200.0, ans=0.125
2023-11-20 17:57:12,189 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.279e+01 8.029e+01 8.706e+01 9.735e+01 1.152e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-20 17:57:13,542 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174350
2023-11-20 17:57:35,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1162400.0, ans=0.0
2023-11-20 17:57:38,288 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:57:55,668 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6050, loss[loss=0.1033, simple_loss=0.1401, pruned_loss=0.02666, audio_tagging_loss=0.0066, over 15661.00 frames. ], tot_loss[loss=0.07979, simple_loss=0.1021, pruned_loss=0.01935, audio_tagging_loss=0.009407, over 3051305.58 frames. ], batch size: 55, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 17:58:02,338 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.18 vs. limit=15.0
2023-11-20 17:58:19,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174400
2023-11-20 17:58:19,961 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.56 vs. limit=15.0
2023-11-20 17:58:35,274 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:58:39,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0
2023-11-20 17:58:45,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1162733.3333333333, ans=0.1
2023-11-20 17:58:51,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1162800.0, ans=0.0
2023-11-20 17:58:52,917 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=12.0
2023-11-20 17:58:59,424 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6100, loss[loss=0.1033, simple_loss=0.1286, pruned_loss=0.03138, audio_tagging_loss=0.007634, over 14583.00 frames. ], tot_loss[loss=0.07972, simple_loss=0.1019, pruned_loss=0.01926, audio_tagging_loss=0.009527, over 3055194.46 frames. ], batch size: 56, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 17:59:12,349 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.17 vs. limit=15.0
2023-11-20 17:59:21,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1162933.3333333333, ans=0.125
2023-11-20 17:59:22,140 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 7.952e+01 8.424e+01 9.114e+01 1.496e+02, threshold=1.685e+02, percent-clipped=0.0
2023-11-20 17:59:23,496 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174450
2023-11-20 17:59:23,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1162933.3333333333, ans=0.07
2023-11-20 18:00:04,773 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6150, loss[loss=0.07611, simple_loss=0.09308, pruned_loss=0.01882, audio_tagging_loss=0.01074, over 14855.00 frames. ], tot_loss[loss=0.07889, simple_loss=0.1007, pruned_loss=0.01896, audio_tagging_loss=0.009601, over 3054895.86 frames. ], batch size: 57, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:00:10,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.02 vs. limit=15.0
2023-11-20 18:00:27,087 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174500
2023-11-20 18:01:09,261 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6200, loss[loss=0.07872, simple_loss=0.1011, pruned_loss=0.01595, audio_tagging_loss=0.01224, over 14877.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.0987, pruned_loss=0.01847, audio_tagging_loss=0.009762, over 3055407.86 frames. ], batch size: 53, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:01:09,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1163533.3333333333, ans=0.125
2023-11-20 18:01:18,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1163533.3333333333, ans=0.0
2023-11-20 18:01:21,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0
2023-11-20 18:01:25,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1163600.0, ans=0.0
2023-11-20 18:01:27,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1163600.0, ans=0.125
2023-11-20 18:01:33,408 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 8.002e+01 8.659e+01 9.317e+01 2.710e+02, threshold=1.732e+02, percent-clipped=1.0
2023-11-20 18:01:33,554 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174550
2023-11-20 18:01:38,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1163666.6666666667, ans=0.0
2023-11-20 18:02:01,596 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.86 vs. limit=22.5
2023-11-20 18:02:13,302 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6250, loss[loss=0.07903, simple_loss=0.09584, pruned_loss=0.0216, audio_tagging_loss=0.009503, over 13733.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.09898, pruned_loss=0.01869, audio_tagging_loss=0.009837, over 3051996.61 frames. ], batch size: 53, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:02:36,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1163933.3333333333, ans=0.1
2023-11-20 18:02:37,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174600
2023-11-20 18:02:52,169 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.48 vs. limit=6.0
2023-11-20 18:02:53,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1164066.6666666667, ans=0.0
2023-11-20 18:03:08,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1164133.3333333333, ans=0.0
2023-11-20 18:03:09,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1164133.3333333333, ans=15.0
2023-11-20 18:03:16,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1164133.3333333333, ans=0.0
2023-11-20 18:03:18,949 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6300, loss[loss=0.08887, simple_loss=0.1185, pruned_loss=0.01862, audio_tagging_loss=0.01101, over 15231.00 frames. ], tot_loss[loss=0.0784, simple_loss=0.09953, pruned_loss=0.01876, audio_tagging_loss=0.00987, over 3050378.10 frames. ], batch size: 53, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:03:41,342 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.283e+01 8.184e+01 8.938e+01 9.921e+01 1.399e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 18:03:41,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174650
2023-11-20 18:03:42,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1164333.3333333333, ans=0.125
2023-11-20 18:03:52,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.64 vs. limit=15.0
2023-11-20 18:04:12,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1164466.6666666667, ans=0.1
2023-11-20 18:04:16,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1164466.6666666667, ans=15.0
2023-11-20 18:04:17,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1164466.6666666667, ans=0.1
2023-11-20 18:04:23,033 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6350, loss[loss=0.07186, simple_loss=0.08723, pruned_loss=0.01579, audio_tagging_loss=0.01245, over 16709.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09887, pruned_loss=0.01851, audio_tagging_loss=0.009982, over 3042093.39 frames. ], batch size: 62, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:04:28,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1164533.3333333333, ans=0.07
2023-11-20 18:04:33,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1164533.3333333333, ans=0.1
2023-11-20 18:04:44,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1164600.0, ans=0.125
2023-11-20 18:04:46,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174700
2023-11-20 18:05:21,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1164800.0, ans=0.0
2023-11-20 18:05:26,443 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6400, loss[loss=0.08528, simple_loss=0.09535, pruned_loss=0.02506, audio_tagging_loss=0.01255, over 15957.00 frames. ], tot_loss[loss=0.07816, simple_loss=0.09899, pruned_loss=0.01856, audio_tagging_loss=0.0101, over 3035063.05 frames. ], batch size: 60, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 18:05:49,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1164933.3333333333, ans=0.2
2023-11-20 18:05:50,738 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.152e+01 8.050e+01 8.682e+01 9.459e+01 1.192e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-20 18:05:50,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174750
2023-11-20 18:06:00,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1165000.0, ans=0.1
2023-11-20 18:06:30,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1165200.0, ans=0.125
2023-11-20 18:06:30,858 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.71 vs. limit=15.0
2023-11-20 18:06:31,324 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6450, loss[loss=0.0735, simple_loss=0.08895, pruned_loss=0.01663, audio_tagging_loss=0.01239, over 15958.00 frames. ], tot_loss[loss=0.07767, simple_loss=0.09819, pruned_loss=0.01834, audio_tagging_loss=0.01024, over 3032697.62 frames. ], batch size: 58, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 18:06:38,195 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.41 vs. limit=6.0
2023-11-20 18:06:52,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1165266.6666666667, ans=0.125
2023-11-20 18:06:54,857 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174800
2023-11-20 18:07:05,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.72 vs. limit=15.0
2023-11-20 18:07:21,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1165400.0, ans=0.125
2023-11-20 18:07:36,642 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6500, loss[loss=0.06615, simple_loss=0.09156, pruned_loss=0.01135, audio_tagging_loss=0.009021, over 15926.00 frames. ], tot_loss[loss=0.07803, simple_loss=0.09876, pruned_loss=0.0185, audio_tagging_loss=0.01016, over 3043783.60 frames. ], batch size: 59, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 18:07:45,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1165533.3333333333, ans=0.125
2023-11-20 18:07:57,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1165600.0, ans=0.1
2023-11-20 18:07:58,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174850
2023-11-20 18:08:00,208 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 7.879e+01 8.698e+01 9.570e+01 1.330e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-20 18:08:23,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1165733.3333333333, ans=0.125
2023-11-20 18:08:40,139 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6550, loss[loss=0.08017, simple_loss=0.1078, pruned_loss=0.01884, audio_tagging_loss=0.007405, over 15462.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09823, pruned_loss=0.01858, audio_tagging_loss=0.01005, over 3039879.43 frames. ], batch size: 57, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:09:03,920 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174900
2023-11-20 18:09:28,569 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.40 vs. limit=15.0
2023-11-20 18:09:35,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1166133.3333333333, ans=0.125
2023-11-20 18:09:44,714 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6600, loss[loss=0.0888, simple_loss=0.1105, pruned_loss=0.02482, audio_tagging_loss=0.008746, over 14746.00 frames. ], tot_loss[loss=0.07812, simple_loss=0.09897, pruned_loss=0.01862, audio_tagging_loss=0.01002, over 3038096.02 frames. ], batch size: 55, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:10:08,389 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 174950
2023-11-20 18:10:09,438 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.156e+01 8.016e+01 8.893e+01 9.402e+01 1.270e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 18:10:09,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1166333.3333333333, ans=0.0
2023-11-20 18:10:26,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.92 vs. limit=15.0
2023-11-20 18:10:26,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1166400.0, ans=0.1
2023-11-20 18:10:31,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1166400.0, ans=0.0
2023-11-20 18:10:48,685 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6650, loss[loss=0.0636, simple_loss=0.08173, pruned_loss=0.01275, audio_tagging_loss=0.00999, over 16626.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09799, pruned_loss=0.0185, audio_tagging_loss=0.009958, over 3039278.62 frames. ], batch size: 61, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:11:02,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1166600.0, ans=0.0
2023-11-20 18:11:10,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1166600.0, ans=0.1
2023-11-20 18:11:11,342 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175000
2023-11-20 18:11:28,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1166733.3333333333, ans=0.0
2023-11-20 18:11:53,079 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6700, loss[loss=0.05244, simple_loss=0.05407, pruned_loss=0.01372, audio_tagging_loss=0.01169, over 14395.00 frames. ], tot_loss[loss=0.07832, simple_loss=0.09931, pruned_loss=0.01882, audio_tagging_loss=0.009848, over 3049227.09 frames. ], batch size: 55, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:11:53,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.54 vs. limit=15.0
2023-11-20 18:11:57,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1166866.6666666667, ans=0.125
2023-11-20 18:12:03,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1166866.6666666667, ans=0.1
2023-11-20 18:12:12,621 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.25 vs. limit=15.0
2023-11-20 18:12:16,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175050
2023-11-20 18:12:17,135 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.109e+01 8.688e+01 9.322e+01 1.481e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 18:12:36,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1167066.6666666667, ans=0.1
2023-11-20 18:12:49,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1167133.3333333333, ans=0.0
2023-11-20 18:12:57,099 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6750, loss[loss=0.06527, simple_loss=0.0932, pruned_loss=0.00957, audio_tagging_loss=0.009099, over 14473.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.09944, pruned_loss=0.01906, audio_tagging_loss=0.009843, over 3041494.51 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:13:10,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1167266.6666666667, ans=0.0
2023-11-20 18:13:16,766 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:13:20,209 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175100
2023-11-20 18:13:21,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1167333.3333333333, ans=0.0
2023-11-20 18:13:23,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1167333.3333333333, ans=0.0
2023-11-20 18:14:01,467 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6800, loss[loss=0.05266, simple_loss=0.05705, pruned_loss=0.0103, audio_tagging_loss=0.01384, over 14207.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09898, pruned_loss=0.01905, audio_tagging_loss=0.009826, over 3034304.51 frames. ], batch size: 54, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:14:24,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175150
2023-11-20 18:14:24,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1167600.0, ans=0.125
2023-11-20 18:14:25,239 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.115e+01 8.227e+01 9.120e+01 1.003e+02 1.352e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-20 18:14:30,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.10 vs. limit=22.5
2023-11-20 18:14:46,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.34 vs. limit=15.0
2023-11-20 18:14:46,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1167733.3333333333, ans=0.0
2023-11-20 18:15:05,531 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6850, loss[loss=0.08573, simple_loss=0.1159, pruned_loss=0.01874, audio_tagging_loss=0.009015, over 15123.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.09859, pruned_loss=0.01895, audio_tagging_loss=0.009794, over 3037315.97 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:15:09,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1167866.6666666667, ans=0.0
2023-11-20 18:15:10,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1167866.6666666667, ans=0.125
2023-11-20 18:15:16,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1167933.3333333333, ans=0.125
2023-11-20 18:15:27,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1167933.3333333333, ans=0.125
2023-11-20 18:15:28,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175200
2023-11-20 18:15:45,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1168066.6666666667, ans=0.125
2023-11-20 18:15:59,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1168133.3333333333, ans=0.125
2023-11-20 18:16:01,523 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.82 vs. limit=15.0
2023-11-20 18:16:03,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1168133.3333333333, ans=0.125
2023-11-20 18:16:04,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1168133.3333333333, ans=0.0
2023-11-20 18:16:04,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1168133.3333333333, ans=0.0
2023-11-20 18:16:06,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1168133.3333333333, ans=0.0
2023-11-20 18:16:10,192 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6900, loss[loss=0.06884, simple_loss=0.09075, pruned_loss=0.0123, audio_tagging_loss=0.01117, over 15744.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09881, pruned_loss=0.01871, audio_tagging_loss=0.009696, over 3039621.32 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:16:23,089 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.98 vs. limit=22.5
2023-11-20 18:16:25,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1168266.6666666667, ans=0.0
2023-11-20 18:16:28,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1168266.6666666667, ans=0.2
2023-11-20 18:16:33,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175250
2023-11-20 18:16:34,531 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.834e+01 8.085e+01 8.884e+01 9.597e+01 1.266e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-20 18:16:47,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1168400.0, ans=0.125
2023-11-20 18:17:00,896 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:17:06,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1168466.6666666667, ans=0.125
2023-11-20 18:17:07,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1168466.6666666667, ans=0.125
2023-11-20 18:17:14,810 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 6950, loss[loss=0.07606, simple_loss=0.09801, pruned_loss=0.01662, audio_tagging_loss=0.01044, over 15346.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.099, pruned_loss=0.01876, audio_tagging_loss=0.009692, over 3035449.60 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:17:21,698 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.19 vs. limit=15.0
2023-11-20 18:17:27,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1168600.0, ans=0.125
2023-11-20 18:17:31,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1168600.0, ans=0.1
2023-11-20 18:17:36,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.82 vs. limit=15.0
2023-11-20 18:17:37,502 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175300
2023-11-20 18:18:01,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.34 vs. limit=22.5
2023-11-20 18:18:11,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1168800.0, ans=0.125
2023-11-20 18:18:12,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1168800.0, ans=0.2
2023-11-20 18:18:18,358 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7000, loss[loss=0.08361, simple_loss=0.1048, pruned_loss=0.02061, audio_tagging_loss=0.01061, over 15111.00 frames. ], tot_loss[loss=0.07844, simple_loss=0.09973, pruned_loss=0.01888, audio_tagging_loss=0.009692, over 3029697.07 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:18:21,033 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:18:41,571 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175350
2023-11-20 18:18:42,698 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 7.873e+01 8.691e+01 9.185e+01 1.165e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 18:18:53,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.19 vs. limit=15.0
2023-11-20 18:18:53,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.51 vs. limit=15.0
2023-11-20 18:18:55,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1169066.6666666667, ans=0.125
2023-11-20 18:19:11,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1169133.3333333333, ans=0.0
2023-11-20 18:19:11,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1169133.3333333333, ans=0.09899494936611666
2023-11-20 18:19:13,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1169133.3333333333, ans=0.125
2023-11-20 18:19:17,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1169133.3333333333, ans=0.0
2023-11-20 18:19:19,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1169133.3333333333, ans=0.125
2023-11-20 18:19:21,839 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7050, loss[loss=0.07354, simple_loss=0.09561, pruned_loss=0.01765, audio_tagging_loss=0.008088, over 15186.00 frames. ], tot_loss[loss=0.07824, simple_loss=0.09938, pruned_loss=0.01885, audio_tagging_loss=0.0097, over 3031832.30 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:19:23,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1169200.0, ans=0.125
2023-11-20 18:19:28,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1169200.0, ans=0.125
2023-11-20 18:19:31,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1169200.0, ans=0.2
2023-11-20 18:19:34,121 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0
2023-11-20 18:19:37,327 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.16 vs. limit=15.0
2023-11-20 18:19:40,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1169266.6666666667, ans=0.0
2023-11-20 18:19:41,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1169266.6666666667, ans=0.125
2023-11-20 18:19:45,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175400
2023-11-20 18:20:26,360 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7100, loss[loss=0.07116, simple_loss=0.08492, pruned_loss=0.01803, audio_tagging_loss=0.01067, over 16451.00 frames. ], tot_loss[loss=0.07826, simple_loss=0.0992, pruned_loss=0.01882, audio_tagging_loss=0.009844, over 3031596.38 frames. ], batch size: 63, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:20:34,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1169533.3333333333, ans=0.2
2023-11-20 18:20:35,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1169533.3333333333, ans=0.2
2023-11-20 18:20:42,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1169600.0, ans=0.2
2023-11-20 18:20:44,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1169600.0, ans=0.05
2023-11-20 18:20:44,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1169600.0, ans=0.125
2023-11-20 18:20:48,699 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175450
2023-11-20 18:20:49,779 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.001e+01 8.747e+01 9.712e+01 1.225e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-20 18:21:01,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1169666.6666666667, ans=0.0
2023-11-20 18:21:06,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1169733.3333333333, ans=0.125
2023-11-20 18:21:06,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1169733.3333333333, ans=0.125
2023-11-20 18:21:07,864 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.19 vs. limit=10.0
2023-11-20 18:21:12,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1169733.3333333333, ans=0.07
2023-11-20 18:21:26,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1169800.0, ans=0.125
2023-11-20 18:21:29,444 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7150, loss[loss=0.08183, simple_loss=0.09382, pruned_loss=0.02107, audio_tagging_loss=0.01385, over 15331.00 frames. ], tot_loss[loss=0.07914, simple_loss=0.1004, pruned_loss=0.01912, audio_tagging_loss=0.009828, over 3034070.68 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:21:43,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1169933.3333333333, ans=0.1
2023-11-20 18:21:53,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175500
2023-11-20 18:22:14,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1170066.6666666667, ans=0.0
2023-11-20 18:22:22,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1170133.3333333333, ans=0.04949747468305833
2023-11-20 18:22:32,635 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7200, loss[loss=0.1081, simple_loss=0.1389, pruned_loss=0.02948, audio_tagging_loss=0.009169, over 15629.00 frames. ], tot_loss[loss=0.07921, simple_loss=0.1005, pruned_loss=0.01899, audio_tagging_loss=0.009956, over 3036791.84 frames. ], batch size: 55, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:22:41,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1170200.0, ans=0.125
2023-11-20 18:22:41,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1170200.0, ans=0.125
2023-11-20 18:22:45,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1170266.6666666667, ans=0.125
2023-11-20 18:22:54,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1170266.6666666667, ans=0.0
2023-11-20 18:22:56,698 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175550
2023-11-20 18:22:57,773 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.535e+01 8.191e+01 8.742e+01 9.688e+01 2.740e+02, threshold=1.748e+02, percent-clipped=1.0
2023-11-20 18:22:58,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0
2023-11-20 18:23:04,663 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.80 vs. limit=15.0
2023-11-20 18:23:37,151 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7250, loss[loss=0.0905, simple_loss=0.1217, pruned_loss=0.01927, audio_tagging_loss=0.01038, over 15503.00 frames. ], tot_loss[loss=0.07925, simple_loss=0.1006, pruned_loss=0.01894, audio_tagging_loss=0.01002, over 3036849.61 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:23:42,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1170533.3333333333, ans=0.125
2023-11-20 18:23:59,858 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175600
2023-11-20 18:24:11,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1170666.6666666667, ans=0.1
2023-11-20 18:24:12,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1170666.6666666667, ans=0.0
2023-11-20 18:24:24,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1170733.3333333333, ans=0.0
2023-11-20 18:24:31,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1170800.0, ans=0.035
2023-11-20 18:24:31,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1170800.0, ans=0.1
2023-11-20 18:24:34,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1170800.0, ans=0.0
2023-11-20 18:24:41,086 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7300, loss[loss=0.0817, simple_loss=0.1096, pruned_loss=0.01983, audio_tagging_loss=0.007066, over 14204.00 frames. ], tot_loss[loss=0.07906, simple_loss=0.1005, pruned_loss=0.01889, audio_tagging_loss=0.009941, over 3036996.25 frames. ], batch size: 54, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:24:58,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1170933.3333333333, ans=0.125
2023-11-20 18:24:59,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1170933.3333333333, ans=0.1
2023-11-20 18:25:03,247 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175650
2023-11-20 18:25:06,282 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 7.796e+01 8.638e+01 9.366e+01 1.171e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-20 18:25:10,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1171000.0, ans=0.5
2023-11-20 18:25:18,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1171066.6666666667, ans=0.0
2023-11-20 18:25:19,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1171066.6666666667, ans=0.2
2023-11-20 18:25:27,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1171066.6666666667, ans=0.125
2023-11-20 18:25:43,775 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7350, loss[loss=0.09429, simple_loss=0.1233, pruned_loss=0.02499, audio_tagging_loss=0.007666, over 16561.00 frames. ], tot_loss[loss=0.07898, simple_loss=0.1005, pruned_loss=0.01901, audio_tagging_loss=0.009747, over 3039751.45 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:26:07,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175700
2023-11-20 18:26:21,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1171400.0, ans=0.0
2023-11-20 18:26:38,236 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:26:42,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1171466.6666666667, ans=0.2
2023-11-20 18:26:47,855 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7400, loss[loss=0.06123, simple_loss=0.07181, pruned_loss=0.014, audio_tagging_loss=0.01132, over 14455.00 frames. ], tot_loss[loss=0.07872, simple_loss=0.09999, pruned_loss=0.01904, audio_tagging_loss=0.009688, over 3040264.90 frames. ], batch size: 55, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:27:06,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1171600.0, ans=0.125
2023-11-20 18:27:10,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175750
2023-11-20 18:27:12,661 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 8.125e+01 8.808e+01 9.818e+01 1.259e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-20 18:27:50,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1171866.6666666667, ans=0.125
2023-11-20 18:27:51,322 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7450, loss[loss=0.07321, simple_loss=0.09008, pruned_loss=0.01534, audio_tagging_loss=0.01283, over 15760.00 frames. ], tot_loss[loss=0.07882, simple_loss=0.1001, pruned_loss=0.01904, audio_tagging_loss=0.009713, over 3042883.34 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:28:02,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1171933.3333333333, ans=0.125
2023-11-20 18:28:05,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1171933.3333333333, ans=0.125
2023-11-20 18:28:07,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1171933.3333333333, ans=0.0
2023-11-20 18:28:12,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1171933.3333333333, ans=0.2
2023-11-20 18:28:13,564 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175800
2023-11-20 18:28:27,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1172000.0, ans=0.125
2023-11-20 18:28:37,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1172066.6666666667, ans=0.125
2023-11-20 18:28:47,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1172133.3333333333, ans=0.125
2023-11-20 18:28:48,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1172133.3333333333, ans=0.125
2023-11-20 18:28:54,505 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7500, loss[loss=0.07375, simple_loss=0.08494, pruned_loss=0.01911, audio_tagging_loss=0.01216, over 14640.00 frames. ], tot_loss[loss=0.07889, simple_loss=0.1002, pruned_loss=0.01911, audio_tagging_loss=0.00968, over 3037453.82 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:28:58,750 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:29:18,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1172266.6666666667, ans=0.125
2023-11-20 18:29:19,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175850
2023-11-20 18:29:21,646 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.057e+01 8.747e+01 9.687e+01 1.264e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-20 18:29:25,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1172333.3333333333, ans=0.125
2023-11-20 18:29:32,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1172400.0, ans=0.0
2023-11-20 18:29:36,816 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.34 vs. limit=15.0
2023-11-20 18:29:59,025 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7550, loss[loss=0.08473, simple_loss=0.1132, pruned_loss=0.01927, audio_tagging_loss=0.008881, over 14752.00 frames. ], tot_loss[loss=0.07803, simple_loss=0.09901, pruned_loss=0.0188, audio_tagging_loss=0.009727, over 3028822.23 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:30:08,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1172533.3333333333, ans=0.125
2023-11-20 18:30:09,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.08 vs. limit=15.0
2023-11-20 18:30:22,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175900
2023-11-20 18:30:55,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1172800.0, ans=0.125
2023-11-20 18:31:03,462 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7600, loss[loss=0.05991, simple_loss=0.07495, pruned_loss=0.01228, audio_tagging_loss=0.01015, over 14392.00 frames. ], tot_loss[loss=0.07727, simple_loss=0.09808, pruned_loss=0.01844, audio_tagging_loss=0.009793, over 3031419.00 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:31:03,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1172866.6666666667, ans=0.1
2023-11-20 18:31:05,282 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.35 vs. limit=22.5
2023-11-20 18:31:07,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-20 18:31:23,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1172933.3333333333, ans=0.0
2023-11-20 18:31:25,600 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 175950
2023-11-20 18:31:27,953 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.999e+01 7.835e+01 8.443e+01 9.077e+01 1.271e+02, threshold=1.689e+02, percent-clipped=0.0
2023-11-20 18:31:33,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1173000.0, ans=0.125
2023-11-20 18:32:03,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1173133.3333333333, ans=0.1
2023-11-20 18:32:06,912 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7650, loss[loss=0.05559, simple_loss=0.06044, pruned_loss=0.01437, audio_tagging_loss=0.011, over 13866.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09761, pruned_loss=0.01841, audio_tagging_loss=0.009807, over 3038771.76 frames. ], batch size: 53, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:32:10,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1173200.0, ans=0.1
2023-11-20 18:32:10,854 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=9.356e-02
2023-11-20 18:32:29,148 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:32:30,209 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176000
2023-11-20 18:32:41,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1173333.3333333333, ans=0.125
2023-11-20 18:32:50,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.56 vs. limit=15.0
2023-11-20 18:32:52,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.48 vs. limit=15.0
2023-11-20 18:33:14,108 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7700, loss[loss=0.0615, simple_loss=0.07981, pruned_loss=0.01226, audio_tagging_loss=0.009334, over 14822.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09801, pruned_loss=0.01829, audio_tagging_loss=0.009716, over 3037098.02 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:33:19,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1173533.3333333333, ans=0.0
2023-11-20 18:33:21,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1173533.3333333333, ans=0.2
2023-11-20 18:33:37,307 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176050
2023-11-20 18:33:39,689 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.684e+01 8.040e+01 8.573e+01 9.268e+01 1.195e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-20 18:33:39,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1173666.6666666667, ans=0.0
2023-11-20 18:33:40,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1173666.6666666667, ans=0.1
2023-11-20 18:33:48,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1173666.6666666667, ans=0.1
2023-11-20 18:33:53,634 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:33:54,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1173733.3333333333, ans=0.0
2023-11-20 18:33:54,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1173733.3333333333, ans=0.2
2023-11-20 18:34:10,993 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.70 vs. limit=10.0
2023-11-20 18:34:12,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=22.5
2023-11-20 18:34:13,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1173800.0, ans=0.1
2023-11-20 18:34:17,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1173866.6666666667, ans=0.0
2023-11-20 18:34:18,335 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7750, loss[loss=0.0818, simple_loss=0.1066, pruned_loss=0.01836, audio_tagging_loss=0.01017, over 14325.00 frames. ], tot_loss[loss=0.07784, simple_loss=0.09902, pruned_loss=0.01861, audio_tagging_loss=0.009721, over 3038191.61 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:34:41,045 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176100
2023-11-20 18:35:05,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1174066.6666666667, ans=0.015
2023-11-20 18:35:11,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1174133.3333333333, ans=0.0
2023-11-20 18:35:11,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1174133.3333333333, ans=0.0
2023-11-20 18:35:12,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1174133.3333333333, ans=0.125
2023-11-20 18:35:21,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.32 vs. limit=22.5
2023-11-20 18:35:22,306 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7800, loss[loss=0.07598, simple_loss=0.09276, pruned_loss=0.01595, audio_tagging_loss=0.01365, over 15663.00 frames. ], tot_loss[loss=0.07832, simple_loss=0.09936, pruned_loss=0.01885, audio_tagging_loss=0.009792, over 3037832.48 frames. ], batch size: 58, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:35:27,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1174200.0, ans=0.1
2023-11-20 18:35:45,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176150
2023-11-20 18:35:48,939 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.331e+01 9.045e+01 1.006e+02 1.273e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-20 18:36:09,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1174400.0, ans=0.125
2023-11-20 18:36:26,121 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7850, loss[loss=0.05772, simple_loss=0.06589, pruned_loss=0.0143, audio_tagging_loss=0.01047, over 15389.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09911, pruned_loss=0.01862, audio_tagging_loss=0.009758, over 3036104.78 frames. ], batch size: 59, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:36:35,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1174533.3333333333, ans=0.125
2023-11-20 18:36:35,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1174533.3333333333, ans=0.125
2023-11-20 18:36:35,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1174533.3333333333, ans=0.0
2023-11-20 18:36:35,837 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.79 vs. limit=15.0
2023-11-20 18:36:43,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1174600.0, ans=0.2
2023-11-20 18:36:48,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1174600.0, ans=10.0
2023-11-20 18:36:50,246 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176200
2023-11-20 18:36:50,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1174600.0, ans=0.125
2023-11-20 18:36:54,884 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=12.0
2023-11-20 18:37:23,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1174800.0, ans=0.125
2023-11-20 18:37:26,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1174800.0, ans=0.125
2023-11-20 18:37:30,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1174866.6666666667, ans=0.125
2023-11-20 18:37:31,663 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7900, loss[loss=0.05932, simple_loss=0.0712, pruned_loss=0.01428, audio_tagging_loss=0.009439, over 15062.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09888, pruned_loss=0.01838, audio_tagging_loss=0.009899, over 3036154.13 frames. ], batch size: 57, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:37:36,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1174866.6666666667, ans=0.125
2023-11-20 18:37:44,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1174933.3333333333, ans=0.125
2023-11-20 18:37:47,246 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:37:50,932 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:37:52,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1174933.3333333333, ans=0.0
2023-11-20 18:37:54,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176250
2023-11-20 18:37:57,980 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.028e+01 8.743e+01 9.651e+01 2.118e+02, threshold=1.749e+02, percent-clipped=1.0
2023-11-20 18:38:23,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0
2023-11-20 18:38:28,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1175133.3333333333, ans=0.0
2023-11-20 18:38:30,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1175133.3333333333, ans=0.0
2023-11-20 18:38:35,860 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 7950, loss[loss=0.07137, simple_loss=0.0992, pruned_loss=0.01352, audio_tagging_loss=0.008242, over 14930.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.09811, pruned_loss=0.0183, audio_tagging_loss=0.01007, over 3036809.94 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:38:51,832 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:38:59,238 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176300
2023-11-20 18:39:03,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1175333.3333333333, ans=0.125
2023-11-20 18:39:11,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1175333.3333333333, ans=0.125
2023-11-20 18:39:11,513 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.90 vs. limit=15.0
2023-11-20 18:39:16,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1175400.0, ans=0.0
2023-11-20 18:39:21,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1175400.0, ans=0.1
2023-11-20 18:39:35,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1175466.6666666667, ans=0.125
2023-11-20 18:39:39,807 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8000, loss[loss=0.08667, simple_loss=0.1112, pruned_loss=0.02048, audio_tagging_loss=0.01059, over 15812.00 frames. ], tot_loss[loss=0.07659, simple_loss=0.09657, pruned_loss=0.01801, audio_tagging_loss=0.0103, over 3038008.98 frames. ], batch size: 57, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:39:41,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1175533.3333333333, ans=0.2
2023-11-20 18:39:47,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.96 vs. limit=15.0
2023-11-20 18:39:53,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1175600.0, ans=0.2
2023-11-20 18:40:00,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.02 vs. limit=15.0
2023-11-20 18:40:03,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176350
2023-11-20 18:40:07,193 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.314e+01 8.360e+01 9.089e+01 9.769e+01 1.308e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-20 18:40:33,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1175800.0, ans=0.2
2023-11-20 18:40:38,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1175800.0, ans=0.025
2023-11-20 18:40:44,410 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8050, loss[loss=0.05924, simple_loss=0.06855, pruned_loss=0.01377, audio_tagging_loss=0.0112, over 14495.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.0966, pruned_loss=0.01803, audio_tagging_loss=0.01036, over 3046165.88 frames. ], batch size: 57, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:41:07,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176400
2023-11-20 18:41:49,092 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8100, loss[loss=0.08712, simple_loss=0.09127, pruned_loss=0.02579, audio_tagging_loss=0.01569, over 14501.00 frames. ], tot_loss[loss=0.07676, simple_loss=0.09681, pruned_loss=0.01815, audio_tagging_loss=0.01021, over 3033518.52 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:41:54,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1176200.0, ans=0.1
2023-11-20 18:41:55,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.37 vs. limit=15.0
2023-11-20 18:41:57,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1176200.0, ans=0.0
2023-11-20 18:42:12,980 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176450
2023-11-20 18:42:16,560 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 7.970e+01 8.520e+01 9.468e+01 1.287e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-20 18:42:18,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1176333.3333333333, ans=0.05
2023-11-20 18:42:36,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1176400.0, ans=0.125
2023-11-20 18:42:51,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1176466.6666666667, ans=0.0
2023-11-20 18:42:53,260 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8150, loss[loss=0.07039, simple_loss=0.08604, pruned_loss=0.01854, audio_tagging_loss=0.008829, over 15706.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.0973, pruned_loss=0.01843, audio_tagging_loss=0.01, over 3034558.96 frames. ], batch size: 59, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:42:53,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1176533.3333333333, ans=0.125
2023-11-20 18:43:15,136 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.89 vs. limit=15.0
2023-11-20 18:43:17,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176500
2023-11-20 18:43:27,216 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:43:51,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1176800.0, ans=0.2
2023-11-20 18:43:57,829 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8200, loss[loss=0.09426, simple_loss=0.1305, pruned_loss=0.02196, audio_tagging_loss=0.007029, over 15489.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09797, pruned_loss=0.0187, audio_tagging_loss=0.009922, over 3039513.57 frames. ], batch size: 54, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:43:59,130 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:44:03,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1176866.6666666667, ans=0.0
2023-11-20 18:44:14,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1176933.3333333333, ans=0.1
2023-11-20 18:44:19,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1176933.3333333333, ans=0.0
2023-11-20 18:44:20,503 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176550
2023-11-20 18:44:24,658 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.959e+01 8.657e+01 9.371e+01 1.307e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-20 18:44:25,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1177000.0, ans=0.1
2023-11-20 18:44:46,104 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.62 vs. limit=15.0
2023-11-20 18:44:58,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.26 vs. limit=15.0
2023-11-20 18:45:02,159 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8250, loss[loss=0.08899, simple_loss=0.1165, pruned_loss=0.02146, audio_tagging_loss=0.009295, over 15241.00 frames. ], tot_loss[loss=0.0769, simple_loss=0.09709, pruned_loss=0.01852, audio_tagging_loss=0.009833, over 3045360.32 frames. ], batch size: 54, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:45:04,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1177200.0, ans=0.1
2023-11-20 18:45:20,591 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.18 vs. limit=15.0
2023-11-20 18:45:25,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176600
2023-11-20 18:45:32,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.54 vs. limit=22.5
2023-11-20 18:45:41,362 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:45:43,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1177400.0, ans=0.1
2023-11-20 18:45:58,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1177466.6666666667, ans=0.0
2023-11-20 18:45:58,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1177466.6666666667, ans=0.125
2023-11-20 18:46:03,578 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.01 vs. limit=12.0
2023-11-20 18:46:06,157 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8300, loss[loss=0.07208, simple_loss=0.09722, pruned_loss=0.01454, audio_tagging_loss=0.008932, over 14517.00 frames. ], tot_loss[loss=0.07784, simple_loss=0.0985, pruned_loss=0.01884, audio_tagging_loss=0.009753, over 3043823.62 frames. ], batch size: 55, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:46:10,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1177533.3333333333, ans=0.125
2023-11-20 18:46:30,121 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176650
2023-11-20 18:46:33,813 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.522e+01 7.955e+01 8.600e+01 9.336e+01 1.507e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-20 18:46:35,636 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.10 vs. limit=15.0
2023-11-20 18:46:40,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1177666.6666666667, ans=0.035
2023-11-20 18:46:44,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1177733.3333333333, ans=0.125
2023-11-20 18:46:51,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1177733.3333333333, ans=0.0
2023-11-20 18:46:53,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1177733.3333333333, ans=0.125
2023-11-20 18:47:07,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1177800.0, ans=0.09899494936611666
2023-11-20 18:47:11,252 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8350, loss[loss=0.06502, simple_loss=0.07298, pruned_loss=0.0155, audio_tagging_loss=0.01303, over 14492.00 frames. ], tot_loss[loss=0.07841, simple_loss=0.09929, pruned_loss=0.01906, audio_tagging_loss=0.009696, over 3045753.77 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:47:23,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1177933.3333333333, ans=0.035
2023-11-20 18:47:26,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0
2023-11-20 18:47:29,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1177933.3333333333, ans=0.1
2023-11-20 18:47:33,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176700
2023-11-20 18:47:34,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1177933.3333333333, ans=0.0
2023-11-20 18:47:58,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1178066.6666666667, ans=0.0
2023-11-20 18:48:14,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1178200.0, ans=0.125
2023-11-20 18:48:15,696 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8400, loss[loss=0.06993, simple_loss=0.0881, pruned_loss=0.01755, audio_tagging_loss=0.008329, over 14638.00 frames. ], tot_loss[loss=0.07834, simple_loss=0.0994, pruned_loss=0.01903, audio_tagging_loss=0.009604, over 3047715.48 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:48:18,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1178200.0, ans=0.0
2023-11-20 18:48:24,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1178200.0, ans=0.1
2023-11-20 18:48:35,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1178266.6666666667, ans=0.0
2023-11-20 18:48:38,686 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176750
2023-11-20 18:48:42,871 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.237e+01 7.973e+01 8.686e+01 9.472e+01 1.183e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-20 18:48:58,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.15 vs. limit=22.5
2023-11-20 18:49:19,814 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8450, loss[loss=0.06854, simple_loss=0.08258, pruned_loss=0.01661, audio_tagging_loss=0.01064, over 14853.00 frames. ], tot_loss[loss=0.07756, simple_loss=0.0978, pruned_loss=0.01884, audio_tagging_loss=0.009811, over 3049689.67 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:49:21,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1178533.3333333333, ans=0.125
2023-11-20 18:49:21,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1178533.3333333333, ans=0.0
2023-11-20 18:49:34,057 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-20 18:49:43,445 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176800
2023-11-20 18:49:44,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1178600.0, ans=0.0
2023-11-20 18:50:01,145 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.26 vs. limit=15.0
2023-11-20 18:50:25,058 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8500, loss[loss=0.08173, simple_loss=0.1083, pruned_loss=0.01869, audio_tagging_loss=0.008913, over 15895.00 frames. ], tot_loss[loss=0.07773, simple_loss=0.0984, pruned_loss=0.01874, audio_tagging_loss=0.009796, over 3056836.31 frames. ], batch size: 58, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:50:35,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1178866.6666666667, ans=0.125
2023-11-20 18:50:47,870 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176850
2023-11-20 18:50:51,458 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.113e+01 8.136e+01 8.962e+01 9.902e+01 1.387e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-20 18:50:57,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=1179000.0, ans=0.02
2023-11-20 18:50:59,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1179000.0, ans=0.125
2023-11-20 18:51:12,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1179066.6666666667, ans=0.1
2023-11-20 18:51:25,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1179133.3333333333, ans=0.2
2023-11-20 18:51:28,683 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8550, loss[loss=0.06285, simple_loss=0.08032, pruned_loss=0.01377, audio_tagging_loss=0.008922, over 16007.00 frames. ], tot_loss[loss=0.07769, simple_loss=0.09852, pruned_loss=0.01864, audio_tagging_loss=0.009795, over 3055682.48 frames. ], batch size: 61, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:51:33,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=1179200.0, ans=15.0
2023-11-20 18:51:45,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1179266.6666666667, ans=0.125
2023-11-20 18:51:49,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1179266.6666666667, ans=0.125
2023-11-20 18:51:51,430 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176900
2023-11-20 18:52:15,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1179400.0, ans=0.0
2023-11-20 18:52:30,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1179466.6666666667, ans=0.125
2023-11-20 18:52:32,744 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8600, loss[loss=0.05966, simple_loss=0.07351, pruned_loss=0.01342, audio_tagging_loss=0.00948, over 15466.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09938, pruned_loss=0.01877, audio_tagging_loss=0.009825, over 3059195.33 frames. ], batch size: 58, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:52:35,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1179533.3333333333, ans=0.2
2023-11-20 18:52:39,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1179533.3333333333, ans=0.125
2023-11-20 18:52:44,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1179600.0, ans=0.1
2023-11-20 18:52:44,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1179600.0, ans=0.2
2023-11-20 18:52:52,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0
2023-11-20 18:52:56,713 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 176950
2023-11-20 18:53:00,850 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.645e+01 8.095e+01 8.772e+01 9.480e+01 1.996e+02, threshold=1.754e+02, percent-clipped=1.0
2023-11-20 18:53:12,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.90 vs. limit=22.5
2023-11-20 18:53:13,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1179733.3333333333, ans=0.125
2023-11-20 18:53:18,649 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.29 vs. limit=10.0
2023-11-20 18:53:33,410 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.71 vs. limit=15.0
2023-11-20 18:53:34,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1179800.0, ans=0.125
2023-11-20 18:53:37,675 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8650, loss[loss=0.08007, simple_loss=0.1126, pruned_loss=0.01472, audio_tagging_loss=0.009028, over 14902.00 frames. ], tot_loss[loss=0.07865, simple_loss=0.1, pruned_loss=0.01885, audio_tagging_loss=0.0098, over 3054773.34 frames. ], batch size: 54, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:53:52,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0
2023-11-20 18:54:01,261 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177000
2023-11-20 18:54:16,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1180066.6666666667, ans=10.0
2023-11-20 18:54:19,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1180066.6666666667, ans=0.125
2023-11-20 18:54:31,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=1180133.3333333333, ans=0.025
2023-11-20 18:54:40,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1180133.3333333333, ans=0.125
2023-11-20 18:54:43,915 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8700, loss[loss=0.07554, simple_loss=0.08963, pruned_loss=0.0187, audio_tagging_loss=0.01202, over 14979.00 frames. ], tot_loss[loss=0.07902, simple_loss=0.1006, pruned_loss=0.01884, audio_tagging_loss=0.009886, over 3060551.18 frames. ], batch size: 59, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:54:49,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1180200.0, ans=0.125
2023-11-20 18:55:06,336 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177050
2023-11-20 18:55:10,011 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.109e+01 8.720e+01 9.588e+01 1.618e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-20 18:55:10,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff2.min_abs, batch_count=1180333.3333333333, ans=0.1
2023-11-20 18:55:28,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1180400.0, ans=0.1
2023-11-20 18:55:30,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1180400.0, ans=0.2
2023-11-20 18:55:33,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1180400.0, ans=0.125
2023-11-20 18:55:45,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1180466.6666666667, ans=0.125
2023-11-20 18:55:47,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1180533.3333333333, ans=0.125
2023-11-20 18:55:48,033 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8750, loss[loss=0.07907, simple_loss=0.1001, pruned_loss=0.0201, audio_tagging_loss=0.00892, over 15103.00 frames. ], tot_loss[loss=0.07932, simple_loss=0.1009, pruned_loss=0.01892, audio_tagging_loss=0.009952, over 3060918.77 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 16.0
2023-11-20 18:55:55,869 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.25 vs. limit=15.0
2023-11-20 18:55:59,569 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.49 vs. limit=15.0
2023-11-20 18:56:11,310 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177100
2023-11-20 18:56:17,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1180666.6666666667, ans=0.1
2023-11-20 18:56:26,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.94 vs. limit=15.0
2023-11-20 18:56:47,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.00 vs. limit=15.0
2023-11-20 18:56:49,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0
2023-11-20 18:56:52,291 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8800, loss[loss=0.0791, simple_loss=0.09877, pruned_loss=0.02091, audio_tagging_loss=0.008798, over 15241.00 frames. ], tot_loss[loss=0.07982, simple_loss=0.1017, pruned_loss=0.01898, audio_tagging_loss=0.01, over 3056695.03 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:57:16,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177150
2023-11-20 18:57:21,354 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.135e+01 8.370e+01 9.044e+01 9.709e+01 1.257e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-20 18:57:22,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.40 vs. limit=15.0
2023-11-20 18:57:29,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1181000.0, ans=0.125
2023-11-20 18:57:42,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1181066.6666666667, ans=0.0
2023-11-20 18:57:45,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1181133.3333333333, ans=0.0
2023-11-20 18:57:49,335 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=12.0
2023-11-20 18:57:54,382 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.67 vs. limit=12.0
2023-11-20 18:57:56,022 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.03 vs. limit=22.5
2023-11-20 18:57:58,608 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8850, loss[loss=0.07764, simple_loss=0.09554, pruned_loss=0.01599, audio_tagging_loss=0.01388, over 15659.00 frames. ], tot_loss[loss=0.07952, simple_loss=0.1012, pruned_loss=0.01894, audio_tagging_loss=0.009978, over 3055797.22 frames. ], batch size: 58, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:57:58,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1181200.0, ans=0.125
2023-11-20 18:57:59,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1181200.0, ans=0.2
2023-11-20 18:58:03,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1181200.0, ans=0.125
2023-11-20 18:58:10,977 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:58:13,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1181266.6666666667, ans=0.0
2023-11-20 18:58:20,920 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177200
2023-11-20 18:58:38,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1181400.0, ans=0.0
2023-11-20 18:58:48,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1181400.0, ans=0.0
2023-11-20 18:59:01,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1181466.6666666667, ans=0.05
2023-11-20 18:59:01,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1181466.6666666667, ans=0.1
2023-11-20 18:59:03,527 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8900, loss[loss=0.07217, simple_loss=0.09695, pruned_loss=0.01352, audio_tagging_loss=0.01018, over 15219.00 frames. ], tot_loss[loss=0.0794, simple_loss=0.1013, pruned_loss=0.01896, audio_tagging_loss=0.009801, over 3051359.67 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:59:17,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1181600.0, ans=0.125
2023-11-20 18:59:21,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1181600.0, ans=0.2
2023-11-20 18:59:25,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1181600.0, ans=0.04949747468305833
2023-11-20 18:59:27,686 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177250
2023-11-20 18:59:32,817 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.317e+01 8.861e+01 9.855e+01 1.345e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-20 19:00:08,820 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 8950, loss[loss=0.09677, simple_loss=0.1249, pruned_loss=0.02632, audio_tagging_loss=0.008011, over 14742.00 frames. ], tot_loss[loss=0.07902, simple_loss=0.1007, pruned_loss=0.0189, audio_tagging_loss=0.009774, over 3052123.94 frames. ], batch size: 53, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 19:00:11,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1181866.6666666667, ans=0.0
2023-11-20 19:00:15,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1181866.6666666667, ans=0.125
2023-11-20 19:00:33,293 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177300
2023-11-20 19:00:51,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.13 vs. limit=15.0
2023-11-20 19:01:13,864 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9000, loss[loss=0.07536, simple_loss=0.104, pruned_loss=0.01643, audio_tagging_loss=0.006954, over 15005.00 frames. ], tot_loss[loss=0.07877, simple_loss=0.1005, pruned_loss=0.01892, audio_tagging_loss=0.009591, over 3050135.21 frames. ], batch size: 55, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 19:01:13,865 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 19:01:56,967 INFO [train_asr.py:1253] (2/4) Epoch 15, validation: loss=0.0619, simple_loss=0.05318, pruned_loss=0.005552, audio_tagging_loss=0.02975, over 4681554.00 frames. 
2023-11-20 19:01:56,967 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 19:01:57,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1182200.0, ans=0.1
2023-11-20 19:01:58,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1182200.0, ans=0.0
2023-11-20 19:02:20,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177350
2023-11-20 19:02:25,236 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.224e+01 8.889e+01 9.368e+01 1.234e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-20 19:02:32,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1182333.3333333333, ans=0.0
2023-11-20 19:02:32,693 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.91 vs. limit=15.0
2023-11-20 19:02:34,690 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=13.05 vs. limit=15.0
2023-11-20 19:02:53,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1182466.6666666667, ans=0.04949747468305833
2023-11-20 19:03:02,038 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9050, loss[loss=0.08898, simple_loss=0.1072, pruned_loss=0.02332, audio_tagging_loss=0.01204, over 14127.00 frames. ], tot_loss[loss=0.07949, simple_loss=0.1016, pruned_loss=0.01917, audio_tagging_loss=0.009525, over 3048417.06 frames. ], batch size: 53, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:03:03,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1182533.3333333333, ans=0.1
2023-11-20 19:03:09,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1182533.3333333333, ans=0.125
2023-11-20 19:03:16,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1182600.0, ans=0.0
2023-11-20 19:03:25,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177400
2023-11-20 19:03:33,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1182666.6666666667, ans=0.0
2023-11-20 19:03:52,614 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.00 vs. limit=15.0
2023-11-20 19:04:00,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.73 vs. limit=12.0
2023-11-20 19:04:07,302 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9100, loss[loss=0.08087, simple_loss=0.1081, pruned_loss=0.02091, audio_tagging_loss=0.005905, over 14971.00 frames. ], tot_loss[loss=0.07867, simple_loss=0.1005, pruned_loss=0.0189, audio_tagging_loss=0.009522, over 3048696.71 frames. ], batch size: 54, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:04:12,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1182866.6666666667, ans=0.125
2023-11-20 19:04:30,202 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177450
2023-11-20 19:04:34,930 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.170e+01 8.840e+01 9.659e+01 1.289e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 19:05:00,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1183133.3333333333, ans=0.07
2023-11-20 19:05:04,596 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.11 vs. limit=15.0
2023-11-20 19:05:05,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1183133.3333333333, ans=0.2
2023-11-20 19:05:06,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.12 vs. limit=15.0
2023-11-20 19:05:12,037 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9150, loss[loss=0.0696, simple_loss=0.08341, pruned_loss=0.0157, audio_tagging_loss=0.01219, over 15775.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09939, pruned_loss=0.01843, audio_tagging_loss=0.009586, over 3047756.32 frames. ], batch size: 60, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:05:30,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1183266.6666666667, ans=0.0
2023-11-20 19:05:35,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177500
2023-11-20 19:05:44,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1183333.3333333333, ans=0.2
2023-11-20 19:05:54,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1183400.0, ans=0.0
2023-11-20 19:06:15,564 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9200, loss[loss=0.07298, simple_loss=0.09661, pruned_loss=0.01385, audio_tagging_loss=0.01083, over 16100.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09896, pruned_loss=0.01839, audio_tagging_loss=0.009616, over 3049176.32 frames. ], batch size: 62, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:06:15,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1183533.3333333333, ans=0.125
2023-11-20 19:06:35,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1183600.0, ans=0.015
2023-11-20 19:06:39,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177550
2023-11-20 19:06:44,017 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.006e+01 8.354e+01 8.939e+01 9.701e+01 1.171e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 19:06:46,096 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.16 vs. limit=15.0
2023-11-20 19:06:46,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.46 vs. limit=15.0
2023-11-20 19:06:48,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1183666.6666666667, ans=0.125
2023-11-20 19:06:55,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1183733.3333333333, ans=0.1
2023-11-20 19:06:59,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1183733.3333333333, ans=0.125
2023-11-20 19:07:10,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1183800.0, ans=0.125
2023-11-20 19:07:20,264 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9250, loss[loss=0.08655, simple_loss=0.1133, pruned_loss=0.0228, audio_tagging_loss=0.007078, over 15970.00 frames. ], tot_loss[loss=0.07743, simple_loss=0.09857, pruned_loss=0.01853, audio_tagging_loss=0.00961, over 3046798.01 frames. ], batch size: 60, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:07:35,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1183933.3333333333, ans=0.0
2023-11-20 19:07:43,489 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177600
2023-11-20 19:07:55,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1184000.0, ans=0.125
2023-11-20 19:08:20,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1184133.3333333333, ans=0.125
2023-11-20 19:08:24,595 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9300, loss[loss=0.0845, simple_loss=0.1013, pruned_loss=0.02135, audio_tagging_loss=0.01248, over 13957.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09808, pruned_loss=0.01832, audio_tagging_loss=0.009689, over 3050418.54 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:08:44,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1184266.6666666667, ans=0.0
2023-11-20 19:08:48,437 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177650
2023-11-20 19:08:49,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1184333.3333333333, ans=0.2
2023-11-20 19:08:53,139 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 8.090e+01 8.777e+01 9.713e+01 1.132e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-20 19:09:28,664 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9350, loss[loss=0.06505, simple_loss=0.08244, pruned_loss=0.0138, audio_tagging_loss=0.01003, over 16690.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.09916, pruned_loss=0.01863, audio_tagging_loss=0.009679, over 3051235.20 frames. ], batch size: 62, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:09:40,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1184533.3333333333, ans=0.2
2023-11-20 19:09:52,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177700
2023-11-20 19:09:52,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1184600.0, ans=0.125
2023-11-20 19:10:18,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1184800.0, ans=0.2
2023-11-20 19:10:33,265 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9400, loss[loss=0.08841, simple_loss=0.111, pruned_loss=0.02393, audio_tagging_loss=0.008983, over 16696.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09862, pruned_loss=0.01856, audio_tagging_loss=0.009761, over 3054656.35 frames. ], batch size: 63, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:10:55,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1184933.3333333333, ans=0.125
2023-11-20 19:10:56,064 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177750
2023-11-20 19:11:01,382 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.184e+01 8.847e+01 9.642e+01 1.225e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-20 19:11:09,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1185000.0, ans=0.125
2023-11-20 19:11:09,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1185000.0, ans=0.125
2023-11-20 19:11:19,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1185066.6666666667, ans=0.0
2023-11-20 19:11:35,917 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:11:37,132 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9450, loss[loss=0.07491, simple_loss=0.0979, pruned_loss=0.01707, audio_tagging_loss=0.008891, over 15312.00 frames. ], tot_loss[loss=0.07791, simple_loss=0.0992, pruned_loss=0.01855, audio_tagging_loss=0.009763, over 3052961.17 frames. ], batch size: 57, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:11:52,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1185266.6666666667, ans=0.1
2023-11-20 19:12:00,830 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177800
2023-11-20 19:12:10,424 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:12:20,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1185400.0, ans=0.125
2023-11-20 19:12:24,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.17 vs. limit=15.0
2023-11-20 19:12:27,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1185466.6666666667, ans=0.0
2023-11-20 19:12:30,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1185466.6666666667, ans=0.125
2023-11-20 19:12:31,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1185466.6666666667, ans=0.125
2023-11-20 19:12:31,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1185466.6666666667, ans=0.125
2023-11-20 19:12:41,814 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9500, loss[loss=0.07902, simple_loss=0.09716, pruned_loss=0.01701, audio_tagging_loss=0.01342, over 16237.00 frames. ], tot_loss[loss=0.07805, simple_loss=0.09917, pruned_loss=0.01848, audio_tagging_loss=0.009977, over 3055652.42 frames. ], batch size: 61, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:12:43,408 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:12:47,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1185533.3333333333, ans=0.1
2023-11-20 19:12:47,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1185533.3333333333, ans=0.025
2023-11-20 19:13:05,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177850
2023-11-20 19:13:10,713 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.422e+01 8.141e+01 8.754e+01 9.457e+01 1.227e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-20 19:13:12,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1185666.6666666667, ans=0.125
2023-11-20 19:13:35,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1185800.0, ans=0.125
2023-11-20 19:13:47,318 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9550, loss[loss=0.07092, simple_loss=0.09186, pruned_loss=0.01389, audio_tagging_loss=0.01111, over 14830.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.09856, pruned_loss=0.0184, audio_tagging_loss=0.01012, over 3049461.11 frames. ], batch size: 54, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:14:05,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1185933.3333333333, ans=0.0
2023-11-20 19:14:09,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1185933.3333333333, ans=0.0
2023-11-20 19:14:10,134 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177900
2023-11-20 19:14:13,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1186000.0, ans=0.125
2023-11-20 19:14:13,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1186000.0, ans=0.0
2023-11-20 19:14:26,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1186066.6666666667, ans=0.1
2023-11-20 19:14:27,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1186066.6666666667, ans=0.125
2023-11-20 19:14:51,967 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9600, loss[loss=0.08231, simple_loss=0.1051, pruned_loss=0.02011, audio_tagging_loss=0.009642, over 14277.00 frames. ], tot_loss[loss=0.07803, simple_loss=0.0989, pruned_loss=0.01844, audio_tagging_loss=0.01014, over 3048783.49 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:15:05,212 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0
2023-11-20 19:15:12,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1186266.6666666667, ans=0.1
2023-11-20 19:15:15,737 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 177950
2023-11-20 19:15:22,418 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.290e+01 9.064e+01 1.000e+02 1.775e+02, threshold=1.813e+02, percent-clipped=1.0
2023-11-20 19:15:30,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1186400.0, ans=0.0
2023-11-20 19:15:44,671 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.14 vs. limit=22.5
2023-11-20 19:15:50,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1186466.6666666667, ans=0.125
2023-11-20 19:15:52,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1186466.6666666667, ans=0.125
2023-11-20 19:15:56,301 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9650, loss[loss=0.07424, simple_loss=0.08712, pruned_loss=0.02023, audio_tagging_loss=0.01045, over 15077.00 frames. ], tot_loss[loss=0.07823, simple_loss=0.09912, pruned_loss=0.01849, audio_tagging_loss=0.01018, over 3050749.07 frames. ], batch size: 59, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:16:12,863 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=14.49 vs. limit=15.0
2023-11-20 19:16:20,347 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178000
2023-11-20 19:16:32,414 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.48 vs. limit=15.0
2023-11-20 19:17:02,207 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9700, loss[loss=0.08409, simple_loss=0.1036, pruned_loss=0.0241, audio_tagging_loss=0.008181, over 16108.00 frames. ], tot_loss[loss=0.07831, simple_loss=0.09939, pruned_loss=0.01867, audio_tagging_loss=0.009942, over 3044537.65 frames. ], batch size: 63, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:17:25,252 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178050
2023-11-20 19:17:27,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1187000.0, ans=0.0
2023-11-20 19:17:31,280 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.180e+01 8.975e+01 9.665e+01 1.279e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-20 19:17:40,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1187066.6666666667, ans=0.0
2023-11-20 19:17:48,266 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.23 vs. limit=15.0
2023-11-20 19:18:01,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1187133.3333333333, ans=0.125
2023-11-20 19:18:06,459 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9750, loss[loss=0.05912, simple_loss=0.07071, pruned_loss=0.0123, audio_tagging_loss=0.01147, over 15603.00 frames. ], tot_loss[loss=0.07851, simple_loss=0.09988, pruned_loss=0.01879, audio_tagging_loss=0.00978, over 3051273.85 frames. ], batch size: 60, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:18:23,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1187266.6666666667, ans=0.125
2023-11-20 19:18:28,263 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178100
2023-11-20 19:18:50,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1187400.0, ans=0.125
2023-11-20 19:19:06,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1187466.6666666667, ans=0.125
2023-11-20 19:19:07,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1187466.6666666667, ans=0.125
2023-11-20 19:19:09,562 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9800, loss[loss=0.07979, simple_loss=0.1001, pruned_loss=0.02017, audio_tagging_loss=0.009566, over 15584.00 frames. ], tot_loss[loss=0.07833, simple_loss=0.09961, pruned_loss=0.01873, audio_tagging_loss=0.009795, over 3048846.31 frames. ], batch size: 59, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:19:24,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1187600.0, ans=0.125
2023-11-20 19:19:33,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178150
2023-11-20 19:19:34,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1187666.6666666667, ans=0.125
2023-11-20 19:19:39,612 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.101e+01 8.620e+01 9.316e+01 1.225e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-20 19:19:42,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1187666.6666666667, ans=0.125
2023-11-20 19:19:50,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.26 vs. limit=6.0
2023-11-20 19:20:05,621 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:20:14,290 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9850, loss[loss=0.09237, simple_loss=0.1323, pruned_loss=0.01903, audio_tagging_loss=0.007207, over 15962.00 frames. ], tot_loss[loss=0.07879, simple_loss=0.1004, pruned_loss=0.01891, audio_tagging_loss=0.009696, over 3043173.15 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:20:20,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1187866.6666666667, ans=0.1
2023-11-20 19:20:21,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff2.min_abs, batch_count=1187866.6666666667, ans=0.1
2023-11-20 19:20:37,491 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178200
2023-11-20 19:20:49,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1188000.0, ans=0.2
2023-11-20 19:20:51,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1188066.6666666667, ans=0.125
2023-11-20 19:20:58,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.73 vs. limit=22.5
2023-11-20 19:21:06,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1188133.3333333333, ans=0.1
2023-11-20 19:21:11,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1188133.3333333333, ans=0.0
2023-11-20 19:21:19,008 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9900, loss[loss=0.09564, simple_loss=0.1221, pruned_loss=0.02574, audio_tagging_loss=0.008844, over 15600.00 frames. ], tot_loss[loss=0.07866, simple_loss=0.1, pruned_loss=0.01888, audio_tagging_loss=0.009759, over 3040747.01 frames. ], batch size: 59, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:21:36,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1188266.6666666667, ans=0.0
2023-11-20 19:21:41,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178250
2023-11-20 19:21:47,802 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.012e+01 8.821e+01 9.655e+01 1.193e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-20 19:21:53,963 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.36 vs. limit=15.0
2023-11-20 19:22:02,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=1188400.0, ans=0.2
2023-11-20 19:22:22,898 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 9950, loss[loss=0.08195, simple_loss=0.1086, pruned_loss=0.0197, audio_tagging_loss=0.00797, over 14291.00 frames. ], tot_loss[loss=0.07839, simple_loss=0.09937, pruned_loss=0.01889, audio_tagging_loss=0.009818, over 3033208.41 frames. ], batch size: 55, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:22:25,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1188533.3333333333, ans=0.125
2023-11-20 19:22:45,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178300
2023-11-20 19:23:05,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1188733.3333333333, ans=0.0
2023-11-20 19:23:08,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1188733.3333333333, ans=0.1
2023-11-20 19:23:16,876 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:23:16,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1188800.0, ans=0.0
2023-11-20 19:23:26,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.12 vs. limit=15.0
2023-11-20 19:23:27,647 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10000, loss[loss=0.0808, simple_loss=0.1004, pruned_loss=0.02002, audio_tagging_loss=0.01059, over 15836.00 frames. ], tot_loss[loss=0.07836, simple_loss=0.09933, pruned_loss=0.01892, audio_tagging_loss=0.009777, over 3038004.45 frames. ], batch size: 58, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:23:31,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1188866.6666666667, ans=0.0
2023-11-20 19:23:44,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.73 vs. limit=15.0
2023-11-20 19:23:50,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178350
2023-11-20 19:23:50,694 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:23:56,459 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 7.984e+01 8.693e+01 9.319e+01 1.323e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-20 19:23:58,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1189000.0, ans=0.125
2023-11-20 19:23:59,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1189000.0, ans=0.2
2023-11-20 19:24:10,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1189066.6666666667, ans=0.125
2023-11-20 19:24:21,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1189133.3333333333, ans=0.125
2023-11-20 19:24:23,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1189133.3333333333, ans=0.0
2023-11-20 19:24:29,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1189133.3333333333, ans=0.0
2023-11-20 19:24:31,279 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10050, loss[loss=0.07491, simple_loss=0.09394, pruned_loss=0.01872, audio_tagging_loss=0.009219, over 16850.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09919, pruned_loss=0.01875, audio_tagging_loss=0.009771, over 3043910.13 frames. ], batch size: 62, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:24:36,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1189200.0, ans=10.0
2023-11-20 19:24:43,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1189266.6666666667, ans=0.125
2023-11-20 19:24:53,360 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178400
2023-11-20 19:24:54,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1189333.3333333333, ans=0.0
2023-11-20 19:24:57,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1189333.3333333333, ans=0.125
2023-11-20 19:25:01,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.52 vs. limit=15.0
2023-11-20 19:25:16,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1189400.0, ans=0.125
2023-11-20 19:25:19,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1189400.0, ans=0.0
2023-11-20 19:25:28,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1189466.6666666667, ans=0.1
2023-11-20 19:25:28,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1189466.6666666667, ans=0.0
2023-11-20 19:25:30,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1189466.6666666667, ans=0.05
2023-11-20 19:25:35,261 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10100, loss[loss=0.05787, simple_loss=0.07182, pruned_loss=0.009082, audio_tagging_loss=0.01288, over 14921.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09938, pruned_loss=0.01879, audio_tagging_loss=0.009801, over 3045200.44 frames. ], batch size: 58, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:25:37,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1189533.3333333333, ans=0.07
2023-11-20 19:25:52,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1189600.0, ans=0.125
2023-11-20 19:25:58,375 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178450
2023-11-20 19:25:59,758 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:26:02,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1189666.6666666667, ans=0.07
2023-11-20 19:26:04,389 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.521e+01 8.112e+01 8.895e+01 9.698e+01 1.490e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 19:26:22,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1189733.3333333333, ans=0.07
2023-11-20 19:26:24,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1189733.3333333333, ans=0.125
2023-11-20 19:26:26,332 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:26:27,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1189800.0, ans=0.0
2023-11-20 19:26:38,695 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10150, loss[loss=0.06842, simple_loss=0.07648, pruned_loss=0.01543, audio_tagging_loss=0.01474, over 16244.00 frames. ], tot_loss[loss=0.07841, simple_loss=0.09939, pruned_loss=0.01882, audio_tagging_loss=0.009895, over 3048568.48 frames. ], batch size: 62, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:26:40,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1189866.6666666667, ans=0.125
2023-11-20 19:26:42,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1189866.6666666667, ans=0.1
2023-11-20 19:26:52,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1189933.3333333333, ans=0.125
2023-11-20 19:27:03,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178500
2023-11-20 19:27:07,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.44 vs. limit=15.0
2023-11-20 19:27:09,269 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:27:12,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1190000.0, ans=0.07
2023-11-20 19:27:29,627 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:27:43,573 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10200, loss[loss=0.0978, simple_loss=0.1293, pruned_loss=0.02459, audio_tagging_loss=0.008573, over 15926.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09952, pruned_loss=0.01867, audio_tagging_loss=0.009946, over 3041857.20 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:27:44,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1190200.0, ans=0.1
2023-11-20 19:27:56,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1190266.6666666667, ans=0.125
2023-11-20 19:28:07,009 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178550
2023-11-20 19:28:08,196 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:28:11,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.61 vs. limit=15.0
2023-11-20 19:28:13,023 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.453e+01 8.265e+01 8.993e+01 9.730e+01 1.182e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-20 19:28:16,100 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.89 vs. limit=22.5
2023-11-20 19:28:32,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.67 vs. limit=15.0
2023-11-20 19:28:34,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1190466.6666666667, ans=0.0
2023-11-20 19:28:44,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1190466.6666666667, ans=0.125
2023-11-20 19:28:48,086 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10250, loss[loss=0.08379, simple_loss=0.1061, pruned_loss=0.02078, audio_tagging_loss=0.009951, over 14643.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09934, pruned_loss=0.01871, audio_tagging_loss=0.009996, over 3048218.81 frames. ], batch size: 55, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:28:53,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1190533.3333333333, ans=0.2
2023-11-20 19:29:11,254 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178600
2023-11-20 19:29:14,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1190666.6666666667, ans=0.1
2023-11-20 19:29:16,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.52 vs. limit=5.0
2023-11-20 19:29:18,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=1190666.6666666667, ans=0.1
2023-11-20 19:29:21,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.92 vs. limit=15.0
2023-11-20 19:29:23,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1190666.6666666667, ans=0.2
2023-11-20 19:29:29,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1190733.3333333333, ans=0.0
2023-11-20 19:29:35,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1190733.3333333333, ans=0.2
2023-11-20 19:29:37,535 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.06 vs. limit=22.5
2023-11-20 19:29:37,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.47 vs. limit=12.0
2023-11-20 19:29:47,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1190800.0, ans=0.2
2023-11-20 19:29:51,529 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10300, loss[loss=0.0716, simple_loss=0.08959, pruned_loss=0.01727, audio_tagging_loss=0.009533, over 14840.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.09927, pruned_loss=0.01853, audio_tagging_loss=0.01002, over 3051748.71 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:30:10,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1190933.3333333333, ans=0.0
2023-11-20 19:30:14,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1190933.3333333333, ans=0.125
2023-11-20 19:30:15,383 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178650
2023-11-20 19:30:21,246 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.948e+01 8.606e+01 9.117e+01 1.225e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-20 19:30:28,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1191000.0, ans=0.125
2023-11-20 19:30:52,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1191133.3333333333, ans=0.1
2023-11-20 19:30:54,215 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.99 vs. limit=15.0
2023-11-20 19:30:55,968 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10350, loss[loss=0.07587, simple_loss=0.1014, pruned_loss=0.01676, audio_tagging_loss=0.008431, over 15263.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09944, pruned_loss=0.01871, audio_tagging_loss=0.01004, over 3049022.96 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 16.0
2023-11-20 19:30:59,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1191200.0, ans=0.125
2023-11-20 19:31:14,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1191266.6666666667, ans=0.0
2023-11-20 19:31:19,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178700
2023-11-20 19:31:24,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1191333.3333333333, ans=0.2
2023-11-20 19:31:47,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1191466.6666666667, ans=0.125
2023-11-20 19:31:55,375 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.43 vs. limit=15.0
2023-11-20 19:31:58,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1191533.3333333333, ans=0.035
2023-11-20 19:31:59,606 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10400, loss[loss=0.08164, simple_loss=0.09494, pruned_loss=0.02184, audio_tagging_loss=0.01233, over 14303.00 frames. ], tot_loss[loss=0.07865, simple_loss=0.09942, pruned_loss=0.01875, audio_tagging_loss=0.01019, over 3049163.93 frames. ], batch size: 55, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:32:04,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1191533.3333333333, ans=0.125
2023-11-20 19:32:23,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178750
2023-11-20 19:32:30,360 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.343e+01 9.051e+01 9.727e+01 1.230e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-20 19:32:42,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1191733.3333333333, ans=0.09899494936611666
2023-11-20 19:32:43,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1191733.3333333333, ans=0.09899494936611666
2023-11-20 19:32:50,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1191800.0, ans=0.025
2023-11-20 19:32:58,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1191800.0, ans=0.125
2023-11-20 19:33:03,407 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10450, loss[loss=0.09943, simple_loss=0.1356, pruned_loss=0.02306, audio_tagging_loss=0.008593, over 16845.00 frames. ], tot_loss[loss=0.07916, simple_loss=0.1002, pruned_loss=0.01893, audio_tagging_loss=0.0101, over 3049860.66 frames. ], batch size: 60, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:33:27,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178800
2023-11-20 19:33:33,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.28 vs. limit=6.0
2023-11-20 19:33:39,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1192000.0, ans=0.0
2023-11-20 19:33:54,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1192133.3333333333, ans=0.125
2023-11-20 19:33:55,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.08 vs. limit=22.5
2023-11-20 19:34:04,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1192133.3333333333, ans=0.0
2023-11-20 19:34:08,051 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10500, loss[loss=0.0618, simple_loss=0.07678, pruned_loss=0.01409, audio_tagging_loss=0.00932, over 14785.00 frames. ], tot_loss[loss=0.0786, simple_loss=0.09937, pruned_loss=0.01888, audio_tagging_loss=0.01003, over 3047553.67 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:34:16,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1192200.0, ans=0.2
2023-11-20 19:34:20,981 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:34:23,806 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.04 vs. limit=15.0
2023-11-20 19:34:30,407 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178850
2023-11-20 19:34:31,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1192333.3333333333, ans=0.1
2023-11-20 19:34:37,988 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.449e+01 8.858e+01 9.853e+01 1.185e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-20 19:34:44,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1192400.0, ans=0.1
2023-11-20 19:34:52,993 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=14.73 vs. limit=15.0
2023-11-20 19:34:57,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1192466.6666666667, ans=0.0
2023-11-20 19:35:00,833 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.63 vs. limit=22.5
2023-11-20 19:35:05,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1192466.6666666667, ans=0.125
2023-11-20 19:35:11,110 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10550, loss[loss=0.06773, simple_loss=0.0887, pruned_loss=0.01271, audio_tagging_loss=0.01066, over 14026.00 frames. ], tot_loss[loss=0.07959, simple_loss=0.1011, pruned_loss=0.01921, audio_tagging_loss=0.009822, over 3045526.20 frames. ], batch size: 54, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:35:11,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1192533.3333333333, ans=0.125
2023-11-20 19:35:13,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1192533.3333333333, ans=0.125
2023-11-20 19:35:33,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178900
2023-11-20 19:35:39,898 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:35:43,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1192666.6666666667, ans=0.95
2023-11-20 19:36:14,459 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10600, loss[loss=0.07745, simple_loss=0.09215, pruned_loss=0.01918, audio_tagging_loss=0.01219, over 15236.00 frames. ], tot_loss[loss=0.07904, simple_loss=0.1006, pruned_loss=0.01899, audio_tagging_loss=0.009769, over 3042191.66 frames. ], batch size: 59, lr: 4.54e-03, grad_scale: 16.0
2023-11-20 19:36:37,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1192933.3333333333, ans=0.125
2023-11-20 19:36:38,420 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 178950
2023-11-20 19:36:40,168 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.20 vs. limit=15.0
2023-11-20 19:36:46,946 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.146e+01 8.630e+01 9.635e+01 1.232e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-20 19:37:18,824 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10650, loss[loss=0.0746, simple_loss=0.08764, pruned_loss=0.02095, audio_tagging_loss=0.009828, over 15009.00 frames. ], tot_loss[loss=0.07892, simple_loss=0.1006, pruned_loss=0.01889, audio_tagging_loss=0.009739, over 3049678.53 frames. ], batch size: 60, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:37:19,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.53 vs. limit=15.0
2023-11-20 19:37:41,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179000
2023-11-20 19:37:59,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1193400.0, ans=0.0
2023-11-20 19:38:08,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=1193466.6666666667, ans=0.02
2023-11-20 19:38:10,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1193466.6666666667, ans=0.2
2023-11-20 19:38:15,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1193466.6666666667, ans=0.125
2023-11-20 19:38:22,585 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10700, loss[loss=0.08186, simple_loss=0.1067, pruned_loss=0.02047, audio_tagging_loss=0.008053, over 16627.00 frames. ], tot_loss[loss=0.0791, simple_loss=0.1008, pruned_loss=0.01903, audio_tagging_loss=0.009663, over 3045263.34 frames. ], batch size: 63, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:38:32,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1193533.3333333333, ans=0.125
2023-11-20 19:38:44,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1193600.0, ans=0.2
2023-11-20 19:38:45,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179050
2023-11-20 19:38:51,144 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0
2023-11-20 19:38:54,229 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.026e+01 8.604e+01 9.471e+01 1.335e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-20 19:39:06,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1193733.3333333333, ans=0.125
2023-11-20 19:39:25,049 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10750, loss[loss=0.0752, simple_loss=0.09648, pruned_loss=0.01776, audio_tagging_loss=0.009195, over 13972.00 frames. ], tot_loss[loss=0.07895, simple_loss=0.1011, pruned_loss=0.01883, audio_tagging_loss=0.009596, over 3049510.96 frames. ], batch size: 55, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:39:44,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1193933.3333333333, ans=0.125
2023-11-20 19:39:47,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179100
2023-11-20 19:40:15,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1194133.3333333333, ans=0.125
2023-11-20 19:40:26,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1194133.3333333333, ans=0.125
2023-11-20 19:40:28,464 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10800, loss[loss=0.06455, simple_loss=0.0785, pruned_loss=0.0117, audio_tagging_loss=0.0136, over 14608.00 frames. ], tot_loss[loss=0.07839, simple_loss=0.1001, pruned_loss=0.01868, audio_tagging_loss=0.00965, over 3048087.93 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:40:29,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.50 vs. limit=22.5
2023-11-20 19:40:37,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1194200.0, ans=0.125
2023-11-20 19:40:50,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179150
2023-11-20 19:41:00,347 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.337e+01 8.834e+01 9.923e+01 1.321e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-20 19:41:09,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1194400.0, ans=0.5
2023-11-20 19:41:31,481 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10850, loss[loss=0.08898, simple_loss=0.1174, pruned_loss=0.02135, audio_tagging_loss=0.008939, over 15412.00 frames. ], tot_loss[loss=0.0785, simple_loss=0.1002, pruned_loss=0.0188, audio_tagging_loss=0.00962, over 3049544.22 frames. ], batch size: 55, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:41:34,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten.whitening_limit, batch_count=1194533.3333333333, ans=15.0
2023-11-20 19:41:53,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179200
2023-11-20 19:42:18,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1194733.3333333333, ans=0.125
2023-11-20 19:42:26,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1194800.0, ans=0.125
2023-11-20 19:42:27,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1194800.0, ans=0.125
2023-11-20 19:42:30,664 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:42:34,262 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10900, loss[loss=0.07117, simple_loss=0.08547, pruned_loss=0.01504, audio_tagging_loss=0.0134, over 15204.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.1004, pruned_loss=0.01868, audio_tagging_loss=0.009757, over 3047327.77 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:42:55,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1194933.3333333333, ans=0.125
2023-11-20 19:42:58,109 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179250
2023-11-20 19:43:02,864 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.97 vs. limit=15.0
2023-11-20 19:43:04,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=18.26 vs. limit=22.5
2023-11-20 19:43:04,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1195000.0, ans=0.125
2023-11-20 19:43:08,255 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.081e+01 8.823e+01 9.357e+01 1.279e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 19:43:19,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1195066.6666666667, ans=0.1
2023-11-20 19:43:24,508 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.47 vs. limit=22.5
2023-11-20 19:43:29,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.80 vs. limit=22.5
2023-11-20 19:43:38,315 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 10950, loss[loss=0.068, simple_loss=0.08035, pruned_loss=0.01416, audio_tagging_loss=0.01367, over 14412.00 frames. ], tot_loss[loss=0.07831, simple_loss=0.09983, pruned_loss=0.01851, audio_tagging_loss=0.009884, over 3044976.04 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:43:48,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1195200.0, ans=0.0
2023-11-20 19:43:52,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1195266.6666666667, ans=0.125
2023-11-20 19:43:59,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1195266.6666666667, ans=0.125
2023-11-20 19:43:59,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.whiten.whitening_limit, batch_count=1195266.6666666667, ans=12.0
2023-11-20 19:44:01,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179300
2023-11-20 19:44:06,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1195333.3333333333, ans=0.125
2023-11-20 19:44:14,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1195333.3333333333, ans=0.2
2023-11-20 19:44:27,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1195400.0, ans=0.125
2023-11-20 19:44:42,830 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11000, loss[loss=0.06254, simple_loss=0.08238, pruned_loss=0.01245, audio_tagging_loss=0.008911, over 14203.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.0987, pruned_loss=0.01819, audio_tagging_loss=0.01004, over 3043871.88 frames. ], batch size: 53, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:44:51,292 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:45:04,547 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179350
2023-11-20 19:45:13,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1195666.6666666667, ans=0.125
2023-11-20 19:45:15,397 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.295e+01 8.189e+01 8.881e+01 9.768e+01 1.271e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 19:45:16,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1195666.6666666667, ans=0.0
2023-11-20 19:45:45,438 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11050, loss[loss=0.08025, simple_loss=0.08946, pruned_loss=0.01994, audio_tagging_loss=0.01558, over 14485.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09877, pruned_loss=0.01826, audio_tagging_loss=0.01014, over 3041972.94 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:45:45,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1195866.6666666667, ans=0.0
2023-11-20 19:45:45,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1195866.6666666667, ans=0.125
2023-11-20 19:46:05,631 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:46:07,931 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179400
2023-11-20 19:46:20,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1196000.0, ans=0.0
2023-11-20 19:46:21,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.01 vs. limit=10.0
2023-11-20 19:46:44,080 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.23 vs. limit=15.0
2023-11-20 19:46:48,004 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11100, loss[loss=0.07622, simple_loss=0.09421, pruned_loss=0.01948, audio_tagging_loss=0.009628, over 15089.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.09867, pruned_loss=0.01835, audio_tagging_loss=0.0103, over 3041951.44 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:46:51,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1196200.0, ans=0.0
2023-11-20 19:46:58,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.78 vs. limit=15.0
2023-11-20 19:47:11,881 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179450
2023-11-20 19:47:21,550 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.315e+01 8.689e+01 9.550e+01 1.347e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 19:47:44,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1196466.6666666667, ans=0.2
2023-11-20 19:47:52,013 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11150, loss[loss=0.09496, simple_loss=0.1315, pruned_loss=0.02187, audio_tagging_loss=0.007325, over 14919.00 frames. ], tot_loss[loss=0.07825, simple_loss=0.0988, pruned_loss=0.01851, audio_tagging_loss=0.01035, over 3043394.87 frames. ], batch size: 55, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:47:56,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1196533.3333333333, ans=0.125
2023-11-20 19:48:13,501 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179500
2023-11-20 19:48:29,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten.whitening_limit, batch_count=1196733.3333333333, ans=22.5
2023-11-20 19:48:47,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1196800.0, ans=0.0
2023-11-20 19:48:54,079 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11200, loss[loss=0.06465, simple_loss=0.06978, pruned_loss=0.01278, audio_tagging_loss=0.01697, over 15321.00 frames. ], tot_loss[loss=0.07805, simple_loss=0.09856, pruned_loss=0.01836, audio_tagging_loss=0.01042, over 3046433.30 frames. ], batch size: 60, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:49:02,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1196866.6666666667, ans=0.1
2023-11-20 19:49:06,995 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.36 vs. limit=15.0
2023-11-20 19:49:14,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1196933.3333333333, ans=0.95
2023-11-20 19:49:16,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179550
2023-11-20 19:49:25,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1197000.0, ans=0.125
2023-11-20 19:49:26,733 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.291e+01 7.963e+01 8.549e+01 9.018e+01 1.277e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-20 19:49:31,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1197066.6666666667, ans=0.125
2023-11-20 19:49:48,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1197133.3333333333, ans=0.0
2023-11-20 19:49:50,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1197133.3333333333, ans=0.125
2023-11-20 19:49:50,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1197133.3333333333, ans=0.125
2023-11-20 19:49:56,605 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11250, loss[loss=0.08901, simple_loss=0.1061, pruned_loss=0.027, audio_tagging_loss=0.008933, over 13734.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09845, pruned_loss=0.0183, audio_tagging_loss=0.0103, over 3048440.03 frames. ], batch size: 54, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:49:58,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1197200.0, ans=0.1
2023-11-20 19:50:04,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1197200.0, ans=0.0
2023-11-20 19:50:20,337 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179600
2023-11-20 19:50:20,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1197266.6666666667, ans=0.0
2023-11-20 19:50:25,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1197333.3333333333, ans=0.04949747468305833
2023-11-20 19:50:30,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1197333.3333333333, ans=0.1
2023-11-20 19:50:31,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.98 vs. limit=15.0
2023-11-20 19:50:47,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1197466.6666666667, ans=0.0
2023-11-20 19:50:53,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=1197466.6666666667, ans=15.0
2023-11-20 19:50:59,723 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11300, loss[loss=0.09041, simple_loss=0.1204, pruned_loss=0.02015, audio_tagging_loss=0.01005, over 15747.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.09839, pruned_loss=0.01835, audio_tagging_loss=0.01014, over 3053999.09 frames. ], batch size: 59, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:51:05,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1197533.3333333333, ans=0.125
2023-11-20 19:51:15,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_na.min_abs, batch_count=1197600.0, ans=0.02
2023-11-20 19:51:16,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-20 19:51:22,921 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179650
2023-11-20 19:51:31,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1197666.6666666667, ans=0.0
2023-11-20 19:51:32,418 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.320e+01 8.034e+01 8.460e+01 9.162e+01 1.233e+02, threshold=1.692e+02, percent-clipped=0.0
2023-11-20 19:51:33,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1197666.6666666667, ans=0.1
2023-11-20 19:52:03,265 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11350, loss[loss=0.06883, simple_loss=0.08173, pruned_loss=0.01806, audio_tagging_loss=0.009904, over 14353.00 frames. ], tot_loss[loss=0.0772, simple_loss=0.09804, pruned_loss=0.01819, audio_tagging_loss=0.009994, over 3042956.80 frames. ], batch size: 55, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:52:25,651 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179700
2023-11-20 19:52:29,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=1198000.0, ans=0.2
2023-11-20 19:52:31,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1198000.0, ans=0.125
2023-11-20 19:52:35,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.01 vs. limit=15.0
2023-11-20 19:52:35,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=1198000.0, ans=0.2
2023-11-20 19:52:41,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1198066.6666666667, ans=0.1
2023-11-20 19:53:02,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1198133.3333333333, ans=0.1
2023-11-20 19:53:05,827 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11400, loss[loss=0.08075, simple_loss=0.09029, pruned_loss=0.02428, audio_tagging_loss=0.01132, over 15548.00 frames. ], tot_loss[loss=0.07801, simple_loss=0.0995, pruned_loss=0.0185, audio_tagging_loss=0.009765, over 3048515.64 frames. ], batch size: 61, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:53:09,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1198200.0, ans=0.125
2023-11-20 19:53:29,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179750
2023-11-20 19:53:39,662 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.357e+01 7.998e+01 8.618e+01 9.401e+01 1.167e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-20 19:53:41,437 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=12.0
2023-11-20 19:53:49,778 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:54:09,902 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11450, loss[loss=0.09303, simple_loss=0.1158, pruned_loss=0.02611, audio_tagging_loss=0.009015, over 15181.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09939, pruned_loss=0.01842, audio_tagging_loss=0.009782, over 3048543.35 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:54:21,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1198600.0, ans=0.0
2023-11-20 19:54:32,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179800
2023-11-20 19:55:07,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1198800.0, ans=0.2
2023-11-20 19:55:13,638 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11500, loss[loss=0.09341, simple_loss=0.1252, pruned_loss=0.02381, audio_tagging_loss=0.006998, over 14419.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09842, pruned_loss=0.0183, audio_tagging_loss=0.009856, over 3039773.94 frames. ], batch size: 54, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:55:19,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1198866.6666666667, ans=0.125
2023-11-20 19:55:29,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1198933.3333333333, ans=0.125
2023-11-20 19:55:35,621 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=15.0
2023-11-20 19:55:36,221 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179850
2023-11-20 19:55:43,005 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.04 vs. limit=15.0
2023-11-20 19:55:47,194 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.150e+01 8.270e+01 9.064e+01 9.843e+01 1.286e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-20 19:55:56,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1199066.6666666667, ans=0.0
2023-11-20 19:56:01,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1199066.6666666667, ans=0.125
2023-11-20 19:56:01,750 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.43 vs. limit=6.0
2023-11-20 19:56:10,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1199133.3333333333, ans=0.125
2023-11-20 19:56:17,614 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11550, loss[loss=0.07096, simple_loss=0.08915, pruned_loss=0.01689, audio_tagging_loss=0.009494, over 14838.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09882, pruned_loss=0.01816, audio_tagging_loss=0.009744, over 3044769.18 frames. ], batch size: 61, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:56:17,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1199200.0, ans=0.1
2023-11-20 19:56:31,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1199266.6666666667, ans=0.125
2023-11-20 19:56:33,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1199266.6666666667, ans=0.0
2023-11-20 19:56:38,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1199266.6666666667, ans=0.1
2023-11-20 19:56:38,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1199266.6666666667, ans=0.125
2023-11-20 19:56:41,388 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179900
2023-11-20 19:56:56,113 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:57:12,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1199466.6666666667, ans=0.125
2023-11-20 19:57:22,572 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11600, loss[loss=0.07962, simple_loss=0.1161, pruned_loss=0.01462, audio_tagging_loss=0.00695, over 15921.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.09959, pruned_loss=0.01834, audio_tagging_loss=0.009785, over 3042959.55 frames. ], batch size: 57, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:57:36,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1199600.0, ans=0.0
2023-11-20 19:57:44,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 179950
2023-11-20 19:57:47,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1199666.6666666667, ans=0.125
2023-11-20 19:57:51,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1199666.6666666667, ans=0.0
2023-11-20 19:57:55,288 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.413e+01 9.073e+01 1.016e+02 1.405e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-20 19:57:56,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1199666.6666666667, ans=0.0
2023-11-20 19:58:02,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1199733.3333333333, ans=0.1
2023-11-20 19:58:05,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1199733.3333333333, ans=0.125
2023-11-20 19:58:16,178 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:58:26,085 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11650, loss[loss=0.07232, simple_loss=0.09446, pruned_loss=0.01432, audio_tagging_loss=0.01077, over 14354.00 frames. ], tot_loss[loss=0.07829, simple_loss=0.09993, pruned_loss=0.01852, audio_tagging_loss=0.009809, over 3040739.55 frames. ], batch size: 55, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:58:29,038 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.22 vs. limit=15.0
2023-11-20 19:58:42,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1199933.3333333333, ans=0.125
2023-11-20 19:58:48,769 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180000
2023-11-20 19:58:57,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1200000.0, ans=0.125
2023-11-20 19:59:25,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1200133.3333333333, ans=0.0
2023-11-20 19:59:27,234 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.76 vs. limit=10.0
2023-11-20 19:59:32,099 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11700, loss[loss=0.06977, simple_loss=0.09086, pruned_loss=0.01516, audio_tagging_loss=0.009178, over 15162.00 frames. ], tot_loss[loss=0.07827, simple_loss=0.09992, pruned_loss=0.01851, audio_tagging_loss=0.009798, over 3044607.52 frames. ], batch size: 61, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:59:42,393 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.75 vs. limit=15.0
2023-11-20 19:59:47,512 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.83 vs. limit=15.0
2023-11-20 19:59:49,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1200266.6666666667, ans=0.1
2023-11-20 19:59:55,370 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180050
2023-11-20 20:00:04,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1200333.3333333333, ans=0.1
2023-11-20 20:00:06,810 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.544e+01 8.114e+01 8.755e+01 9.534e+01 1.280e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-20 20:00:07,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1200333.3333333333, ans=0.0
2023-11-20 20:00:14,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1200400.0, ans=0.0
2023-11-20 20:00:26,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1200466.6666666667, ans=0.2
2023-11-20 20:00:35,972 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11750, loss[loss=0.05901, simple_loss=0.07125, pruned_loss=0.01196, audio_tagging_loss=0.01143, over 15285.00 frames. ], tot_loss[loss=0.07834, simple_loss=0.09967, pruned_loss=0.01866, audio_tagging_loss=0.009834, over 3040596.52 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:00:40,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1200533.3333333333, ans=0.125
2023-11-20 20:00:41,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1200533.3333333333, ans=0.2
2023-11-20 20:00:58,954 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180100
2023-11-20 20:01:15,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1200733.3333333333, ans=0.5
2023-11-20 20:01:40,522 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11800, loss[loss=0.07401, simple_loss=0.08943, pruned_loss=0.01893, audio_tagging_loss=0.01037, over 14768.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09784, pruned_loss=0.01845, audio_tagging_loss=0.009952, over 3040636.45 frames. ], batch size: 57, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:01:55,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1200933.3333333333, ans=0.1
2023-11-20 20:02:03,357 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180150
2023-11-20 20:02:10,026 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.99 vs. limit=15.0
2023-11-20 20:02:15,383 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.209e+01 9.058e+01 9.835e+01 1.362e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-20 20:02:19,357 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:02:19,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1201066.6666666667, ans=0.125
2023-11-20 20:02:31,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1201133.3333333333, ans=0.125
2023-11-20 20:02:43,898 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11850, loss[loss=0.0797, simple_loss=0.1026, pruned_loss=0.01785, audio_tagging_loss=0.01053, over 15229.00 frames. ], tot_loss[loss=0.07766, simple_loss=0.09824, pruned_loss=0.01851, audio_tagging_loss=0.01003, over 3039269.25 frames. ], batch size: 56, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:02:56,168 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.67 vs. limit=15.0
2023-11-20 20:03:04,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1201266.6666666667, ans=0.125
2023-11-20 20:03:07,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180200
2023-11-20 20:03:29,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1201400.0, ans=0.1
2023-11-20 20:03:39,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1201466.6666666667, ans=0.125
2023-11-20 20:03:42,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-20 20:03:48,549 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11900, loss[loss=0.07398, simple_loss=0.09099, pruned_loss=0.01454, audio_tagging_loss=0.01394, over 15302.00 frames. ], tot_loss[loss=0.07777, simple_loss=0.09832, pruned_loss=0.01852, audio_tagging_loss=0.01008, over 3042830.10 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:03:56,962 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.65 vs. limit=15.0
2023-11-20 20:04:11,025 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.99 vs. limit=15.0
2023-11-20 20:04:11,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180250
2023-11-20 20:04:22,706 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.009e+01 8.683e+01 9.562e+01 2.232e+02, threshold=1.737e+02, percent-clipped=1.0
2023-11-20 20:04:30,652 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=12.0
2023-11-20 20:04:32,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1201733.3333333333, ans=0.125
2023-11-20 20:04:52,344 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 11950, loss[loss=0.07844, simple_loss=0.105, pruned_loss=0.01615, audio_tagging_loss=0.009812, over 15688.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09849, pruned_loss=0.01858, audio_tagging_loss=0.01007, over 3043342.72 frames. ], batch size: 57, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:04:56,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1201866.6666666667, ans=0.2
2023-11-20 20:05:09,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1201933.3333333333, ans=0.2
2023-11-20 20:05:14,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180300
2023-11-20 20:05:26,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.97 vs. limit=22.5
2023-11-20 20:05:31,222 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.45 vs. limit=10.0
2023-11-20 20:05:36,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=1202066.6666666667, ans=15.0
2023-11-20 20:05:47,492 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0
2023-11-20 20:05:52,620 INFO [train_asr.py:1221] (2/4) Epoch 15, batch 12000, loss[loss=0.07093, simple_loss=0.09491, pruned_loss=0.01481, audio_tagging_loss=0.008666, over 16562.00 frames. ], tot_loss[loss=0.07835, simple_loss=0.09916, pruned_loss=0.01865, audio_tagging_loss=0.01012, over 3042838.80 frames. ], batch size: 63, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 20:05:52,621 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 20:06:33,114 INFO [train_asr.py:1253] (2/4) Epoch 15, validation: loss=0.06134, simple_loss=0.05315, pruned_loss=0.00551, audio_tagging_loss=0.02926, over 4681554.00 frames. 
2023-11-20 20:06:33,115 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 20:06:37,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.92 vs. limit=15.0
2023-11-20 20:06:46,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1202266.6666666667, ans=0.125
2023-11-20 20:06:54,552 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180350
2023-11-20 20:07:00,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.19 vs. limit=15.0
2023-11-20 20:07:37,558 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 0, loss[loss=0.07692, simple_loss=0.08612, pruned_loss=0.01162, audio_tagging_loss=0.02224, over 14103.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.08612, pruned_loss=0.01162, audio_tagging_loss=0.02224, over 14103.00 frames. ], batch size: 53, lr: 4.37e-03, grad_scale: 32.0
2023-11-20 20:07:37,559 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 20:07:56,133 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2216, 2.5457, 4.0796, 4.1878, 4.3214, 3.8967, 4.1467, 3.9733],
       device='cuda:2')
2023-11-20 20:08:13,631 INFO [train_asr.py:1253] (2/4) Epoch 16, validation: loss=0.06129, simple_loss=0.0532, pruned_loss=0.005566, audio_tagging_loss=0.02913, over 4681554.00 frames. 
2023-11-20 20:08:13,632 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 20:08:17,303 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.282e+01 8.389e+01 9.121e+01 1.002e+02 1.446e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-20 20:08:25,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1202426.6666666667, ans=0.2
2023-11-20 20:08:44,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1202493.3333333333, ans=0.2
2023-11-20 20:08:59,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1202560.0, ans=0.125
2023-11-20 20:09:11,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180400
2023-11-20 20:09:19,020 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 50, loss[loss=0.09763, simple_loss=0.1253, pruned_loss=0.0214, audio_tagging_loss=0.0136, over 15882.00 frames. ], tot_loss[loss=0.09021, simple_loss=0.1035, pruned_loss=0.01977, audio_tagging_loss=0.01867, over 685093.41 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:09:39,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1202760.0, ans=0.125
2023-11-20 20:09:58,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1202893.3333333333, ans=0.09899494936611666
2023-11-20 20:10:15,520 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.89 vs. limit=22.5
2023-11-20 20:10:16,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180450
2023-11-20 20:10:23,389 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 100, loss[loss=0.08852, simple_loss=0.1075, pruned_loss=0.02048, audio_tagging_loss=0.0143, over 15100.00 frames. ], tot_loss[loss=0.08676, simple_loss=0.09986, pruned_loss=0.01854, audio_tagging_loss=0.01829, over 1211660.45 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:10:29,503 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.348e+01 8.922e+01 9.752e+01 1.060e+02 1.405e+02, threshold=1.950e+02, percent-clipped=0.0
2023-11-20 20:10:33,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1203026.6666666667, ans=22.5
2023-11-20 20:11:00,844 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.07 vs. limit=15.0
2023-11-20 20:11:06,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.21 vs. limit=15.0
2023-11-20 20:11:21,280 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180500
2023-11-20 20:11:28,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1203360.0, ans=0.0
2023-11-20 20:11:29,069 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 150, loss[loss=0.08951, simple_loss=0.1088, pruned_loss=0.02091, audio_tagging_loss=0.01421, over 15101.00 frames. ], tot_loss[loss=0.08628, simple_loss=0.1022, pruned_loss=0.0192, audio_tagging_loss=0.01597, over 1621200.07 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:11:33,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1203360.0, ans=0.125
2023-11-20 20:11:49,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1203426.6666666667, ans=0.1
2023-11-20 20:11:53,366 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=3.374e-01
2023-11-20 20:12:03,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1203493.3333333333, ans=0.125
2023-11-20 20:12:14,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1203560.0, ans=0.2
2023-11-20 20:12:25,596 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180550
2023-11-20 20:12:33,417 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 200, loss[loss=0.09428, simple_loss=0.1246, pruned_loss=0.02183, audio_tagging_loss=0.01018, over 15379.00 frames. ], tot_loss[loss=0.0841, simple_loss=0.1017, pruned_loss=0.01913, audio_tagging_loss=0.01414, over 1938218.31 frames. ], batch size: 56, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:12:39,629 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.954e+01 8.158e+01 8.825e+01 9.398e+01 1.176e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 20:13:12,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1203893.3333333333, ans=0.125
2023-11-20 20:13:17,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1203893.3333333333, ans=0.125
2023-11-20 20:13:22,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1203893.3333333333, ans=0.125
2023-11-20 20:13:29,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180600
2023-11-20 20:13:32,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1203960.0, ans=0.125
2023-11-20 20:13:36,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1204026.6666666667, ans=0.125
2023-11-20 20:13:37,507 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 250, loss[loss=0.07024, simple_loss=0.08255, pruned_loss=0.02085, audio_tagging_loss=0.008123, over 15000.00 frames. ], tot_loss[loss=0.08234, simple_loss=0.1015, pruned_loss=0.0189, audio_tagging_loss=0.01271, over 2181270.90 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:13:52,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1204093.3333333333, ans=0.125
2023-11-20 20:13:52,888 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.70 vs. limit=10.0
2023-11-20 20:14:02,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1204160.0, ans=0.2
2023-11-20 20:14:04,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.28 vs. limit=22.5
2023-11-20 20:14:05,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1204160.0, ans=0.125
2023-11-20 20:14:14,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1204160.0, ans=0.0
2023-11-20 20:14:19,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1204226.6666666667, ans=0.125
2023-11-20 20:14:35,057 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180650
2023-11-20 20:14:42,348 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 300, loss[loss=0.07511, simple_loss=0.09086, pruned_loss=0.02135, audio_tagging_loss=0.008332, over 15459.00 frames. ], tot_loss[loss=0.08281, simple_loss=0.103, pruned_loss=0.01943, audio_tagging_loss=0.01189, over 2371664.03 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:14:48,971 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.454e+01 9.235e+01 1.018e+02 1.447e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-20 20:15:08,470 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:15:16,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1204493.3333333333, ans=10.0
2023-11-20 20:15:18,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1204493.3333333333, ans=0.125
2023-11-20 20:15:36,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1204626.6666666667, ans=0.0
2023-11-20 20:15:39,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180700
2023-11-20 20:15:46,852 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 350, loss[loss=0.07364, simple_loss=0.09325, pruned_loss=0.01563, audio_tagging_loss=0.01138, over 14599.00 frames. ], tot_loss[loss=0.08127, simple_loss=0.1019, pruned_loss=0.01888, audio_tagging_loss=0.01143, over 2518582.37 frames. ], batch size: 53, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:16:17,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.52 vs. limit=12.0
2023-11-20 20:16:26,049 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.88 vs. limit=22.5
2023-11-20 20:16:37,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1204893.3333333333, ans=0.125
2023-11-20 20:16:40,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1204960.0, ans=0.125
2023-11-20 20:16:44,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180750
2023-11-20 20:16:51,456 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 400, loss[loss=0.08722, simple_loss=0.1176, pruned_loss=0.02098, audio_tagging_loss=0.007414, over 16275.00 frames. ], tot_loss[loss=0.08011, simple_loss=0.1008, pruned_loss=0.01872, audio_tagging_loss=0.01101, over 2633918.62 frames. ], batch size: 60, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:16:54,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1205026.6666666667, ans=0.0
2023-11-20 20:16:58,137 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.022e+01 8.670e+01 9.315e+01 1.277e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-20 20:17:11,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1205093.3333333333, ans=0.0
2023-11-20 20:17:14,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1205093.3333333333, ans=0.1
2023-11-20 20:17:23,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1205160.0, ans=0.125
2023-11-20 20:17:33,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1205226.6666666667, ans=0.2
2023-11-20 20:17:40,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1205226.6666666667, ans=0.125
2023-11-20 20:17:48,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180800
2023-11-20 20:17:56,389 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 450, loss[loss=0.0882, simple_loss=0.1086, pruned_loss=0.02361, audio_tagging_loss=0.0103, over 15173.00 frames. ], tot_loss[loss=0.08037, simple_loss=0.1017, pruned_loss=0.019, audio_tagging_loss=0.01053, over 2717567.02 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:18:00,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0
2023-11-20 20:18:33,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1205560.0, ans=0.1
2023-11-20 20:18:38,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1205560.0, ans=0.125
2023-11-20 20:18:45,667 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.414e-01
2023-11-20 20:18:52,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180850
2023-11-20 20:18:59,910 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 500, loss[loss=0.07443, simple_loss=0.08945, pruned_loss=0.01745, audio_tagging_loss=0.01226, over 14756.00 frames. ], tot_loss[loss=0.0793, simple_loss=0.1003, pruned_loss=0.01873, audio_tagging_loss=0.01044, over 2788200.91 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:19:07,187 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.820e+01 8.156e+01 8.896e+01 9.725e+01 1.312e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 20:19:21,434 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.85 vs. limit=15.0
2023-11-20 20:19:26,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1205826.6666666667, ans=0.2
2023-11-20 20:19:43,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1205893.3333333333, ans=0.2
2023-11-20 20:19:56,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180900
2023-11-20 20:19:58,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1205960.0, ans=0.125
2023-11-20 20:20:03,751 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 550, loss[loss=0.06916, simple_loss=0.08539, pruned_loss=0.01304, audio_tagging_loss=0.01342, over 15035.00 frames. ], tot_loss[loss=0.07887, simple_loss=0.09973, pruned_loss=0.01869, audio_tagging_loss=0.01032, over 2851928.65 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:20:06,917 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.09 vs. limit=12.0
2023-11-20 20:20:26,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1206093.3333333333, ans=0.125
2023-11-20 20:20:30,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1206160.0, ans=0.125
2023-11-20 20:20:38,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1206160.0, ans=0.2
2023-11-20 20:20:38,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1206160.0, ans=0.95
2023-11-20 20:20:39,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1206160.0, ans=0.1
2023-11-20 20:20:51,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1206226.6666666667, ans=0.125
2023-11-20 20:20:53,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1206293.3333333333, ans=0.1
2023-11-20 20:20:56,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.65 vs. limit=5.0
2023-11-20 20:20:58,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1206293.3333333333, ans=0.125
2023-11-20 20:20:59,467 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 180950
2023-11-20 20:21:06,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1206360.0, ans=0.125
2023-11-20 20:21:07,313 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 600, loss[loss=0.08315, simple_loss=0.1065, pruned_loss=0.01964, audio_tagging_loss=0.01025, over 16452.00 frames. ], tot_loss[loss=0.0787, simple_loss=0.09989, pruned_loss=0.01857, audio_tagging_loss=0.01018, over 2893935.50 frames. ], batch size: 62, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:21:13,337 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.725e+01 8.009e+01 8.820e+01 9.311e+01 1.105e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-20 20:21:24,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1206426.6666666667, ans=0.125
2023-11-20 20:21:27,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1206426.6666666667, ans=0.125
2023-11-20 20:21:37,049 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.19 vs. limit=5.0
2023-11-20 20:21:51,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1206560.0, ans=0.125
2023-11-20 20:21:55,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1206560.0, ans=0.2
2023-11-20 20:21:55,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.94 vs. limit=22.5
2023-11-20 20:21:58,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.19 vs. limit=12.0
2023-11-20 20:22:01,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1206626.6666666667, ans=0.0
2023-11-20 20:22:02,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181000
2023-11-20 20:22:03,336 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0
2023-11-20 20:22:03,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1206626.6666666667, ans=0.125
2023-11-20 20:22:10,125 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 650, loss[loss=0.08346, simple_loss=0.1096, pruned_loss=0.019, audio_tagging_loss=0.009656, over 14721.00 frames. ], tot_loss[loss=0.07898, simple_loss=0.1003, pruned_loss=0.01862, audio_tagging_loss=0.01022, over 2934459.59 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:22:15,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1206693.3333333333, ans=0.2
2023-11-20 20:22:26,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1206760.0, ans=0.0
2023-11-20 20:22:47,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1206893.3333333333, ans=0.1
2023-11-20 20:23:06,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181050
2023-11-20 20:23:14,301 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 700, loss[loss=0.05369, simple_loss=0.06452, pruned_loss=0.008836, audio_tagging_loss=0.0126, over 15522.00 frames. ], tot_loss[loss=0.07836, simple_loss=0.09957, pruned_loss=0.0184, audio_tagging_loss=0.01018, over 2967482.43 frames. ], batch size: 59, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:23:14,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1207026.6666666667, ans=0.1
2023-11-20 20:23:15,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1207026.6666666667, ans=0.0
2023-11-20 20:23:16,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1207026.6666666667, ans=0.015
2023-11-20 20:23:20,232 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.218e+01 8.797e+01 9.608e+01 1.489e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-20 20:23:29,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1207093.3333333333, ans=0.125
2023-11-20 20:23:36,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1207093.3333333333, ans=0.0
2023-11-20 20:23:57,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1207226.6666666667, ans=0.125
2023-11-20 20:23:58,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1207226.6666666667, ans=0.125
2023-11-20 20:24:07,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1207293.3333333333, ans=0.0
2023-11-20 20:24:09,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181100
2023-11-20 20:24:09,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1207293.3333333333, ans=0.05
2023-11-20 20:24:17,484 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 750, loss[loss=0.06482, simple_loss=0.07263, pruned_loss=0.01556, audio_tagging_loss=0.01294, over 14855.00 frames. ], tot_loss[loss=0.07856, simple_loss=0.09978, pruned_loss=0.01851, audio_tagging_loss=0.01016, over 2990259.38 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:24:25,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1207360.0, ans=0.0
2023-11-20 20:24:45,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.21 vs. limit=22.5
2023-11-20 20:25:13,029 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181150
2023-11-20 20:25:15,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1207626.6666666667, ans=0.125
2023-11-20 20:25:20,307 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 800, loss[loss=0.06657, simple_loss=0.08237, pruned_loss=0.0152, audio_tagging_loss=0.01019, over 15191.00 frames. ], tot_loss[loss=0.07867, simple_loss=0.1, pruned_loss=0.01861, audio_tagging_loss=0.01005, over 3015316.82 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:25:26,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.116e+01 8.797e+01 9.654e+01 1.312e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-20 20:25:42,715 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.76 vs. limit=15.0
2023-11-20 20:25:46,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1207826.6666666667, ans=0.125
2023-11-20 20:26:14,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1207960.0, ans=0.125
2023-11-20 20:26:16,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181200
2023-11-20 20:26:23,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1208026.6666666667, ans=0.0
2023-11-20 20:26:23,961 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 850, loss[loss=0.08667, simple_loss=0.1106, pruned_loss=0.01969, audio_tagging_loss=0.01168, over 15685.00 frames. ], tot_loss[loss=0.07902, simple_loss=0.1005, pruned_loss=0.01878, audio_tagging_loss=0.009989, over 3027247.40 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:26:39,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1208093.3333333333, ans=0.125
2023-11-20 20:26:47,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1208093.3333333333, ans=0.025
2023-11-20 20:27:14,401 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.55 vs. limit=6.0
2023-11-20 20:27:20,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181250
2023-11-20 20:27:28,684 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 900, loss[loss=0.08581, simple_loss=0.1043, pruned_loss=0.01913, audio_tagging_loss=0.01455, over 15303.00 frames. ], tot_loss[loss=0.07957, simple_loss=0.1011, pruned_loss=0.01891, audio_tagging_loss=0.0101, over 3035209.25 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:27:35,940 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.347e+01 8.999e+01 9.634e+01 1.332e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-20 20:28:23,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1208626.6666666667, ans=0.125
2023-11-20 20:28:23,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1208626.6666666667, ans=0.125
2023-11-20 20:28:24,524 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181300
2023-11-20 20:28:24,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1208626.6666666667, ans=0.0
2023-11-20 20:28:31,774 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 950, loss[loss=0.07531, simple_loss=0.09175, pruned_loss=0.02017, audio_tagging_loss=0.009266, over 14567.00 frames. ], tot_loss[loss=0.07944, simple_loss=0.1013, pruned_loss=0.01882, audio_tagging_loss=0.009976, over 3038077.52 frames. ], batch size: 55, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:28:35,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1208693.3333333333, ans=0.125
2023-11-20 20:28:38,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1208693.3333333333, ans=0.2
2023-11-20 20:29:06,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1208826.6666666667, ans=0.1
2023-11-20 20:29:12,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1208893.3333333333, ans=0.125
2023-11-20 20:29:14,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1208893.3333333333, ans=0.2
2023-11-20 20:29:18,524 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.33 vs. limit=15.0
2023-11-20 20:29:21,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1208960.0, ans=0.125
2023-11-20 20:29:27,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181350
2023-11-20 20:29:35,693 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1000, loss[loss=0.07596, simple_loss=0.08961, pruned_loss=0.01929, audio_tagging_loss=0.01186, over 14316.00 frames. ], tot_loss[loss=0.07914, simple_loss=0.1011, pruned_loss=0.01874, audio_tagging_loss=0.009828, over 3035044.89 frames. ], batch size: 53, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:29:42,951 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.425e+01 8.384e+01 9.027e+01 9.924e+01 1.224e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-20 20:29:51,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.56 vs. limit=15.0
2023-11-20 20:29:55,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1209093.3333333333, ans=0.1
2023-11-20 20:30:02,967 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:30:17,870 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.30 vs. limit=12.0
2023-11-20 20:30:17,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=13.38 vs. limit=15.0
2023-11-20 20:30:31,917 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181400
2023-11-20 20:30:40,295 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1050, loss[loss=0.1016, simple_loss=0.1362, pruned_loss=0.02664, audio_tagging_loss=0.006851, over 15717.00 frames. ], tot_loss[loss=0.07843, simple_loss=0.1005, pruned_loss=0.01854, audio_tagging_loss=0.009657, over 3037668.62 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:30:48,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1209360.0, ans=0.125
2023-11-20 20:30:53,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.51 vs. limit=15.0
2023-11-20 20:31:00,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1209426.6666666667, ans=0.125
2023-11-20 20:31:05,006 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.81 vs. limit=15.0
2023-11-20 20:31:19,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1209560.0, ans=0.2
2023-11-20 20:31:21,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1209560.0, ans=0.125
2023-11-20 20:31:26,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1209560.0, ans=0.09899494936611666
2023-11-20 20:31:35,690 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181450
2023-11-20 20:31:42,790 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1100, loss[loss=0.07623, simple_loss=0.09346, pruned_loss=0.0166, audio_tagging_loss=0.01289, over 14355.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09936, pruned_loss=0.01833, audio_tagging_loss=0.009737, over 3032207.67 frames. ], batch size: 54, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:31:45,256 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:31:49,905 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.927e+01 7.892e+01 8.572e+01 9.142e+01 1.110e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-20 20:32:02,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1209760.0, ans=0.1
2023-11-20 20:32:04,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1209760.0, ans=0.0
2023-11-20 20:32:25,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1209893.3333333333, ans=0.0
2023-11-20 20:32:37,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1209960.0, ans=0.125
2023-11-20 20:32:38,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181500
2023-11-20 20:32:39,524 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:32:46,239 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1150, loss[loss=0.06597, simple_loss=0.08308, pruned_loss=0.01667, audio_tagging_loss=0.007755, over 15097.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09889, pruned_loss=0.01823, audio_tagging_loss=0.009701, over 3031079.14 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:32:48,228 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.44 vs. limit=15.0
2023-11-20 20:33:12,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1210160.0, ans=0.125
2023-11-20 20:33:29,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1210226.6666666667, ans=0.125
2023-11-20 20:33:34,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1210226.6666666667, ans=0.2
2023-11-20 20:33:37,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1210293.3333333333, ans=0.2
2023-11-20 20:33:38,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1210293.3333333333, ans=0.125
2023-11-20 20:33:42,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181550
2023-11-20 20:33:44,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1210293.3333333333, ans=0.0
2023-11-20 20:33:49,371 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1200, loss[loss=0.08647, simple_loss=0.1152, pruned_loss=0.02064, audio_tagging_loss=0.008217, over 16186.00 frames. ], tot_loss[loss=0.07754, simple_loss=0.09904, pruned_loss=0.01835, audio_tagging_loss=0.009675, over 3020353.29 frames. ], batch size: 60, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:33:52,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1210360.0, ans=0.0
2023-11-20 20:33:56,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1210360.0, ans=0.0
2023-11-20 20:33:57,416 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.318e+01 9.034e+01 9.990e+01 1.243e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-20 20:33:59,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1210360.0, ans=0.0
2023-11-20 20:34:11,162 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.79 vs. limit=6.0
2023-11-20 20:34:17,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1210493.3333333333, ans=0.125
2023-11-20 20:34:31,151 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:34:45,215 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.02 vs. limit=15.0
2023-11-20 20:34:45,939 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181600
2023-11-20 20:34:49,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1210626.6666666667, ans=0.125
2023-11-20 20:34:54,250 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1250, loss[loss=0.07721, simple_loss=0.1138, pruned_loss=0.01377, audio_tagging_loss=0.006561, over 15162.00 frames. ], tot_loss[loss=0.07761, simple_loss=0.09896, pruned_loss=0.01845, audio_tagging_loss=0.009671, over 3021814.97 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:34:54,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1210693.3333333333, ans=0.0
2023-11-20 20:34:55,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1210693.3333333333, ans=0.125
2023-11-20 20:35:05,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1210760.0, ans=0.125
2023-11-20 20:35:28,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1210826.6666666667, ans=0.125
2023-11-20 20:35:37,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1210893.3333333333, ans=0.1
2023-11-20 20:35:50,138 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181650
2023-11-20 20:35:50,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1210960.0, ans=0.04949747468305833
2023-11-20 20:35:50,992 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.64 vs. limit=8.0
2023-11-20 20:35:53,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1210960.0, ans=0.125
2023-11-20 20:35:56,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1211026.6666666667, ans=0.125
2023-11-20 20:35:57,736 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1300, loss[loss=0.06747, simple_loss=0.09024, pruned_loss=0.01353, audio_tagging_loss=0.008825, over 15661.00 frames. ], tot_loss[loss=0.07784, simple_loss=0.09916, pruned_loss=0.01863, audio_tagging_loss=0.009635, over 3024507.57 frames. ], batch size: 60, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:36:06,281 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.079e+01 9.042e+01 9.644e+01 1.533e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-20 20:36:09,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1211093.3333333333, ans=0.035
2023-11-20 20:36:34,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1211226.6666666667, ans=0.2
2023-11-20 20:36:53,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181700
2023-11-20 20:37:00,278 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1350, loss[loss=0.09391, simple_loss=0.1321, pruned_loss=0.02084, audio_tagging_loss=0.00703, over 15763.00 frames. ], tot_loss[loss=0.07823, simple_loss=0.09962, pruned_loss=0.01882, audio_tagging_loss=0.0096, over 3035910.40 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:37:16,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1211426.6666666667, ans=0.125
2023-11-20 20:37:46,293 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:37:56,931 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181750
2023-11-20 20:37:57,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1211626.6666666667, ans=0.125
2023-11-20 20:38:04,308 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1400, loss[loss=0.07986, simple_loss=0.0979, pruned_loss=0.01967, audio_tagging_loss=0.01124, over 14430.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.0992, pruned_loss=0.01862, audio_tagging_loss=0.009817, over 3040156.37 frames. ], batch size: 53, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:38:13,597 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.529e+01 8.254e+01 8.824e+01 9.596e+01 1.357e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 20:38:21,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1211760.0, ans=0.125
2023-11-20 20:38:25,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1211760.0, ans=0.1
2023-11-20 20:38:39,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1211826.6666666667, ans=0.125
2023-11-20 20:39:01,471 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181800
2023-11-20 20:39:08,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1212026.6666666667, ans=0.025
2023-11-20 20:39:09,392 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1450, loss[loss=0.08705, simple_loss=0.1068, pruned_loss=0.02252, audio_tagging_loss=0.01111, over 16755.00 frames. ], tot_loss[loss=0.07765, simple_loss=0.09881, pruned_loss=0.01842, audio_tagging_loss=0.009827, over 3043741.56 frames. ], batch size: 62, lr: 4.35e-03, grad_scale: 16.0
2023-11-20 20:39:13,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1212026.6666666667, ans=0.035
2023-11-20 20:39:15,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1212026.6666666667, ans=0.0
2023-11-20 20:39:35,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1212160.0, ans=0.125
2023-11-20 20:39:46,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1212226.6666666667, ans=0.0
2023-11-20 20:39:54,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1212226.6666666667, ans=6.0
2023-11-20 20:40:03,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1212293.3333333333, ans=0.1
2023-11-20 20:40:04,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.83 vs. limit=15.0
2023-11-20 20:40:05,968 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181850
2023-11-20 20:40:13,042 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1500, loss[loss=0.09372, simple_loss=0.1211, pruned_loss=0.02524, audio_tagging_loss=0.007932, over 15104.00 frames. ], tot_loss[loss=0.07812, simple_loss=0.0991, pruned_loss=0.01863, audio_tagging_loss=0.009938, over 3041296.61 frames. ], batch size: 54, lr: 4.35e-03, grad_scale: 16.0
2023-11-20 20:40:22,247 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.058e+01 8.717e+01 9.373e+01 1.477e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 20:40:40,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1212493.3333333333, ans=0.1
2023-11-20 20:40:49,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1212493.3333333333, ans=0.0
2023-11-20 20:41:08,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181900
2023-11-20 20:41:16,544 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1550, loss[loss=0.07676, simple_loss=0.105, pruned_loss=0.01663, audio_tagging_loss=0.007632, over 14158.00 frames. ], tot_loss[loss=0.07817, simple_loss=0.09903, pruned_loss=0.01865, audio_tagging_loss=0.01, over 3036842.61 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 16.0
2023-11-20 20:41:16,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1212693.3333333333, ans=0.125
2023-11-20 20:41:26,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1212693.3333333333, ans=0.05
2023-11-20 20:41:26,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.89 vs. limit=15.0
2023-11-20 20:42:13,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 181950
2023-11-20 20:42:20,297 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1600, loss[loss=0.05896, simple_loss=0.0653, pruned_loss=0.01081, audio_tagging_loss=0.01549, over 14597.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09822, pruned_loss=0.01858, audio_tagging_loss=0.01003, over 3032005.46 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:42:20,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1213026.6666666667, ans=0.0
2023-11-20 20:42:30,052 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.255e+01 8.909e+01 9.457e+01 1.581e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-20 20:42:44,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1213093.3333333333, ans=0.5
2023-11-20 20:42:44,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1213093.3333333333, ans=0.1
2023-11-20 20:42:45,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1213160.0, ans=0.1
2023-11-20 20:42:53,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1213160.0, ans=0.125
2023-11-20 20:43:13,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1213293.3333333333, ans=0.0
2023-11-20 20:43:17,351 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182000
2023-11-20 20:43:24,933 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1650, loss[loss=0.09519, simple_loss=0.1249, pruned_loss=0.0222, audio_tagging_loss=0.01055, over 16092.00 frames. ], tot_loss[loss=0.07824, simple_loss=0.09926, pruned_loss=0.01861, audio_tagging_loss=0.00999, over 3045596.94 frames. ], batch size: 58, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:43:28,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1213360.0, ans=0.0
2023-11-20 20:44:14,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1213626.6666666667, ans=0.0
2023-11-20 20:44:18,257 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:44:18,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1213626.6666666667, ans=0.0
2023-11-20 20:44:20,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182050
2023-11-20 20:44:22,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1213626.6666666667, ans=0.05
2023-11-20 20:44:28,445 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1700, loss[loss=0.06564, simple_loss=0.08051, pruned_loss=0.01525, audio_tagging_loss=0.01013, over 15377.00 frames. ], tot_loss[loss=0.07872, simple_loss=0.1, pruned_loss=0.01861, audio_tagging_loss=0.01008, over 3049189.18 frames. ], batch size: 58, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:44:36,999 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.816e+01 8.092e+01 8.985e+01 9.523e+01 1.287e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-20 20:44:38,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1213693.3333333333, ans=0.125
2023-11-20 20:44:43,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1213760.0, ans=0.1
2023-11-20 20:44:46,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1213760.0, ans=0.2
2023-11-20 20:44:53,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1213826.6666666667, ans=0.125
2023-11-20 20:45:23,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182100
2023-11-20 20:45:26,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1213960.0, ans=0.125
2023-11-20 20:45:31,264 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1750, loss[loss=0.08853, simple_loss=0.123, pruned_loss=0.01917, audio_tagging_loss=0.00788, over 15042.00 frames. ], tot_loss[loss=0.07836, simple_loss=0.09986, pruned_loss=0.01846, audio_tagging_loss=0.00997, over 3051071.55 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:45:45,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1214093.3333333333, ans=0.125
2023-11-20 20:46:18,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1214226.6666666667, ans=0.0
2023-11-20 20:46:21,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1214293.3333333333, ans=0.0
2023-11-20 20:46:27,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182150
2023-11-20 20:46:30,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1214293.3333333333, ans=0.025
2023-11-20 20:46:35,463 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1800, loss[loss=0.08564, simple_loss=0.1098, pruned_loss=0.02376, audio_tagging_loss=0.006991, over 16159.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09855, pruned_loss=0.01816, audio_tagging_loss=0.009944, over 3050326.01 frames. ], batch size: 58, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:46:43,829 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 8.260e+01 8.761e+01 9.578e+01 1.185e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 20:46:46,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1214426.6666666667, ans=0.0
2023-11-20 20:47:03,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1214493.3333333333, ans=0.125
2023-11-20 20:47:31,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182200
2023-11-20 20:47:40,218 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1850, loss[loss=0.0557, simple_loss=0.07422, pruned_loss=0.007247, audio_tagging_loss=0.01135, over 15752.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09936, pruned_loss=0.01831, audio_tagging_loss=0.009887, over 3051780.02 frames. ], batch size: 59, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:47:50,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=1214693.3333333333, ans=15.0
2023-11-20 20:47:54,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.79 vs. limit=22.5
2023-11-20 20:48:18,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1214893.3333333333, ans=0.0
2023-11-20 20:48:20,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1214893.3333333333, ans=0.125
2023-11-20 20:48:36,995 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182250
2023-11-20 20:48:44,206 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1900, loss[loss=0.1081, simple_loss=0.1359, pruned_loss=0.03098, audio_tagging_loss=0.009153, over 15536.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09863, pruned_loss=0.01816, audio_tagging_loss=0.009843, over 3045952.36 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:48:53,242 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 7.917e+01 8.580e+01 9.433e+01 1.165e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-20 20:49:11,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1215160.0, ans=0.0
2023-11-20 20:49:12,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1215160.0, ans=0.1
2023-11-20 20:49:13,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1215160.0, ans=0.125
2023-11-20 20:49:35,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1215293.3333333333, ans=0.0
2023-11-20 20:49:40,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182300
2023-11-20 20:49:48,454 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 1950, loss[loss=0.06762, simple_loss=0.07831, pruned_loss=0.01995, audio_tagging_loss=0.008512, over 13792.00 frames. ], tot_loss[loss=0.07682, simple_loss=0.0981, pruned_loss=0.01808, audio_tagging_loss=0.009684, over 3042583.68 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:49:48,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1215360.0, ans=0.1
2023-11-20 20:50:11,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1215426.6666666667, ans=0.0
2023-11-20 20:50:25,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.90 vs. limit=6.0
2023-11-20 20:50:26,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1215560.0, ans=0.2
2023-11-20 20:50:45,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182350
2023-11-20 20:50:53,457 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2000, loss[loss=0.1001, simple_loss=0.1297, pruned_loss=0.02456, audio_tagging_loss=0.01065, over 14885.00 frames. ], tot_loss[loss=0.07693, simple_loss=0.09819, pruned_loss=0.0181, audio_tagging_loss=0.009735, over 3041659.59 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:50:55,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1215693.3333333333, ans=0.125
2023-11-20 20:50:58,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1215693.3333333333, ans=0.125
2023-11-20 20:50:59,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.51 vs. limit=22.5
2023-11-20 20:51:01,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1215693.3333333333, ans=0.125
2023-11-20 20:51:01,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1215693.3333333333, ans=0.125
2023-11-20 20:51:02,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.938e+01 8.733e+01 9.578e+01 1.409e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-20 20:51:04,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1215693.3333333333, ans=0.125
2023-11-20 20:51:05,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1215760.0, ans=0.0
2023-11-20 20:51:07,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1215760.0, ans=0.125
2023-11-20 20:51:49,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1215960.0, ans=0.0
2023-11-20 20:51:50,607 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182400
2023-11-20 20:51:50,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1215960.0, ans=0.07
2023-11-20 20:51:55,082 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.89 vs. limit=10.0
2023-11-20 20:51:56,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1215960.0, ans=0.125
2023-11-20 20:51:58,156 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2050, loss[loss=0.06751, simple_loss=0.09207, pruned_loss=0.01323, audio_tagging_loss=0.008245, over 14418.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09959, pruned_loss=0.01838, audio_tagging_loss=0.009689, over 3047391.18 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:51:58,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1216026.6666666667, ans=0.125
2023-11-20 20:52:28,073 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.62 vs. limit=10.0
2023-11-20 20:52:41,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1216226.6666666667, ans=0.0
2023-11-20 20:52:44,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1216226.6666666667, ans=0.0
2023-11-20 20:52:53,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1216293.3333333333, ans=0.125
2023-11-20 20:52:54,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182450
2023-11-20 20:53:02,335 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2100, loss[loss=0.06779, simple_loss=0.08694, pruned_loss=0.01402, audio_tagging_loss=0.0103, over 15123.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09973, pruned_loss=0.01842, audio_tagging_loss=0.009627, over 3047920.49 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:53:11,018 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.190e+01 8.920e+01 9.750e+01 1.833e+02, threshold=1.784e+02, percent-clipped=1.0
2023-11-20 20:53:37,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1216493.3333333333, ans=0.0
2023-11-20 20:53:58,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182500
2023-11-20 20:53:59,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1216626.6666666667, ans=0.0
2023-11-20 20:54:03,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.60 vs. limit=22.5
2023-11-20 20:54:06,752 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2150, loss[loss=0.07004, simple_loss=0.08691, pruned_loss=0.01731, audio_tagging_loss=0.009281, over 15649.00 frames. ], tot_loss[loss=0.07827, simple_loss=0.1, pruned_loss=0.01858, audio_tagging_loss=0.009671, over 3042136.39 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:54:18,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.97 vs. limit=22.5
2023-11-20 20:54:27,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1216760.0, ans=0.125
2023-11-20 20:54:40,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1216826.6666666667, ans=0.125
2023-11-20 20:54:43,817 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:54:45,639 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.13 vs. limit=15.0
2023-11-20 20:54:57,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1216960.0, ans=0.125
2023-11-20 20:55:03,998 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182550
2023-11-20 20:55:04,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1216960.0, ans=0.0
2023-11-20 20:55:05,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1216960.0, ans=0.2
2023-11-20 20:55:11,332 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2200, loss[loss=0.07095, simple_loss=0.09868, pruned_loss=0.01428, audio_tagging_loss=0.007328, over 15977.00 frames. ], tot_loss[loss=0.07881, simple_loss=0.1006, pruned_loss=0.0188, audio_tagging_loss=0.009708, over 3039540.07 frames. ], batch size: 58, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:55:19,777 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.514e+01 9.084e+01 9.823e+01 1.256e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-20 20:55:24,718 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.26 vs. limit=15.0
2023-11-20 20:55:29,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1217093.3333333333, ans=0.125
2023-11-20 20:55:30,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1217093.3333333333, ans=0.0
2023-11-20 20:55:53,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1217226.6666666667, ans=0.07
2023-11-20 20:56:00,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1217226.6666666667, ans=0.125
2023-11-20 20:56:03,554 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.52 vs. limit=22.5
2023-11-20 20:56:07,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182600
2023-11-20 20:56:13,429 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:56:16,477 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2250, loss[loss=0.08339, simple_loss=0.1093, pruned_loss=0.01971, audio_tagging_loss=0.009014, over 15243.00 frames. ], tot_loss[loss=0.07895, simple_loss=0.1006, pruned_loss=0.01887, audio_tagging_loss=0.009752, over 3040535.65 frames. ], batch size: 59, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:56:17,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1217360.0, ans=0.0
2023-11-20 20:56:34,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.37 vs. limit=22.5
2023-11-20 20:56:39,485 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.79 vs. limit=15.0
2023-11-20 20:56:45,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.00 vs. limit=15.0
2023-11-20 20:57:00,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1217560.0, ans=0.1
2023-11-20 20:57:11,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182650
2023-11-20 20:57:17,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1217626.6666666667, ans=0.1
2023-11-20 20:57:19,141 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2300, loss[loss=0.06324, simple_loss=0.07476, pruned_loss=0.01226, audio_tagging_loss=0.01361, over 14734.00 frames. ], tot_loss[loss=0.07857, simple_loss=0.1002, pruned_loss=0.01863, audio_tagging_loss=0.009849, over 3042332.13 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 20:57:29,289 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.519e+01 8.084e+01 8.941e+01 9.693e+01 1.110e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 20:57:30,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1217693.3333333333, ans=0.125
2023-11-20 20:57:34,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1217760.0, ans=0.1
2023-11-20 20:58:04,156 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.66 vs. limit=6.0
2023-11-20 20:58:12,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1217960.0, ans=0.5
2023-11-20 20:58:15,666 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:58:16,919 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182700
2023-11-20 20:58:16,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1217960.0, ans=0.1
2023-11-20 20:58:17,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1217960.0, ans=0.125
2023-11-20 20:58:20,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1217960.0, ans=0.125
2023-11-20 20:58:23,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1218026.6666666667, ans=0.07
2023-11-20 20:58:24,654 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2350, loss[loss=0.05473, simple_loss=0.06693, pruned_loss=0.01019, audio_tagging_loss=0.01107, over 14589.00 frames. ], tot_loss[loss=0.07824, simple_loss=0.09965, pruned_loss=0.01848, audio_tagging_loss=0.009927, over 3046522.89 frames. ], batch size: 54, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 20:58:28,147 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.70 vs. limit=15.0
2023-11-20 20:58:32,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1218026.6666666667, ans=0.125
2023-11-20 20:58:36,714 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.33 vs. limit=22.5
2023-11-20 20:58:55,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1218160.0, ans=0.125
2023-11-20 20:58:58,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1218160.0, ans=0.125
2023-11-20 20:59:03,539 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.45 vs. limit=6.0
2023-11-20 20:59:19,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.67 vs. limit=15.0
2023-11-20 20:59:21,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182750
2023-11-20 20:59:28,331 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2400, loss[loss=0.06477, simple_loss=0.08118, pruned_loss=0.01439, audio_tagging_loss=0.009786, over 17338.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.09902, pruned_loss=0.01839, audio_tagging_loss=0.009967, over 3041871.11 frames. ], batch size: 65, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 20:59:31,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1218360.0, ans=0.125
2023-11-20 20:59:38,635 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.066e+01 8.791e+01 9.748e+01 1.239e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-20 20:59:39,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1218360.0, ans=0.125
2023-11-20 20:59:53,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1218493.3333333333, ans=0.125
2023-11-20 20:59:55,931 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:00:07,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=1218560.0, ans=0.05
2023-11-20 21:00:20,322 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:00:24,999 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182800
2023-11-20 21:00:32,672 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2450, loss[loss=0.06221, simple_loss=0.0738, pruned_loss=0.01534, audio_tagging_loss=0.009971, over 14367.00 frames. ], tot_loss[loss=0.07803, simple_loss=0.09952, pruned_loss=0.0183, audio_tagging_loss=0.009966, over 3046748.57 frames. ], batch size: 55, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:00:45,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1218760.0, ans=0.125
2023-11-20 21:01:00,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.96 vs. limit=15.0
2023-11-20 21:01:27,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1218960.0, ans=0.0
2023-11-20 21:01:28,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1218960.0, ans=0.125
2023-11-20 21:01:30,038 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182850
2023-11-20 21:01:37,755 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2500, loss[loss=0.07604, simple_loss=0.09182, pruned_loss=0.01843, audio_tagging_loss=0.0117, over 15089.00 frames. ], tot_loss[loss=0.07829, simple_loss=0.1, pruned_loss=0.01836, audio_tagging_loss=0.009902, over 3054175.80 frames. ], batch size: 57, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:01:48,243 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 7.891e+01 8.728e+01 9.490e+01 1.260e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-20 21:01:57,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1219093.3333333333, ans=0.125
2023-11-20 21:02:24,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.21 vs. limit=15.0
2023-11-20 21:02:30,504 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.35 vs. limit=10.0
2023-11-20 21:02:35,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182900
2023-11-20 21:02:39,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1219293.3333333333, ans=0.0
2023-11-20 21:02:42,993 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2550, loss[loss=0.07492, simple_loss=0.08437, pruned_loss=0.0244, audio_tagging_loss=0.008328, over 13976.00 frames. ], tot_loss[loss=0.07841, simple_loss=0.1001, pruned_loss=0.0185, audio_tagging_loss=0.009848, over 3047789.99 frames. ], batch size: 55, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:02:44,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1219360.0, ans=0.2
2023-11-20 21:02:49,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1219360.0, ans=0.125
2023-11-20 21:02:53,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1219360.0, ans=0.5
2023-11-20 21:02:55,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1219426.6666666667, ans=0.0
2023-11-20 21:03:14,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1219493.3333333333, ans=0.125
2023-11-20 21:03:30,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1219560.0, ans=0.1
2023-11-20 21:03:39,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1219626.6666666667, ans=0.125
2023-11-20 21:03:40,334 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 182950
2023-11-20 21:03:47,485 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2600, loss[loss=0.08594, simple_loss=0.1089, pruned_loss=0.02097, audio_tagging_loss=0.01051, over 15531.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.09879, pruned_loss=0.0183, audio_tagging_loss=0.009775, over 3049430.75 frames. ], batch size: 55, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:03:48,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1219693.3333333333, ans=0.125
2023-11-20 21:03:55,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1219693.3333333333, ans=0.1
2023-11-20 21:03:57,814 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.668e+01 8.265e+01 9.445e+01 1.045e+02 1.287e+02, threshold=1.889e+02, percent-clipped=0.0
2023-11-20 21:04:06,033 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.117e-01
2023-11-20 21:04:11,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1219760.0, ans=0.0
2023-11-20 21:04:23,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1219826.6666666667, ans=0.2
2023-11-20 21:04:36,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1219893.3333333333, ans=0.125
2023-11-20 21:04:40,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1219960.0, ans=0.0
2023-11-20 21:04:43,946 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183000
2023-11-20 21:04:51,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1220026.6666666667, ans=0.125
2023-11-20 21:04:52,362 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2650, loss[loss=0.07882, simple_loss=0.09496, pruned_loss=0.01821, audio_tagging_loss=0.01313, over 13877.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09939, pruned_loss=0.0185, audio_tagging_loss=0.009687, over 3046448.55 frames. ], batch size: 54, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:05:11,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.80 vs. limit=12.0
2023-11-20 21:05:19,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1220160.0, ans=0.0
2023-11-20 21:05:22,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1220160.0, ans=0.2
2023-11-20 21:05:48,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183050
2023-11-20 21:05:56,273 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2700, loss[loss=0.07179, simple_loss=0.09561, pruned_loss=0.01469, audio_tagging_loss=0.009297, over 15612.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09845, pruned_loss=0.01822, audio_tagging_loss=0.009664, over 3042619.56 frames. ], batch size: 59, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:05:59,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.18 vs. limit=15.0
2023-11-20 21:06:06,369 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.355e+01 9.034e+01 9.882e+01 1.379e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-20 21:06:13,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1220426.6666666667, ans=0.125
2023-11-20 21:06:36,289 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.60 vs. limit=12.0
2023-11-20 21:06:38,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1220560.0, ans=0.2
2023-11-20 21:06:52,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183100
2023-11-20 21:06:56,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1220626.6666666667, ans=0.125
2023-11-20 21:06:58,003 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.96 vs. limit=15.0
2023-11-20 21:06:59,893 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2750, loss[loss=0.06308, simple_loss=0.07803, pruned_loss=0.01371, audio_tagging_loss=0.01035, over 14797.00 frames. ], tot_loss[loss=0.07686, simple_loss=0.09796, pruned_loss=0.01823, audio_tagging_loss=0.009652, over 3041579.58 frames. ], batch size: 58, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:07:08,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1220693.3333333333, ans=0.125
2023-11-20 21:07:14,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1220760.0, ans=0.0
2023-11-20 21:07:24,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1220826.6666666667, ans=0.1
2023-11-20 21:07:32,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1220826.6666666667, ans=0.04949747468305833
2023-11-20 21:07:36,810 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:07:52,654 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:07:56,339 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183150
2023-11-20 21:08:04,598 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2800, loss[loss=0.05402, simple_loss=0.06945, pruned_loss=0.00945, audio_tagging_loss=0.009846, over 14381.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.09751, pruned_loss=0.01822, audio_tagging_loss=0.009663, over 3044119.94 frames. ], batch size: 54, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:08:07,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1221026.6666666667, ans=0.1
2023-11-20 21:08:15,663 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.121e+01 8.798e+01 9.627e+01 1.849e+02, threshold=1.760e+02, percent-clipped=1.0
2023-11-20 21:08:16,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1221093.3333333333, ans=0.125
2023-11-20 21:08:20,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1221093.3333333333, ans=0.0
2023-11-20 21:08:40,547 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:08:42,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.91 vs. limit=22.5
2023-11-20 21:08:57,906 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:09:01,376 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183200
2023-11-20 21:09:03,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1221293.3333333333, ans=10.0
2023-11-20 21:09:08,975 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2850, loss[loss=0.06607, simple_loss=0.07844, pruned_loss=0.01544, audio_tagging_loss=0.01141, over 15495.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09669, pruned_loss=0.01806, audio_tagging_loss=0.009706, over 3047105.05 frames. ], batch size: 59, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:09:20,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.81 vs. limit=15.0
2023-11-20 21:09:31,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1221426.6666666667, ans=0.125
2023-11-20 21:09:44,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1221493.3333333333, ans=0.025
2023-11-20 21:10:05,972 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183250
2023-11-20 21:10:13,755 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2900, loss[loss=0.06523, simple_loss=0.07917, pruned_loss=0.01536, audio_tagging_loss=0.01029, over 15072.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09603, pruned_loss=0.01793, audio_tagging_loss=0.009776, over 3042737.05 frames. ], batch size: 60, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:10:16,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1221693.3333333333, ans=0.125
2023-11-20 21:10:22,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1221693.3333333333, ans=0.2
2023-11-20 21:10:26,047 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.821e+01 8.647e+01 9.509e+01 1.495e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-20 21:10:27,466 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:10:34,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1221760.0, ans=0.1
2023-11-20 21:10:42,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1221826.6666666667, ans=0.125
2023-11-20 21:11:09,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183300
2023-11-20 21:11:17,175 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 2950, loss[loss=0.07417, simple_loss=0.09669, pruned_loss=0.01792, audio_tagging_loss=0.007904, over 16079.00 frames. ], tot_loss[loss=0.07667, simple_loss=0.09757, pruned_loss=0.01817, audio_tagging_loss=0.009723, over 3049327.65 frames. ], batch size: 62, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:11:40,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1222093.3333333333, ans=0.125
2023-11-20 21:12:13,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183350
2023-11-20 21:12:15,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1222293.3333333333, ans=0.125
2023-11-20 21:12:21,092 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3000, loss[loss=0.09827, simple_loss=0.1237, pruned_loss=0.02477, audio_tagging_loss=0.01165, over 15400.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.0989, pruned_loss=0.01875, audio_tagging_loss=0.00978, over 3047913.96 frames. ], batch size: 55, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:12:21,093 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 21:12:37,239 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.7988, 3.0857, 2.8142, 3.4745, 3.1773, 2.8778, 3.1474, 2.8904],
       device='cuda:2')
2023-11-20 21:12:44,240 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8660, 3.3601, 4.7930, 4.4126], device='cuda:2')
2023-11-20 21:12:59,447 INFO [train_asr.py:1253] (2/4) Epoch 16, validation: loss=0.06057, simple_loss=0.053, pruned_loss=0.005481, audio_tagging_loss=0.02859, over 4681554.00 frames. 
2023-11-20 21:12:59,448 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 21:13:12,785 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.173e+01 8.695e+01 9.591e+01 1.296e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-20 21:13:26,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1222493.3333333333, ans=0.125
2023-11-20 21:13:31,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1222493.3333333333, ans=0.125
2023-11-20 21:13:49,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1222626.6666666667, ans=0.125
2023-11-20 21:13:50,009 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.92 vs. limit=15.0
2023-11-20 21:13:55,695 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183400
2023-11-20 21:14:04,217 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3050, loss[loss=0.05582, simple_loss=0.06699, pruned_loss=0.00944, audio_tagging_loss=0.01288, over 16203.00 frames. ], tot_loss[loss=0.07767, simple_loss=0.09857, pruned_loss=0.01854, audio_tagging_loss=0.009839, over 3042934.28 frames. ], batch size: 63, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:14:21,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1222760.0, ans=0.025
2023-11-20 21:14:26,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1222760.0, ans=0.1
2023-11-20 21:14:39,815 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:14:49,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1222893.3333333333, ans=0.125
2023-11-20 21:15:00,543 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183450
2023-11-20 21:15:01,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1222960.0, ans=0.0
2023-11-20 21:15:07,761 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3100, loss[loss=0.06462, simple_loss=0.07534, pruned_loss=0.01528, audio_tagging_loss=0.01168, over 15254.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09924, pruned_loss=0.01862, audio_tagging_loss=0.009911, over 3041361.81 frames. ], batch size: 58, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:15:14,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1223026.6666666667, ans=0.125
2023-11-20 21:15:18,207 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:15:20,359 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.321e+01 8.878e+01 9.736e+01 1.416e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 21:15:42,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1223160.0, ans=0.1
2023-11-20 21:16:03,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183500
2023-11-20 21:16:11,732 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3150, loss[loss=0.06465, simple_loss=0.08354, pruned_loss=0.01622, audio_tagging_loss=0.006663, over 14998.00 frames. ], tot_loss[loss=0.07843, simple_loss=0.09955, pruned_loss=0.01873, audio_tagging_loss=0.009925, over 3047216.21 frames. ], batch size: 58, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:16:12,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1223360.0, ans=0.125
2023-11-20 21:16:14,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1223360.0, ans=0.125
2023-11-20 21:16:18,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1223360.0, ans=0.125
2023-11-20 21:16:18,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1223360.0, ans=0.2
2023-11-20 21:16:26,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1223426.6666666667, ans=0.125
2023-11-20 21:16:30,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1223426.6666666667, ans=0.125
2023-11-20 21:16:48,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1223560.0, ans=0.0
2023-11-20 21:17:07,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183550
2023-11-20 21:17:16,812 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3200, loss[loss=0.07064, simple_loss=0.08973, pruned_loss=0.01577, audio_tagging_loss=0.01001, over 16346.00 frames. ], tot_loss[loss=0.07857, simple_loss=0.09973, pruned_loss=0.01873, audio_tagging_loss=0.009974, over 3040468.93 frames. ], batch size: 61, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:17:24,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1223693.3333333333, ans=0.125
2023-11-20 21:17:28,828 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.173e+01 8.835e+01 9.713e+01 1.308e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-20 21:17:32,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1223760.0, ans=0.125
2023-11-20 21:17:54,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1223893.3333333333, ans=0.125
2023-11-20 21:17:56,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.99 vs. limit=22.5
2023-11-20 21:18:07,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1223960.0, ans=0.0
2023-11-20 21:18:12,582 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183600
2023-11-20 21:18:20,305 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3250, loss[loss=0.08519, simple_loss=0.1017, pruned_loss=0.02331, audio_tagging_loss=0.01102, over 14567.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09909, pruned_loss=0.01854, audio_tagging_loss=0.01004, over 3044858.30 frames. ], batch size: 55, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:18:36,490 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.53 vs. limit=12.0
2023-11-20 21:18:39,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1224093.3333333333, ans=0.125
2023-11-20 21:18:44,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1224160.0, ans=0.1
2023-11-20 21:18:46,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1224160.0, ans=0.2
2023-11-20 21:18:51,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1224160.0, ans=0.0
2023-11-20 21:18:55,254 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.25 vs. limit=15.0
2023-11-20 21:19:16,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183650
2023-11-20 21:19:21,674 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.54 vs. limit=10.0
2023-11-20 21:19:24,570 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3300, loss[loss=0.08209, simple_loss=0.111, pruned_loss=0.01648, audio_tagging_loss=0.0101, over 16154.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09912, pruned_loss=0.01834, audio_tagging_loss=0.01003, over 3055535.81 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:19:38,005 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.036e+01 8.648e+01 9.640e+01 1.721e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 21:19:43,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1224426.6666666667, ans=0.2
2023-11-20 21:19:50,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1224493.3333333333, ans=0.125
2023-11-20 21:19:54,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1224493.3333333333, ans=0.0
2023-11-20 21:19:59,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1224493.3333333333, ans=0.125
2023-11-20 21:20:20,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183700
2023-11-20 21:20:25,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1224626.6666666667, ans=0.125
2023-11-20 21:20:28,190 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3350, loss[loss=0.0714, simple_loss=0.08503, pruned_loss=0.01834, audio_tagging_loss=0.01054, over 15537.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09903, pruned_loss=0.01838, audio_tagging_loss=0.01001, over 3057569.43 frames. ], batch size: 58, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:20:35,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1224693.3333333333, ans=0.0
2023-11-20 21:20:49,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1224760.0, ans=0.0
2023-11-20 21:21:20,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1224960.0, ans=0.2
2023-11-20 21:21:26,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183750
2023-11-20 21:21:33,823 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3400, loss[loss=0.07937, simple_loss=0.09958, pruned_loss=0.02069, audio_tagging_loss=0.008897, over 15078.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09885, pruned_loss=0.01841, audio_tagging_loss=0.009951, over 3054462.55 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:21:38,116 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0
2023-11-20 21:21:47,180 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.496e+01 8.162e+01 8.769e+01 9.534e+01 1.233e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-20 21:22:16,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1225226.6666666667, ans=0.125
2023-11-20 21:22:20,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1225226.6666666667, ans=0.125
2023-11-20 21:22:29,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183800
2023-11-20 21:22:38,207 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3450, loss[loss=0.1029, simple_loss=0.1412, pruned_loss=0.02377, audio_tagging_loss=0.008505, over 16590.00 frames. ], tot_loss[loss=0.07883, simple_loss=0.1005, pruned_loss=0.0188, audio_tagging_loss=0.009786, over 3052073.83 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:22:44,836 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.00 vs. limit=6.0
2023-11-20 21:22:47,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.31 vs. limit=10.0
2023-11-20 21:22:49,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1225426.6666666667, ans=0.125
2023-11-20 21:22:54,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1225426.6666666667, ans=0.125
2023-11-20 21:23:09,871 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.22 vs. limit=22.5
2023-11-20 21:23:21,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1225560.0, ans=0.125
2023-11-20 21:23:32,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1225626.6666666667, ans=0.1
2023-11-20 21:23:34,411 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183850
2023-11-20 21:23:41,720 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3500, loss[loss=0.08655, simple_loss=0.1046, pruned_loss=0.01965, audio_tagging_loss=0.01462, over 15763.00 frames. ], tot_loss[loss=0.07785, simple_loss=0.09895, pruned_loss=0.01845, audio_tagging_loss=0.009922, over 3045176.63 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:23:48,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1225693.3333333333, ans=0.1
2023-11-20 21:23:57,069 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.353e+01 9.150e+01 1.014e+02 1.535e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-20 21:23:57,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1225760.0, ans=0.035
2023-11-20 21:23:58,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1225760.0, ans=0.2
2023-11-20 21:24:07,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1225826.6666666667, ans=0.0
2023-11-20 21:24:07,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1225826.6666666667, ans=0.0
2023-11-20 21:24:13,351 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:24:26,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1225893.3333333333, ans=0.0
2023-11-20 21:24:34,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1225960.0, ans=0.125
2023-11-20 21:24:39,931 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183900
2023-11-20 21:24:41,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1225960.0, ans=0.2
2023-11-20 21:24:47,800 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3550, loss[loss=0.06921, simple_loss=0.0868, pruned_loss=0.01603, audio_tagging_loss=0.009782, over 15700.00 frames. ], tot_loss[loss=0.07686, simple_loss=0.09759, pruned_loss=0.01821, audio_tagging_loss=0.009852, over 3051915.34 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:24:53,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1226026.6666666667, ans=0.125
2023-11-20 21:24:53,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1226026.6666666667, ans=0.2
2023-11-20 21:25:44,363 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 183950
2023-11-20 21:25:45,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1226293.3333333333, ans=0.0
2023-11-20 21:25:51,797 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3600, loss[loss=0.04681, simple_loss=0.06241, pruned_loss=0.005945, audio_tagging_loss=0.009663, over 14713.00 frames. ], tot_loss[loss=0.07624, simple_loss=0.09676, pruned_loss=0.01808, audio_tagging_loss=0.009781, over 3045462.31 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:25:55,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1226360.0, ans=0.125
2023-11-20 21:26:01,584 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.37 vs. limit=22.5
2023-11-20 21:26:05,873 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.147e+01 8.022e+01 8.880e+01 9.578e+01 1.162e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 21:26:13,353 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.77 vs. limit=15.0
2023-11-20 21:26:39,728 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:26:48,803 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184000
2023-11-20 21:26:48,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1226626.6666666667, ans=0.125
2023-11-20 21:27:00,014 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3650, loss[loss=0.06614, simple_loss=0.08628, pruned_loss=0.01285, audio_tagging_loss=0.01015, over 14745.00 frames. ], tot_loss[loss=0.07584, simple_loss=0.09628, pruned_loss=0.01798, audio_tagging_loss=0.009721, over 3041406.66 frames. ], batch size: 55, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:27:11,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1226693.3333333333, ans=0.025
2023-11-20 21:27:15,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1226760.0, ans=0.125
2023-11-20 21:27:24,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1226760.0, ans=0.0
2023-11-20 21:27:37,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1226826.6666666667, ans=0.125
2023-11-20 21:27:42,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1226893.3333333333, ans=0.125
2023-11-20 21:27:55,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1226960.0, ans=0.125
2023-11-20 21:27:57,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184050
2023-11-20 21:28:05,153 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3700, loss[loss=0.05516, simple_loss=0.06479, pruned_loss=0.01118, audio_tagging_loss=0.01158, over 15186.00 frames. ], tot_loss[loss=0.07673, simple_loss=0.09759, pruned_loss=0.01827, audio_tagging_loss=0.009659, over 3047033.35 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:28:06,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1227026.6666666667, ans=0.09899494936611666
2023-11-20 21:28:13,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1227026.6666666667, ans=0.2
2023-11-20 21:28:14,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1227026.6666666667, ans=0.125
2023-11-20 21:28:16,175 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.34 vs. limit=15.0
2023-11-20 21:28:19,424 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.009e+01 8.576e+01 9.316e+01 1.305e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-20 21:28:28,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1227093.3333333333, ans=15.0
2023-11-20 21:28:41,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1227160.0, ans=0.2
2023-11-20 21:28:52,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.93 vs. limit=15.0
2023-11-20 21:29:02,766 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184100
2023-11-20 21:29:06,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1227293.3333333333, ans=0.125
2023-11-20 21:29:10,245 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3750, loss[loss=0.06279, simple_loss=0.08049, pruned_loss=0.01188, audio_tagging_loss=0.01067, over 16160.00 frames. ], tot_loss[loss=0.07733, simple_loss=0.09851, pruned_loss=0.01846, audio_tagging_loss=0.009618, over 3049392.65 frames. ], batch size: 63, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:29:39,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1227493.3333333333, ans=0.0
2023-11-20 21:29:49,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1227560.0, ans=0.1
2023-11-20 21:29:52,873 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:30:05,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1227626.6666666667, ans=0.0
2023-11-20 21:30:06,707 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184150
2023-11-20 21:30:10,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1227626.6666666667, ans=0.125
2023-11-20 21:30:13,953 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3800, loss[loss=0.07668, simple_loss=0.09472, pruned_loss=0.01612, audio_tagging_loss=0.0132, over 15132.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.09901, pruned_loss=0.01848, audio_tagging_loss=0.009729, over 3052636.47 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:30:16,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1227693.3333333333, ans=0.0
2023-11-20 21:30:19,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1227693.3333333333, ans=0.09899494936611666
2023-11-20 21:30:29,923 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.193e+01 8.281e+01 9.046e+01 9.670e+01 1.407e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-20 21:30:43,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.16 vs. limit=22.5
2023-11-20 21:30:56,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1227893.3333333333, ans=0.125
2023-11-20 21:31:05,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1227960.0, ans=0.0
2023-11-20 21:31:09,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.31 vs. limit=15.0
2023-11-20 21:31:11,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184200
2023-11-20 21:31:11,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1227960.0, ans=0.0
2023-11-20 21:31:11,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1227960.0, ans=0.0
2023-11-20 21:31:18,623 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3850, loss[loss=0.08002, simple_loss=0.09892, pruned_loss=0.01983, audio_tagging_loss=0.01073, over 13965.00 frames. ], tot_loss[loss=0.07773, simple_loss=0.09907, pruned_loss=0.01841, audio_tagging_loss=0.009781, over 3051762.65 frames. ], batch size: 53, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:31:39,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1228093.3333333333, ans=0.0
2023-11-20 21:31:41,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1228093.3333333333, ans=0.1
2023-11-20 21:31:45,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1228160.0, ans=0.125
2023-11-20 21:31:59,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1228226.6666666667, ans=0.125
2023-11-20 21:32:14,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1228293.3333333333, ans=0.0
2023-11-20 21:32:15,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184250
2023-11-20 21:32:23,135 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3900, loss[loss=0.06311, simple_loss=0.07935, pruned_loss=0.01405, audio_tagging_loss=0.009378, over 14984.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09845, pruned_loss=0.01838, audio_tagging_loss=0.009868, over 3045248.60 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:32:35,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1228426.6666666667, ans=0.1
2023-11-20 21:32:36,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1228426.6666666667, ans=0.1
2023-11-20 21:32:38,161 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.176e+01 8.890e+01 9.815e+01 1.146e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-20 21:32:47,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1228493.3333333333, ans=0.125
2023-11-20 21:32:50,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1228493.3333333333, ans=0.0
2023-11-20 21:32:50,566 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.39 vs. limit=22.5
2023-11-20 21:33:20,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184300
2023-11-20 21:33:27,293 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.34 vs. limit=6.0
2023-11-20 21:33:27,626 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 3950, loss[loss=0.08002, simple_loss=0.09633, pruned_loss=0.02014, audio_tagging_loss=0.01171, over 15375.00 frames. ], tot_loss[loss=0.07773, simple_loss=0.09872, pruned_loss=0.01851, audio_tagging_loss=0.009859, over 3052722.81 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:33:29,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=1228693.3333333333, ans=0.02
2023-11-20 21:33:37,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1228693.3333333333, ans=0.09899494936611666
2023-11-20 21:33:51,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1228760.0, ans=0.125
2023-11-20 21:33:56,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1228826.6666666667, ans=0.125
2023-11-20 21:34:08,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1228893.3333333333, ans=0.125
2023-11-20 21:34:17,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1228893.3333333333, ans=0.2
2023-11-20 21:34:22,021 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.492e-01
2023-11-20 21:34:24,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184350
2023-11-20 21:34:32,585 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4000, loss[loss=0.06426, simple_loss=0.08466, pruned_loss=0.01254, audio_tagging_loss=0.009393, over 14875.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09877, pruned_loss=0.01848, audio_tagging_loss=0.009921, over 3055356.19 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 32.0
2023-11-20 21:34:39,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.59 vs. limit=10.0
2023-11-20 21:34:41,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.44 vs. limit=15.0
2023-11-20 21:34:47,102 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.174e+01 8.766e+01 9.644e+01 1.258e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-20 21:35:14,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1229226.6666666667, ans=0.125
2023-11-20 21:35:28,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184400
2023-11-20 21:35:36,134 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4050, loss[loss=0.0862, simple_loss=0.1074, pruned_loss=0.02259, audio_tagging_loss=0.009929, over 16469.00 frames. ], tot_loss[loss=0.07766, simple_loss=0.09867, pruned_loss=0.0183, audio_tagging_loss=0.01003, over 3045862.05 frames. ], batch size: 63, lr: 4.32e-03, grad_scale: 32.0
2023-11-20 21:35:37,457 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:35:40,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1229360.0, ans=0.2
2023-11-20 21:35:41,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1229360.0, ans=0.1
2023-11-20 21:35:48,641 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.14 vs. limit=15.0
2023-11-20 21:35:49,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1229426.6666666667, ans=0.0
2023-11-20 21:35:50,008 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.48 vs. limit=15.0
2023-11-20 21:36:06,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1229493.3333333333, ans=0.0
2023-11-20 21:36:14,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1229560.0, ans=0.125
2023-11-20 21:36:22,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=15.0
2023-11-20 21:36:26,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1229626.6666666667, ans=0.0
2023-11-20 21:36:32,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184450
2023-11-20 21:36:35,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1229626.6666666667, ans=0.1
2023-11-20 21:36:37,711 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.33 vs. limit=22.5
2023-11-20 21:36:38,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1229626.6666666667, ans=0.0
2023-11-20 21:36:40,468 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4100, loss[loss=0.08793, simple_loss=0.1174, pruned_loss=0.02258, audio_tagging_loss=0.006644, over 16587.00 frames. ], tot_loss[loss=0.07869, simple_loss=0.1005, pruned_loss=0.01851, audio_tagging_loss=0.009937, over 3044768.64 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:36:50,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1229693.3333333333, ans=0.125
2023-11-20 21:36:57,519 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.177e+01 8.908e+01 9.687e+01 1.316e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-20 21:37:05,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1229826.6666666667, ans=0.125
2023-11-20 21:37:07,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1229826.6666666667, ans=0.0
2023-11-20 21:37:36,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184500
2023-11-20 21:37:42,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1229960.0, ans=0.04949747468305833
2023-11-20 21:37:44,731 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4150, loss[loss=0.05705, simple_loss=0.06761, pruned_loss=0.01142, audio_tagging_loss=0.01182, over 15018.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.1006, pruned_loss=0.01854, audio_tagging_loss=0.009801, over 3051910.98 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:37:48,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1230026.6666666667, ans=0.0
2023-11-20 21:37:50,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1230026.6666666667, ans=0.1
2023-11-20 21:37:50,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1230026.6666666667, ans=0.1
2023-11-20 21:37:53,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1230026.6666666667, ans=0.125
2023-11-20 21:37:56,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1230093.3333333333, ans=0.2
2023-11-20 21:37:59,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1230093.3333333333, ans=0.0
2023-11-20 21:38:00,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1230093.3333333333, ans=0.1
2023-11-20 21:38:28,761 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:38:32,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.27 vs. limit=10.0
2023-11-20 21:38:36,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1230293.3333333333, ans=0.0
2023-11-20 21:38:42,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184550
2023-11-20 21:38:49,332 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4200, loss[loss=0.1087, simple_loss=0.1444, pruned_loss=0.02737, audio_tagging_loss=0.009104, over 15100.00 frames. ], tot_loss[loss=0.07812, simple_loss=0.09987, pruned_loss=0.0184, audio_tagging_loss=0.00978, over 3048109.78 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:38:52,446 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=23.22 vs. limit=22.5
2023-11-20 21:39:05,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.99 vs. limit=15.0
2023-11-20 21:39:07,576 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.428e+01 8.014e+01 8.553e+01 9.363e+01 1.203e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-20 21:39:26,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1230493.3333333333, ans=0.0
2023-11-20 21:39:27,603 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=15.0
2023-11-20 21:39:40,011 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.38 vs. limit=15.0
2023-11-20 21:39:41,905 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:39:45,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184600
2023-11-20 21:39:46,251 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.59 vs. limit=10.0
2023-11-20 21:39:53,578 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4250, loss[loss=0.07706, simple_loss=0.0959, pruned_loss=0.01907, audio_tagging_loss=0.01004, over 13932.00 frames. ], tot_loss[loss=0.07748, simple_loss=0.09902, pruned_loss=0.01822, audio_tagging_loss=0.009757, over 3044602.81 frames. ], batch size: 53, lr: 4.32e-03, grad_scale: 8.0
2023-11-20 21:40:14,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1230760.0, ans=0.1
2023-11-20 21:40:19,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1230826.6666666667, ans=15.0
2023-11-20 21:40:23,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1230826.6666666667, ans=0.0
2023-11-20 21:40:50,036 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184650
2023-11-20 21:40:50,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1230960.0, ans=0.125
2023-11-20 21:40:57,901 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4300, loss[loss=0.07302, simple_loss=0.09766, pruned_loss=0.01345, audio_tagging_loss=0.01075, over 15163.00 frames. ], tot_loss[loss=0.07748, simple_loss=0.09926, pruned_loss=0.01818, audio_tagging_loss=0.00967, over 3048364.42 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 8.0
2023-11-20 21:41:15,300 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.115e+01 8.273e+01 8.996e+01 9.641e+01 1.140e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-20 21:41:29,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1231160.0, ans=0.1
2023-11-20 21:41:54,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184700
2023-11-20 21:41:56,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1231293.3333333333, ans=0.1
2023-11-20 21:41:59,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1231293.3333333333, ans=0.0
2023-11-20 21:42:01,376 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4350, loss[loss=0.07718, simple_loss=0.09337, pruned_loss=0.02176, audio_tagging_loss=0.008732, over 14584.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09975, pruned_loss=0.01822, audio_tagging_loss=0.009692, over 3053185.67 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 8.0
2023-11-20 21:42:03,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.94 vs. limit=15.0
2023-11-20 21:42:14,685 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.09 vs. limit=6.0
2023-11-20 21:42:17,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.78 vs. limit=22.5
2023-11-20 21:42:25,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1231493.3333333333, ans=0.125
2023-11-20 21:42:26,087 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.19 vs. limit=15.0
2023-11-20 21:42:49,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1231560.0, ans=0.0
2023-11-20 21:42:50,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1231560.0, ans=0.125
2023-11-20 21:42:57,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184750
2023-11-20 21:43:05,111 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4400, loss[loss=0.06622, simple_loss=0.0924, pruned_loss=0.01244, audio_tagging_loss=0.007579, over 15201.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09979, pruned_loss=0.01829, audio_tagging_loss=0.009607, over 3051549.13 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:43:10,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1231693.3333333333, ans=0.0
2023-11-20 21:43:22,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1231760.0, ans=0.05
2023-11-20 21:43:23,440 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.140e+01 8.871e+01 9.490e+01 1.181e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-20 21:43:26,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1231760.0, ans=0.0
2023-11-20 21:43:47,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.66 vs. limit=12.0
2023-11-20 21:43:54,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1231893.3333333333, ans=0.125
2023-11-20 21:43:59,337 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:44:01,682 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184800
2023-11-20 21:44:09,949 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4450, loss[loss=0.0776, simple_loss=0.1028, pruned_loss=0.01705, audio_tagging_loss=0.009144, over 15400.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.09997, pruned_loss=0.01842, audio_tagging_loss=0.009582, over 3054470.58 frames. ], batch size: 57, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:44:10,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=1232026.6666666667, ans=0.02
2023-11-20 21:44:13,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1232026.6666666667, ans=0.04949747468305833
2023-11-20 21:44:31,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1232093.3333333333, ans=0.2
2023-11-20 21:44:45,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1232160.0, ans=0.125
2023-11-20 21:44:56,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1232226.6666666667, ans=0.125
2023-11-20 21:45:07,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184850
2023-11-20 21:45:11,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1232293.3333333333, ans=0.0
2023-11-20 21:45:13,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.80 vs. limit=22.5
2023-11-20 21:45:15,008 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4500, loss[loss=0.07024, simple_loss=0.09679, pruned_loss=0.01282, audio_tagging_loss=0.00903, over 14684.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09983, pruned_loss=0.01827, audio_tagging_loss=0.009576, over 3051475.51 frames. ], batch size: 57, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:45:25,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.34 vs. limit=6.0
2023-11-20 21:45:32,799 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.142e+01 8.902e+01 9.690e+01 1.395e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-20 21:45:38,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1232426.6666666667, ans=0.0
2023-11-20 21:45:38,376 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.87 vs. limit=15.0
2023-11-20 21:45:40,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1232493.3333333333, ans=0.0
2023-11-20 21:46:03,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1232560.0, ans=0.125
2023-11-20 21:46:10,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1232626.6666666667, ans=0.125
2023-11-20 21:46:11,413 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184900
2023-11-20 21:46:18,631 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4550, loss[loss=0.08194, simple_loss=0.1109, pruned_loss=0.01653, audio_tagging_loss=0.009974, over 14750.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09905, pruned_loss=0.01802, audio_tagging_loss=0.009611, over 3047683.70 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:46:24,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1232693.3333333333, ans=0.125
2023-11-20 21:46:57,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1232893.3333333333, ans=0.125
2023-11-20 21:46:59,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1232893.3333333333, ans=0.0
2023-11-20 21:47:01,836 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:47:04,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1232893.3333333333, ans=0.125
2023-11-20 21:47:05,290 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:47:15,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 184950
2023-11-20 21:47:22,240 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.48 vs. limit=22.5
2023-11-20 21:47:22,847 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4600, loss[loss=0.06559, simple_loss=0.07447, pruned_loss=0.01284, audio_tagging_loss=0.01552, over 15132.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09894, pruned_loss=0.01822, audio_tagging_loss=0.009764, over 3050089.83 frames. ], batch size: 57, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:47:29,129 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=5.94 vs. limit=15.0
2023-11-20 21:47:32,270 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:47:40,086 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:47:40,995 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.813e+01 7.895e+01 8.619e+01 9.471e+01 1.250e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-20 21:47:42,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1233093.3333333333, ans=0.0
2023-11-20 21:48:18,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1233293.3333333333, ans=0.0
2023-11-20 21:48:19,134 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185000
2023-11-20 21:48:27,604 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4650, loss[loss=0.06175, simple_loss=0.0858, pruned_loss=0.0079, audio_tagging_loss=0.01095, over 15066.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09919, pruned_loss=0.01823, audio_tagging_loss=0.009772, over 3051388.12 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:49:13,747 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.90 vs. limit=15.0
2023-11-20 21:49:23,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185050
2023-11-20 21:49:30,667 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4700, loss[loss=0.097, simple_loss=0.1202, pruned_loss=0.02822, audio_tagging_loss=0.008669, over 15469.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09946, pruned_loss=0.01853, audio_tagging_loss=0.009961, over 3053743.64 frames. ], batch size: 54, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:49:36,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1233693.3333333333, ans=0.125
2023-11-20 21:49:41,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1233693.3333333333, ans=0.125
2023-11-20 21:49:48,745 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.027e+01 8.804e+01 9.916e+01 1.555e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-20 21:50:15,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.20 vs. limit=15.0
2023-11-20 21:50:28,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185100
2023-11-20 21:50:35,792 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4750, loss[loss=0.08416, simple_loss=0.1127, pruned_loss=0.01962, audio_tagging_loss=0.008165, over 14718.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.09979, pruned_loss=0.01866, audio_tagging_loss=0.01008, over 3054628.69 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:50:54,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1234093.3333333333, ans=0.0
2023-11-20 21:51:23,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1234226.6666666667, ans=0.2
2023-11-20 21:51:32,329 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185150
2023-11-20 21:51:34,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1234293.3333333333, ans=0.125
2023-11-20 21:51:36,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1234293.3333333333, ans=0.125
2023-11-20 21:51:39,959 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4800, loss[loss=0.07499, simple_loss=0.09684, pruned_loss=0.01738, audio_tagging_loss=0.009186, over 14199.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.0985, pruned_loss=0.01835, audio_tagging_loss=0.01014, over 3049081.34 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 32.0
2023-11-20 21:51:40,526 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.32 vs. limit=22.5
2023-11-20 21:51:42,137 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.93 vs. limit=15.0
2023-11-20 21:51:58,550 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.531e+01 9.078e+01 1.027e+02 1.282e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-20 21:52:03,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1234426.6666666667, ans=0.125
2023-11-20 21:52:35,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185200
2023-11-20 21:52:44,113 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4850, loss[loss=0.0693, simple_loss=0.0891, pruned_loss=0.01354, audio_tagging_loss=0.01121, over 15630.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.09876, pruned_loss=0.01833, audio_tagging_loss=0.01027, over 3055028.07 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:52:52,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1234693.3333333333, ans=0.1
2023-11-20 21:52:53,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1234693.3333333333, ans=0.07
2023-11-20 21:52:58,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1234760.0, ans=0.1
2023-11-20 21:53:07,087 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.79 vs. limit=22.5
2023-11-20 21:53:38,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1234960.0, ans=0.125
2023-11-20 21:53:40,498 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185250
2023-11-20 21:53:44,556 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.63 vs. limit=22.5
2023-11-20 21:53:47,770 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4900, loss[loss=0.07088, simple_loss=0.08669, pruned_loss=0.02057, audio_tagging_loss=0.006959, over 15042.00 frames. ], tot_loss[loss=0.0785, simple_loss=0.09966, pruned_loss=0.01855, audio_tagging_loss=0.01012, over 3052095.76 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:53:53,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.73 vs. limit=10.0
2023-11-20 21:54:01,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1235093.3333333333, ans=0.125
2023-11-20 21:54:06,864 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.089e+01 8.842e+01 9.821e+01 1.319e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 21:54:24,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1235226.6666666667, ans=0.0
2023-11-20 21:54:26,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1235226.6666666667, ans=0.0
2023-11-20 21:54:36,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1235226.6666666667, ans=0.0
2023-11-20 21:54:43,206 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.73 vs. limit=10.0
2023-11-20 21:54:43,791 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185300
2023-11-20 21:54:49,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1235293.3333333333, ans=0.125
2023-11-20 21:54:52,118 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 4950, loss[loss=0.08966, simple_loss=0.1085, pruned_loss=0.02772, audio_tagging_loss=0.007674, over 15028.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.09904, pruned_loss=0.01843, audio_tagging_loss=0.01, over 3052588.71 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:55:43,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.85 vs. limit=15.0
2023-11-20 21:55:48,223 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185350
2023-11-20 21:55:49,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1235626.6666666667, ans=0.125
2023-11-20 21:55:53,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1235626.6666666667, ans=0.0
2023-11-20 21:55:55,706 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5000, loss[loss=0.06837, simple_loss=0.07628, pruned_loss=0.01615, audio_tagging_loss=0.01408, over 14560.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09858, pruned_loss=0.01827, audio_tagging_loss=0.00984, over 3044545.58 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:55:58,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1235693.3333333333, ans=0.125
2023-11-20 21:56:16,468 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.618e+01 7.784e+01 8.478e+01 9.107e+01 1.087e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-20 21:56:19,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1235760.0, ans=0.0
2023-11-20 21:56:20,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1235826.6666666667, ans=0.1
2023-11-20 21:56:32,796 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:56:44,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1235893.3333333333, ans=0.0
2023-11-20 21:56:47,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1235960.0, ans=0.0
2023-11-20 21:56:52,767 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185400
2023-11-20 21:57:01,015 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5050, loss[loss=0.07634, simple_loss=0.104, pruned_loss=0.01626, audio_tagging_loss=0.008066, over 14580.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.098, pruned_loss=0.01807, audio_tagging_loss=0.009873, over 3034403.55 frames. ], batch size: 54, lr: 4.31e-03, grad_scale: 8.0
2023-11-20 21:57:39,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1236226.6666666667, ans=0.125
2023-11-20 21:57:57,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185450
2023-11-20 21:58:05,604 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5100, loss[loss=0.06128, simple_loss=0.07779, pruned_loss=0.01199, audio_tagging_loss=0.0104, over 14563.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.09843, pruned_loss=0.0182, audio_tagging_loss=0.009821, over 3041634.15 frames. ], batch size: 56, lr: 4.31e-03, grad_scale: 8.0
2023-11-20 21:58:07,521 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.52 vs. limit=22.5
2023-11-20 21:58:25,679 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.354e+01 9.032e+01 1.014e+02 3.240e+02, threshold=1.806e+02, percent-clipped=1.0
2023-11-20 21:58:35,151 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.49 vs. limit=22.5
2023-11-20 21:58:54,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1236560.0, ans=0.125
2023-11-20 21:59:01,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.15 vs. limit=22.5
2023-11-20 21:59:02,158 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185500
2023-11-20 21:59:09,302 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5150, loss[loss=0.07966, simple_loss=0.09588, pruned_loss=0.02343, audio_tagging_loss=0.008287, over 14532.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09802, pruned_loss=0.01812, audio_tagging_loss=0.009796, over 3039076.22 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 8.0
2023-11-20 21:59:10,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1236693.3333333333, ans=0.035
2023-11-20 21:59:23,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1236760.0, ans=0.0
2023-11-20 21:59:26,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1236760.0, ans=0.125
2023-11-20 21:59:48,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1236893.3333333333, ans=0.125
2023-11-20 22:00:05,790 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185550
2023-11-20 22:00:06,238 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.46 vs. limit=22.5
2023-11-20 22:00:09,474 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.03 vs. limit=22.5
2023-11-20 22:00:12,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1237026.6666666667, ans=0.1
2023-11-20 22:00:14,334 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5200, loss[loss=0.08562, simple_loss=0.1189, pruned_loss=0.01749, audio_tagging_loss=0.008673, over 15781.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.09873, pruned_loss=0.01839, audio_tagging_loss=0.009793, over 3037466.91 frames. ], batch size: 58, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:00:16,144 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.64 vs. limit=15.0
2023-11-20 22:00:17,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1237026.6666666667, ans=0.07
2023-11-20 22:00:24,331 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:00:24,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.82 vs. limit=6.0
2023-11-20 22:00:34,947 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.910e+01 8.252e+01 8.979e+01 9.707e+01 1.443e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-20 22:00:40,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1237160.0, ans=0.2
2023-11-20 22:00:57,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1237226.6666666667, ans=0.0
2023-11-20 22:01:10,294 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185600
2023-11-20 22:01:18,568 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5250, loss[loss=0.1035, simple_loss=0.1399, pruned_loss=0.0263, audio_tagging_loss=0.007279, over 15942.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.0985, pruned_loss=0.01826, audio_tagging_loss=0.009646, over 3039102.34 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:01:32,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1237426.6666666667, ans=0.1
2023-11-20 22:02:07,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=12.0
2023-11-20 22:02:15,568 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185650
2023-11-20 22:02:22,621 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5300, loss[loss=0.08136, simple_loss=0.1036, pruned_loss=0.02234, audio_tagging_loss=0.007207, over 14842.00 frames. ], tot_loss[loss=0.07794, simple_loss=0.0996, pruned_loss=0.01857, audio_tagging_loss=0.009574, over 3040496.85 frames. ], batch size: 55, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:02:32,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1237693.3333333333, ans=0.1
2023-11-20 22:02:42,486 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.578e+01 8.164e+01 8.663e+01 9.355e+01 1.569e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-20 22:02:49,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1237826.6666666667, ans=0.05
2023-11-20 22:02:57,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.52 vs. limit=6.0
2023-11-20 22:03:18,385 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185700
2023-11-20 22:03:26,267 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5350, loss[loss=0.07352, simple_loss=0.09284, pruned_loss=0.01544, audio_tagging_loss=0.01166, over 14718.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.09981, pruned_loss=0.01855, audio_tagging_loss=0.009531, over 3035387.25 frames. ], batch size: 54, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:03:34,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1238026.6666666667, ans=0.125
2023-11-20 22:03:50,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1238093.3333333333, ans=0.125
2023-11-20 22:03:51,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1238160.0, ans=0.0
2023-11-20 22:03:53,331 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-20 22:04:07,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1238226.6666666667, ans=0.125
2023-11-20 22:04:23,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185750
2023-11-20 22:04:25,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1238293.3333333333, ans=0.2
2023-11-20 22:04:31,058 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5400, loss[loss=0.05592, simple_loss=0.06747, pruned_loss=0.01077, audio_tagging_loss=0.01141, over 15091.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.09956, pruned_loss=0.01858, audio_tagging_loss=0.009621, over 3033865.11 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:04:50,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1238426.6666666667, ans=0.125
2023-11-20 22:04:51,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.64 vs. limit=15.0
2023-11-20 22:04:51,606 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.664e+01 7.937e+01 8.475e+01 9.191e+01 1.261e+02, threshold=1.695e+02, percent-clipped=0.0
2023-11-20 22:05:04,809 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.15 vs. limit=15.0
2023-11-20 22:05:08,146 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:05:22,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1238626.6666666667, ans=0.2
2023-11-20 22:05:28,231 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185800
2023-11-20 22:05:32,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1238626.6666666667, ans=0.0
2023-11-20 22:05:35,786 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5450, loss[loss=0.07591, simple_loss=0.101, pruned_loss=0.01589, audio_tagging_loss=0.009537, over 15704.00 frames. ], tot_loss[loss=0.07846, simple_loss=0.1002, pruned_loss=0.01876, audio_tagging_loss=0.009587, over 3033816.20 frames. ], batch size: 58, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:05:36,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1238693.3333333333, ans=0.0
2023-11-20 22:05:49,735 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=22.5
2023-11-20 22:06:03,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1238826.6666666667, ans=0.125
2023-11-20 22:06:17,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1238893.3333333333, ans=0.1
2023-11-20 22:06:31,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185850
2023-11-20 22:06:38,730 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5500, loss[loss=0.08147, simple_loss=0.1001, pruned_loss=0.01888, audio_tagging_loss=0.01257, over 15471.00 frames. ], tot_loss[loss=0.07936, simple_loss=0.1013, pruned_loss=0.01908, audio_tagging_loss=0.009646, over 3043911.17 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:06:45,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1239026.6666666667, ans=0.1
2023-11-20 22:07:00,049 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.304e+01 8.468e+01 9.055e+01 9.983e+01 1.429e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-20 22:07:00,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1239093.3333333333, ans=0.0
2023-11-20 22:07:35,527 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185900
2023-11-20 22:07:42,621 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5550, loss[loss=0.09755, simple_loss=0.1192, pruned_loss=0.02933, audio_tagging_loss=0.008617, over 15536.00 frames. ], tot_loss[loss=0.07952, simple_loss=0.1016, pruned_loss=0.01901, audio_tagging_loss=0.009723, over 3045182.09 frames. ], batch size: 56, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:08:04,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1239426.6666666667, ans=0.0
2023-11-20 22:08:06,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1239426.6666666667, ans=0.125
2023-11-20 22:08:16,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1239493.3333333333, ans=0.125
2023-11-20 22:08:40,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 185950
2023-11-20 22:08:48,139 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5600, loss[loss=0.06396, simple_loss=0.07763, pruned_loss=0.01066, audio_tagging_loss=0.01449, over 14956.00 frames. ], tot_loss[loss=0.07888, simple_loss=0.1007, pruned_loss=0.01863, audio_tagging_loss=0.009893, over 3045675.50 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 32.0
2023-11-20 22:08:49,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1239693.3333333333, ans=0.1
2023-11-20 22:09:08,126 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.915e+01 8.506e+01 9.381e+01 1.103e+02, threshold=1.701e+02, percent-clipped=0.0
2023-11-20 22:09:14,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1239826.6666666667, ans=0.0
2023-11-20 22:09:16,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.77 vs. limit=12.0
2023-11-20 22:09:31,251 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 22:09:43,263 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186000
2023-11-20 22:09:50,698 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5650, loss[loss=0.0695, simple_loss=0.08186, pruned_loss=0.01723, audio_tagging_loss=0.01133, over 15506.00 frames. ], tot_loss[loss=0.07843, simple_loss=0.09986, pruned_loss=0.01851, audio_tagging_loss=0.009996, over 3045424.63 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 32.0
2023-11-20 22:09:54,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1240026.6666666667, ans=0.0
2023-11-20 22:10:02,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=15.0
2023-11-20 22:10:34,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1240226.6666666667, ans=0.125
2023-11-20 22:10:47,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186050
2023-11-20 22:10:54,514 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5700, loss[loss=0.08226, simple_loss=0.1084, pruned_loss=0.01953, audio_tagging_loss=0.0085, over 15700.00 frames. ], tot_loss[loss=0.07782, simple_loss=0.09879, pruned_loss=0.01837, audio_tagging_loss=0.01006, over 3048787.19 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:11:16,789 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.091e+01 8.750e+01 9.672e+01 1.332e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 22:11:21,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1240493.3333333333, ans=0.125
2023-11-20 22:11:30,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1240493.3333333333, ans=0.125
2023-11-20 22:11:33,632 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:11:38,766 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.05 vs. limit=15.0
2023-11-20 22:11:49,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1240626.6666666667, ans=0.125
2023-11-20 22:11:51,014 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186100
2023-11-20 22:11:58,994 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5750, loss[loss=0.07995, simple_loss=0.1071, pruned_loss=0.01687, audio_tagging_loss=0.00955, over 14723.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09888, pruned_loss=0.01828, audio_tagging_loss=0.00992, over 3051909.55 frames. ], batch size: 53, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:12:00,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1240693.3333333333, ans=0.125
2023-11-20 22:12:01,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1240693.3333333333, ans=0.1
2023-11-20 22:12:07,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1240693.3333333333, ans=0.125
2023-11-20 22:12:07,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1240693.3333333333, ans=0.125
2023-11-20 22:12:16,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1240760.0, ans=0.2
2023-11-20 22:12:23,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1240826.6666666667, ans=0.09899494936611666
2023-11-20 22:12:55,235 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186150
2023-11-20 22:12:56,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1240960.0, ans=0.125
2023-11-20 22:12:57,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1240960.0, ans=0.09899494936611666
2023-11-20 22:13:02,400 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5800, loss[loss=0.08018, simple_loss=0.09552, pruned_loss=0.02052, audio_tagging_loss=0.0119, over 14575.00 frames. ], tot_loss[loss=0.07829, simple_loss=0.09978, pruned_loss=0.01868, audio_tagging_loss=0.009722, over 3049083.12 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:13:10,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.53 vs. limit=22.5
2023-11-20 22:13:11,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1241026.6666666667, ans=0.015
2023-11-20 22:13:16,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1241093.3333333333, ans=0.05
2023-11-20 22:13:23,626 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.344e+01 8.039e+01 8.914e+01 9.519e+01 1.369e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-20 22:13:26,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.41 vs. limit=15.0
2023-11-20 22:13:30,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1241160.0, ans=0.1
2023-11-20 22:13:30,862 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.29 vs. limit=12.0
2023-11-20 22:13:37,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1241160.0, ans=0.1
2023-11-20 22:13:40,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1241226.6666666667, ans=0.125
2023-11-20 22:13:41,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1241226.6666666667, ans=0.1
2023-11-20 22:13:41,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1241226.6666666667, ans=0.125
2023-11-20 22:13:43,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1241226.6666666667, ans=0.0
2023-11-20 22:13:48,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1241226.6666666667, ans=0.0
2023-11-20 22:13:49,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1241226.6666666667, ans=0.09899494936611666
2023-11-20 22:13:58,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186200
2023-11-20 22:14:00,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1241293.3333333333, ans=0.125
2023-11-20 22:14:00,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1241293.3333333333, ans=0.125
2023-11-20 22:14:06,327 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5850, loss[loss=0.08896, simple_loss=0.1196, pruned_loss=0.02118, audio_tagging_loss=0.00796, over 15780.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09898, pruned_loss=0.01843, audio_tagging_loss=0.009703, over 3047593.23 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:14:07,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1241360.0, ans=0.125
2023-11-20 22:14:09,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1241360.0, ans=0.1
2023-11-20 22:14:25,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.13 vs. limit=15.0
2023-11-20 22:14:33,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1241493.3333333333, ans=0.1
2023-11-20 22:14:38,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1241493.3333333333, ans=0.0
2023-11-20 22:14:48,118 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:14:51,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1241560.0, ans=0.0
2023-11-20 22:15:00,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1241626.6666666667, ans=0.0
2023-11-20 22:15:02,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186250
2023-11-20 22:15:11,173 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5900, loss[loss=0.06752, simple_loss=0.08565, pruned_loss=0.01449, audio_tagging_loss=0.0102, over 16344.00 frames. ], tot_loss[loss=0.07794, simple_loss=0.09929, pruned_loss=0.01856, audio_tagging_loss=0.009738, over 3055260.21 frames. ], batch size: 61, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:15:17,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.23 vs. limit=15.0
2023-11-20 22:15:19,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1241693.3333333333, ans=0.0
2023-11-20 22:15:32,174 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.032e+01 8.653e+01 9.726e+01 1.327e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-20 22:15:34,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-20 22:16:06,686 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186300
2023-11-20 22:16:14,541 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 5950, loss[loss=0.06409, simple_loss=0.07357, pruned_loss=0.01702, audio_tagging_loss=0.01029, over 15701.00 frames. ], tot_loss[loss=0.07765, simple_loss=0.09907, pruned_loss=0.01843, audio_tagging_loss=0.009688, over 3060092.10 frames. ], batch size: 59, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:16:15,074 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.58 vs. limit=15.0
2023-11-20 22:17:11,497 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186350
2023-11-20 22:17:17,491 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.01 vs. limit=6.0
2023-11-20 22:17:19,221 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6000, loss[loss=0.06901, simple_loss=0.0881, pruned_loss=0.01485, audio_tagging_loss=0.01011, over 16947.00 frames. ], tot_loss[loss=0.07707, simple_loss=0.0984, pruned_loss=0.01822, audio_tagging_loss=0.009652, over 3051573.67 frames. ], batch size: 64, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:17:19,222 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 22:17:46,987 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.9030, 5.7357, 5.3842, 5.4734], device='cuda:2')
2023-11-20 22:18:03,616 INFO [train_asr.py:1253] (2/4) Epoch 16, validation: loss=0.06177, simple_loss=0.05296, pruned_loss=0.005445, audio_tagging_loss=0.02985, over 4681554.00 frames. 
2023-11-20 22:18:03,617 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 22:18:05,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1242360.0, ans=0.125
2023-11-20 22:18:25,408 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.385e+01 8.060e+01 8.625e+01 9.633e+01 1.979e+02, threshold=1.725e+02, percent-clipped=1.0
2023-11-20 22:18:37,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1242493.3333333333, ans=0.0
2023-11-20 22:18:37,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1242493.3333333333, ans=0.2
2023-11-20 22:18:47,824 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 22:18:52,214 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:18:56,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1242626.6666666667, ans=0.0
2023-11-20 22:18:59,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186400
2023-11-20 22:19:07,779 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6050, loss[loss=0.06234, simple_loss=0.07418, pruned_loss=0.01264, audio_tagging_loss=0.01261, over 14865.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.09826, pruned_loss=0.01821, audio_tagging_loss=0.009629, over 3045006.58 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:19:15,047 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.34 vs. limit=15.0
2023-11-20 22:19:31,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=8.40 vs. limit=8.0
2023-11-20 22:20:04,985 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186450
2023-11-20 22:20:11,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1243026.6666666667, ans=0.1
2023-11-20 22:20:12,057 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6100, loss[loss=0.0724, simple_loss=0.09643, pruned_loss=0.01502, audio_tagging_loss=0.009163, over 14972.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09835, pruned_loss=0.01821, audio_tagging_loss=0.009707, over 3045801.57 frames. ], batch size: 57, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:20:34,065 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.005e+01 8.728e+01 9.355e+01 1.254e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-20 22:20:36,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1243160.0, ans=0.125
2023-11-20 22:21:04,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.85 vs. limit=22.5
2023-11-20 22:21:08,396 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186500
2023-11-20 22:21:11,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.11 vs. limit=15.0
2023-11-20 22:21:15,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1243360.0, ans=0.125
2023-11-20 22:21:16,674 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6150, loss[loss=0.07393, simple_loss=0.09402, pruned_loss=0.01596, audio_tagging_loss=0.01096, over 17147.00 frames. ], tot_loss[loss=0.0783, simple_loss=0.09985, pruned_loss=0.01869, audio_tagging_loss=0.009686, over 3048523.46 frames. ], batch size: 63, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:21:26,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1243360.0, ans=0.0
2023-11-20 22:21:28,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1243426.6666666667, ans=0.0
2023-11-20 22:21:29,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1243426.6666666667, ans=0.0
2023-11-20 22:21:48,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1243493.3333333333, ans=0.2
2023-11-20 22:21:59,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1243560.0, ans=0.125
2023-11-20 22:21:59,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1243560.0, ans=0.0
2023-11-20 22:22:12,922 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186550
2023-11-20 22:22:20,053 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6200, loss[loss=0.1028, simple_loss=0.1313, pruned_loss=0.02887, audio_tagging_loss=0.008306, over 16873.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.09928, pruned_loss=0.01851, audio_tagging_loss=0.009739, over 3046241.82 frames. ], batch size: 60, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:22:39,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1243760.0, ans=0.125
2023-11-20 22:22:42,088 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.113e+01 8.838e+01 9.519e+01 1.384e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 22:22:43,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1243760.0, ans=0.125
2023-11-20 22:23:03,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1243893.3333333333, ans=0.125
2023-11-20 22:23:08,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.73 vs. limit=22.5
2023-11-20 22:23:17,808 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186600
2023-11-20 22:23:25,503 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6250, loss[loss=0.04329, simple_loss=0.0528, pruned_loss=0.007886, audio_tagging_loss=0.009004, over 14460.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09899, pruned_loss=0.01852, audio_tagging_loss=0.009808, over 3043950.74 frames. ], batch size: 57, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:23:44,320 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.29 vs. limit=22.5
2023-11-20 22:24:02,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1244226.6666666667, ans=0.0
2023-11-20 22:24:05,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1244226.6666666667, ans=0.125
2023-11-20 22:24:14,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1244226.6666666667, ans=0.125
2023-11-20 22:24:17,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1244293.3333333333, ans=0.1
2023-11-20 22:24:21,334 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186650
2023-11-20 22:24:29,277 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6300, loss[loss=0.08544, simple_loss=0.1074, pruned_loss=0.01926, audio_tagging_loss=0.01248, over 15196.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.09847, pruned_loss=0.01842, audio_tagging_loss=0.009924, over 3041405.14 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:24:29,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1244360.0, ans=0.125
2023-11-20 22:24:30,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1244360.0, ans=0.0
2023-11-20 22:24:34,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1244360.0, ans=0.0
2023-11-20 22:24:38,828 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.12 vs. limit=15.0
2023-11-20 22:24:47,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1244426.6666666667, ans=0.1
2023-11-20 22:24:47,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1244426.6666666667, ans=0.1
2023-11-20 22:24:51,659 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.323e+01 8.066e+01 8.583e+01 9.288e+01 1.245e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-20 22:25:02,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1244493.3333333333, ans=0.5
2023-11-20 22:25:15,413 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.05 vs. limit=6.0
2023-11-20 22:25:16,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1244560.0, ans=0.0
2023-11-20 22:25:22,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1244626.6666666667, ans=0.0
2023-11-20 22:25:25,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186700
2023-11-20 22:25:32,746 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6350, loss[loss=0.1007, simple_loss=0.1277, pruned_loss=0.02783, audio_tagging_loss=0.008995, over 15295.00 frames. ], tot_loss[loss=0.07819, simple_loss=0.0992, pruned_loss=0.0187, audio_tagging_loss=0.009882, over 3047814.46 frames. ], batch size: 58, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:25:34,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1244693.3333333333, ans=0.125
2023-11-20 22:26:06,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1244826.6666666667, ans=0.0
2023-11-20 22:26:09,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1244826.6666666667, ans=0.2
2023-11-20 22:26:29,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186750
2023-11-20 22:26:37,568 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6400, loss[loss=0.06765, simple_loss=0.08836, pruned_loss=0.01059, audio_tagging_loss=0.01287, over 14908.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09849, pruned_loss=0.01848, audio_tagging_loss=0.009981, over 3042986.47 frames. ], batch size: 53, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:26:52,538 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:26:55,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1245093.3333333333, ans=0.0
2023-11-20 22:27:00,824 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.359e+01 8.103e+01 8.559e+01 9.205e+01 1.089e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-20 22:27:01,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.27 vs. limit=22.5
2023-11-20 22:27:20,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1245226.6666666667, ans=0.125
2023-11-20 22:27:22,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1245226.6666666667, ans=0.1
2023-11-20 22:27:34,037 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186800
2023-11-20 22:27:41,522 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6450, loss[loss=0.102, simple_loss=0.1337, pruned_loss=0.02662, audio_tagging_loss=0.008559, over 16856.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09869, pruned_loss=0.01842, audio_tagging_loss=0.01011, over 3043115.82 frames. ], batch size: 61, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:27:56,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1245426.6666666667, ans=0.125
2023-11-20 22:28:03,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1245426.6666666667, ans=0.0
2023-11-20 22:28:39,170 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186850
2023-11-20 22:28:44,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1245626.6666666667, ans=0.125
2023-11-20 22:28:46,242 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6500, loss[loss=0.05886, simple_loss=0.07015, pruned_loss=0.01245, audio_tagging_loss=0.01134, over 16447.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.0995, pruned_loss=0.01843, audio_tagging_loss=0.01, over 3040353.49 frames. ], batch size: 63, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:28:55,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1245693.3333333333, ans=0.125
2023-11-20 22:28:58,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.32 vs. limit=15.0
2023-11-20 22:28:59,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1245760.0, ans=0.125
2023-11-20 22:29:02,025 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:29:09,944 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.106e+01 8.759e+01 9.505e+01 1.385e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 22:29:10,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1245760.0, ans=0.1
2023-11-20 22:29:18,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1245826.6666666667, ans=0.2
2023-11-20 22:29:42,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186900
2023-11-20 22:29:50,871 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6550, loss[loss=0.06241, simple_loss=0.07145, pruned_loss=0.01646, audio_tagging_loss=0.01023, over 15164.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.09976, pruned_loss=0.0184, audio_tagging_loss=0.009764, over 3041901.46 frames. ], batch size: 60, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:30:00,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1246026.6666666667, ans=0.125
2023-11-20 22:30:14,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1246093.3333333333, ans=0.125
2023-11-20 22:30:39,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0
2023-11-20 22:30:48,469 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 186950
2023-11-20 22:30:55,606 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6600, loss[loss=0.08108, simple_loss=0.1034, pruned_loss=0.02071, audio_tagging_loss=0.008669, over 15778.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09949, pruned_loss=0.01838, audio_tagging_loss=0.009681, over 3049161.51 frames. ], batch size: 59, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:31:03,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1246360.0, ans=0.0
2023-11-20 22:31:18,342 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.31 vs. limit=15.0
2023-11-20 22:31:18,842 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.367e+01 8.854e+01 9.653e+01 1.222e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-20 22:31:30,462 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:31:30,965 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0
2023-11-20 22:31:52,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187000
2023-11-20 22:32:00,541 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6650, loss[loss=0.08958, simple_loss=0.1154, pruned_loss=0.02146, audio_tagging_loss=0.01041, over 15472.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09872, pruned_loss=0.0183, audio_tagging_loss=0.009631, over 3048580.82 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:32:34,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1246826.6666666667, ans=0.125
2023-11-20 22:32:50,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1246960.0, ans=0.125
2023-11-20 22:32:56,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187050
2023-11-20 22:33:04,401 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6700, loss[loss=0.05474, simple_loss=0.0626, pruned_loss=0.008384, audio_tagging_loss=0.01505, over 15619.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.09924, pruned_loss=0.01846, audio_tagging_loss=0.009603, over 3052253.53 frames. ], batch size: 63, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:33:27,693 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.055e+01 8.769e+01 9.493e+01 1.327e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-20 22:33:44,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1247226.6666666667, ans=0.035
2023-11-20 22:33:50,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1247226.6666666667, ans=0.1
2023-11-20 22:33:57,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1247293.3333333333, ans=0.2
2023-11-20 22:33:58,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1247293.3333333333, ans=0.1
2023-11-20 22:34:00,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187100
2023-11-20 22:34:07,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1247360.0, ans=0.125
2023-11-20 22:34:08,147 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6750, loss[loss=0.06977, simple_loss=0.09343, pruned_loss=0.0142, audio_tagging_loss=0.008854, over 16087.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09889, pruned_loss=0.01851, audio_tagging_loss=0.009673, over 3047580.11 frames. ], batch size: 60, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:34:26,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1247426.6666666667, ans=0.125
2023-11-20 22:34:30,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1247426.6666666667, ans=0.1
2023-11-20 22:34:37,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1247493.3333333333, ans=0.0
2023-11-20 22:34:38,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1247493.3333333333, ans=0.2
2023-11-20 22:34:45,400 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.92 vs. limit=6.0
2023-11-20 22:34:48,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1247560.0, ans=0.125
2023-11-20 22:34:53,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1247560.0, ans=0.1
2023-11-20 22:35:05,412 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187150
2023-11-20 22:35:12,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.46 vs. limit=22.5
2023-11-20 22:35:13,206 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6800, loss[loss=0.08825, simple_loss=0.1053, pruned_loss=0.02683, audio_tagging_loss=0.008751, over 15186.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.09979, pruned_loss=0.01875, audio_tagging_loss=0.009557, over 3047634.95 frames. ], batch size: 55, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:35:15,184 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.60 vs. limit=15.0
2023-11-20 22:35:19,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1247693.3333333333, ans=0.1
2023-11-20 22:35:35,483 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.835e+01 8.011e+01 8.725e+01 9.590e+01 1.326e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-20 22:35:43,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1247826.6666666667, ans=10.0
2023-11-20 22:35:45,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.69 vs. limit=22.5
2023-11-20 22:36:08,980 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187200
2023-11-20 22:36:16,475 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6850, loss[loss=0.09237, simple_loss=0.1245, pruned_loss=0.01879, audio_tagging_loss=0.01135, over 15326.00 frames. ], tot_loss[loss=0.07832, simple_loss=0.1, pruned_loss=0.01866, audio_tagging_loss=0.009646, over 3046546.62 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:36:19,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1248026.6666666667, ans=0.0
2023-11-20 22:36:50,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1248160.0, ans=0.2
2023-11-20 22:37:04,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1248226.6666666667, ans=0.0
2023-11-20 22:37:04,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.78 vs. limit=22.5
2023-11-20 22:37:05,526 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.76 vs. limit=12.0
2023-11-20 22:37:10,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1248293.3333333333, ans=0.125
2023-11-20 22:37:11,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1248293.3333333333, ans=0.05
2023-11-20 22:37:12,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187250
2023-11-20 22:37:20,277 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6900, loss[loss=0.06559, simple_loss=0.08123, pruned_loss=0.01428, audio_tagging_loss=0.01069, over 15005.00 frames. ], tot_loss[loss=0.07807, simple_loss=0.09965, pruned_loss=0.01853, audio_tagging_loss=0.00971, over 3055106.15 frames. ], batch size: 55, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:37:43,486 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.29 vs. limit=10.0
2023-11-20 22:37:44,208 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.341e+01 8.921e+01 9.872e+01 1.364e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-20 22:37:59,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-20 22:38:01,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1248560.0, ans=0.2
2023-11-20 22:38:06,841 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 22:38:13,495 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.54 vs. limit=12.0
2023-11-20 22:38:16,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187300
2023-11-20 22:38:23,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1248693.3333333333, ans=0.0
2023-11-20 22:38:24,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1248693.3333333333, ans=0.1
2023-11-20 22:38:25,000 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 6950, loss[loss=0.06946, simple_loss=0.08485, pruned_loss=0.01678, audio_tagging_loss=0.01026, over 14269.00 frames. ], tot_loss[loss=0.07858, simple_loss=0.1002, pruned_loss=0.01879, audio_tagging_loss=0.00972, over 3053495.01 frames. ], batch size: 54, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:38:42,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1248760.0, ans=0.125
2023-11-20 22:39:05,023 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=12.0
2023-11-20 22:39:05,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1248893.3333333333, ans=0.2
2023-11-20 22:39:20,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1248960.0, ans=0.09899494936611666
2023-11-20 22:39:21,188 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187350
2023-11-20 22:39:21,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1248960.0, ans=0.125
2023-11-20 22:39:28,598 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7000, loss[loss=0.0689, simple_loss=0.08898, pruned_loss=0.01739, audio_tagging_loss=0.007013, over 15070.00 frames. ], tot_loss[loss=0.07855, simple_loss=0.1004, pruned_loss=0.01868, audio_tagging_loss=0.00968, over 3055648.86 frames. ], batch size: 57, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:39:36,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1249026.6666666667, ans=0.125
2023-11-20 22:39:42,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0
2023-11-20 22:39:52,394 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.148e+01 8.784e+01 9.512e+01 1.267e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-20 22:39:53,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1249160.0, ans=0.0
2023-11-20 22:40:04,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1249160.0, ans=0.1
2023-11-20 22:40:09,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1249226.6666666667, ans=0.125
2023-11-20 22:40:17,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1249226.6666666667, ans=0.125
2023-11-20 22:40:20,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1249293.3333333333, ans=0.125
2023-11-20 22:40:25,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187400
2023-11-20 22:40:33,236 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7050, loss[loss=0.07139, simple_loss=0.08238, pruned_loss=0.01717, audio_tagging_loss=0.01303, over 15367.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09964, pruned_loss=0.0185, audio_tagging_loss=0.009809, over 3056256.88 frames. ], batch size: 58, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:40:50,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=12.0
2023-11-20 22:40:51,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1249426.6666666667, ans=0.035
2023-11-20 22:40:54,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1249426.6666666667, ans=0.0
2023-11-20 22:41:05,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1249493.3333333333, ans=0.025
2023-11-20 22:41:06,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1249493.3333333333, ans=0.1
2023-11-20 22:41:14,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1249560.0, ans=0.035
2023-11-20 22:41:18,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1249560.0, ans=0.05
2023-11-20 22:41:29,399 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187450
2023-11-20 22:41:37,928 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7100, loss[loss=0.07774, simple_loss=0.09514, pruned_loss=0.0195, audio_tagging_loss=0.01068, over 14728.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09877, pruned_loss=0.01826, audio_tagging_loss=0.009848, over 3056325.01 frames. ], batch size: 53, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:41:38,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1249693.3333333333, ans=0.07
2023-11-20 22:41:55,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1249760.0, ans=0.125
2023-11-20 22:42:02,250 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.191e+01 8.839e+01 9.583e+01 1.103e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 22:42:12,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1249826.6666666667, ans=0.2
2023-11-20 22:42:14,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1249893.3333333333, ans=0.0
2023-11-20 22:42:19,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1249893.3333333333, ans=0.1
2023-11-20 22:42:34,635 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187500
2023-11-20 22:42:38,904 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.73 vs. limit=15.0
2023-11-20 22:42:41,828 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7150, loss[loss=0.09118, simple_loss=0.1187, pruned_loss=0.01998, audio_tagging_loss=0.01187, over 16105.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09842, pruned_loss=0.01802, audio_tagging_loss=0.009974, over 3053437.92 frames. ], batch size: 59, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:42:43,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1250026.6666666667, ans=0.0
2023-11-20 22:42:45,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1250026.6666666667, ans=0.125
2023-11-20 22:43:23,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1250226.6666666667, ans=0.0
2023-11-20 22:43:28,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1250226.6666666667, ans=0.125
2023-11-20 22:43:38,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187550
2023-11-20 22:43:45,995 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7200, loss[loss=0.07163, simple_loss=0.08683, pruned_loss=0.01532, audio_tagging_loss=0.01289, over 16219.00 frames. ], tot_loss[loss=0.07744, simple_loss=0.09877, pruned_loss=0.01804, audio_tagging_loss=0.01002, over 3051889.10 frames. ], batch size: 60, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:44:10,056 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.608e+01 8.172e+01 8.869e+01 9.437e+01 2.740e+02, threshold=1.774e+02, percent-clipped=1.0
2023-11-20 22:44:41,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1250626.6666666667, ans=0.2
2023-11-20 22:44:41,987 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187600
2023-11-20 22:44:46,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1250626.6666666667, ans=0.125
2023-11-20 22:44:50,329 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7250, loss[loss=0.0592, simple_loss=0.07495, pruned_loss=0.01191, audio_tagging_loss=0.009812, over 16094.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09907, pruned_loss=0.01818, audio_tagging_loss=0.01005, over 3045923.30 frames. ], batch size: 60, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:44:54,605 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:45:02,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1250760.0, ans=0.2
2023-11-20 22:45:07,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1250760.0, ans=0.125
2023-11-20 22:45:17,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1250826.6666666667, ans=0.09899494936611666
2023-11-20 22:45:47,066 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187650
2023-11-20 22:45:55,048 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7300, loss[loss=0.06195, simple_loss=0.07416, pruned_loss=0.01252, audio_tagging_loss=0.01234, over 15369.00 frames. ], tot_loss[loss=0.07748, simple_loss=0.09888, pruned_loss=0.01813, audio_tagging_loss=0.00991, over 3050150.63 frames. ], batch size: 58, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:46:01,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1251026.6666666667, ans=0.1
2023-11-20 22:46:18,767 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 8.202e+01 8.886e+01 9.738e+01 1.326e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-20 22:46:25,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1251160.0, ans=0.1
2023-11-20 22:46:25,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1251160.0, ans=0.0
2023-11-20 22:46:51,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187700
2023-11-20 22:46:59,067 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7350, loss[loss=0.07442, simple_loss=0.1002, pruned_loss=0.01823, audio_tagging_loss=0.006069, over 15419.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09875, pruned_loss=0.01824, audio_tagging_loss=0.009701, over 3048271.72 frames. ], batch size: 57, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:47:04,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1251360.0, ans=0.125
2023-11-20 22:47:19,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1251426.6666666667, ans=0.125
2023-11-20 22:47:41,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1251560.0, ans=0.0
2023-11-20 22:47:50,636 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.87 vs. limit=15.0
2023-11-20 22:47:55,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187750
2023-11-20 22:47:55,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1251626.6666666667, ans=0.125
2023-11-20 22:48:02,459 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7400, loss[loss=0.07152, simple_loss=0.0824, pruned_loss=0.02007, audio_tagging_loss=0.01025, over 15110.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09811, pruned_loss=0.01811, audio_tagging_loss=0.009669, over 3048940.92 frames. ], batch size: 59, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:48:12,916 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.88 vs. limit=10.0
2023-11-20 22:48:22,707 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.40 vs. limit=10.0
2023-11-20 22:48:27,600 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.213e+01 8.738e+01 9.709e+01 1.431e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-20 22:48:39,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1251826.6666666667, ans=0.0
2023-11-20 22:48:39,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1251826.6666666667, ans=0.0
2023-11-20 22:48:59,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187800
2023-11-20 22:49:06,800 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7450, loss[loss=0.08793, simple_loss=0.1207, pruned_loss=0.01518, audio_tagging_loss=0.01238, over 14996.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.09973, pruned_loss=0.01852, audio_tagging_loss=0.009596, over 3047709.62 frames. ], batch size: 54, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:49:10,963 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0
2023-11-20 22:49:20,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1252093.3333333333, ans=0.125
2023-11-20 22:49:24,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1252093.3333333333, ans=0.0
2023-11-20 22:49:28,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1252093.3333333333, ans=0.1
2023-11-20 22:49:33,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1252160.0, ans=0.125
2023-11-20 22:49:47,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1252226.6666666667, ans=0.1
2023-11-20 22:49:57,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1252293.3333333333, ans=0.0
2023-11-20 22:50:01,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1252293.3333333333, ans=0.125
2023-11-20 22:50:02,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187850
2023-11-20 22:50:10,842 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7500, loss[loss=0.07051, simple_loss=0.07958, pruned_loss=0.02225, audio_tagging_loss=0.008468, over 13799.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.09844, pruned_loss=0.01823, audio_tagging_loss=0.009597, over 3052485.06 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:50:27,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1252426.6666666667, ans=0.04949747468305833
2023-11-20 22:50:28,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1252426.6666666667, ans=0.1
2023-11-20 22:50:28,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1252426.6666666667, ans=10.0
2023-11-20 22:50:32,727 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:50:34,840 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.198e+01 8.879e+01 9.496e+01 1.257e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 22:50:38,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1252493.3333333333, ans=0.1
2023-11-20 22:50:48,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1252560.0, ans=0.05
2023-11-20 22:50:59,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1252560.0, ans=0.125
2023-11-20 22:51:00,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1252626.6666666667, ans=0.0
2023-11-20 22:51:04,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1252626.6666666667, ans=0.125
2023-11-20 22:51:06,694 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187900
2023-11-20 22:51:13,958 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7550, loss[loss=0.09441, simple_loss=0.1254, pruned_loss=0.02447, audio_tagging_loss=0.00724, over 16123.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.09805, pruned_loss=0.01831, audio_tagging_loss=0.009638, over 3049063.28 frames. ], batch size: 63, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:51:14,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1252693.3333333333, ans=0.2
2023-11-20 22:51:24,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1252693.3333333333, ans=0.125
2023-11-20 22:51:28,460 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:51:44,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1252826.6666666667, ans=0.125
2023-11-20 22:51:56,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.88 vs. limit=15.0
2023-11-20 22:52:10,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 187950
2023-11-20 22:52:17,969 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7600, loss[loss=0.08534, simple_loss=0.1097, pruned_loss=0.02319, audio_tagging_loss=0.00731, over 15372.00 frames. ], tot_loss[loss=0.07648, simple_loss=0.09716, pruned_loss=0.01809, audio_tagging_loss=0.009813, over 3050423.26 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:52:42,482 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.162e+01 8.819e+01 9.727e+01 1.348e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-20 22:53:08,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1253293.3333333333, ans=0.05
2023-11-20 22:53:14,913 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188000
2023-11-20 22:53:16,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1253293.3333333333, ans=0.125
2023-11-20 22:53:22,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=23.16 vs. limit=22.5
2023-11-20 22:53:25,793 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7650, loss[loss=0.07436, simple_loss=0.09037, pruned_loss=0.01565, audio_tagging_loss=0.01353, over 14789.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.09795, pruned_loss=0.01823, audio_tagging_loss=0.009738, over 3047296.80 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:53:32,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1253360.0, ans=0.1
2023-11-20 22:53:38,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1253426.6666666667, ans=0.125
2023-11-20 22:53:40,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1253426.6666666667, ans=0.5
2023-11-20 22:54:01,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1253493.3333333333, ans=0.125
2023-11-20 22:54:15,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1253560.0, ans=0.125
2023-11-20 22:54:22,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188050
2023-11-20 22:54:27,338 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.96 vs. limit=15.0
2023-11-20 22:54:30,067 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7700, loss[loss=0.07814, simple_loss=0.1055, pruned_loss=0.01758, audio_tagging_loss=0.007796, over 15656.00 frames. ], tot_loss[loss=0.07733, simple_loss=0.09829, pruned_loss=0.01836, audio_tagging_loss=0.009831, over 3041487.41 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:54:46,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1253760.0, ans=0.125
2023-11-20 22:54:46,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1253760.0, ans=0.2
2023-11-20 22:54:54,497 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.666e+01 7.984e+01 8.508e+01 9.260e+01 1.144e+02, threshold=1.702e+02, percent-clipped=0.0
2023-11-20 22:54:54,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1253826.6666666667, ans=0.125
2023-11-20 22:55:00,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1253826.6666666667, ans=0.05
2023-11-20 22:55:11,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1253893.3333333333, ans=0.0
2023-11-20 22:55:26,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188100
2023-11-20 22:55:30,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=15.0
2023-11-20 22:55:34,504 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7750, loss[loss=0.07004, simple_loss=0.09145, pruned_loss=0.01422, audio_tagging_loss=0.01009, over 16301.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09788, pruned_loss=0.01814, audio_tagging_loss=0.009834, over 3040059.46 frames. ], batch size: 60, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:55:50,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1254093.3333333333, ans=0.125
2023-11-20 22:56:08,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1254160.0, ans=0.125
2023-11-20 22:56:30,404 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188150
2023-11-20 22:56:35,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1254293.3333333333, ans=0.125
2023-11-20 22:56:37,563 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7800, loss[loss=0.08257, simple_loss=0.1075, pruned_loss=0.01969, audio_tagging_loss=0.009116, over 15054.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.09805, pruned_loss=0.0181, audio_tagging_loss=0.00986, over 3040831.49 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:56:40,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1254360.0, ans=0.125
2023-11-20 22:56:41,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1254360.0, ans=0.0
2023-11-20 22:56:52,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1254426.6666666667, ans=0.125
2023-11-20 22:57:01,918 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.318e+01 8.003e+01 8.886e+01 9.530e+01 1.591e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-20 22:57:07,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1254493.3333333333, ans=0.1
2023-11-20 22:57:14,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1254493.3333333333, ans=0.125
2023-11-20 22:57:24,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1254560.0, ans=0.125
2023-11-20 22:57:32,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1254626.6666666667, ans=0.0
2023-11-20 22:57:34,375 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188200
2023-11-20 22:57:42,505 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7850, loss[loss=0.1024, simple_loss=0.1345, pruned_loss=0.02695, audio_tagging_loss=0.008192, over 16021.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.09784, pruned_loss=0.01809, audio_tagging_loss=0.009986, over 3042958.42 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:57:51,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1254693.3333333333, ans=0.0
2023-11-20 22:58:06,462 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.33 vs. limit=12.0
2023-11-20 22:58:09,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1254826.6666666667, ans=0.1
2023-11-20 22:58:27,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1254893.3333333333, ans=0.125
2023-11-20 22:58:33,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1254960.0, ans=0.1
2023-11-20 22:58:39,027 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188250
2023-11-20 22:58:40,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1254960.0, ans=0.125
2023-11-20 22:58:40,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1254960.0, ans=0.125
2023-11-20 22:58:47,247 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7900, loss[loss=0.08583, simple_loss=0.1047, pruned_loss=0.02187, audio_tagging_loss=0.01164, over 15274.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.09911, pruned_loss=0.01846, audio_tagging_loss=0.009983, over 3045151.27 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:59:05,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1255093.3333333333, ans=0.0
2023-11-20 22:59:09,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1255093.3333333333, ans=0.125
2023-11-20 22:59:10,911 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.310e+01 8.933e+01 9.750e+01 1.293e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-20 22:59:14,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1255160.0, ans=0.2
2023-11-20 22:59:26,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1255226.6666666667, ans=0.0
2023-11-20 22:59:27,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1255226.6666666667, ans=0.125
2023-11-20 22:59:43,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188300
2023-11-20 22:59:44,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1255293.3333333333, ans=0.0
2023-11-20 22:59:50,529 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 7950, loss[loss=0.08522, simple_loss=0.1064, pruned_loss=0.0195, audio_tagging_loss=0.01251, over 15027.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09905, pruned_loss=0.01831, audio_tagging_loss=0.01003, over 3055285.18 frames. ], batch size: 55, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:00:04,581 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:00:08,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1255426.6666666667, ans=0.0
2023-11-20 23:00:16,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1255493.3333333333, ans=0.0
2023-11-20 23:00:21,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.55 vs. limit=6.0
2023-11-20 23:00:43,723 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.63 vs. limit=15.0
2023-11-20 23:00:45,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1255626.6666666667, ans=0.125
2023-11-20 23:00:46,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188350
2023-11-20 23:00:53,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1255693.3333333333, ans=0.125
2023-11-20 23:00:54,424 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8000, loss[loss=0.1002, simple_loss=0.1276, pruned_loss=0.02717, audio_tagging_loss=0.009224, over 15436.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09886, pruned_loss=0.01816, audio_tagging_loss=0.01014, over 3048030.95 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:01:11,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1255760.0, ans=0.04949747468305833
2023-11-20 23:01:19,524 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.786e+01 8.042e+01 8.522e+01 9.463e+01 1.266e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-20 23:01:29,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1255826.6666666667, ans=0.125
2023-11-20 23:01:50,573 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188400
2023-11-20 23:01:59,856 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8050, loss[loss=0.07148, simple_loss=0.09585, pruned_loss=0.01423, audio_tagging_loss=0.009328, over 15876.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09848, pruned_loss=0.01807, audio_tagging_loss=0.0102, over 3047876.24 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:02:00,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1256026.6666666667, ans=0.0
2023-11-20 23:02:28,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1256160.0, ans=0.0
2023-11-20 23:02:54,457 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.42 vs. limit=15.0
2023-11-20 23:02:56,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188450
2023-11-20 23:03:03,273 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8100, loss[loss=0.06814, simple_loss=0.09109, pruned_loss=0.01347, audio_tagging_loss=0.009124, over 15179.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09798, pruned_loss=0.01801, audio_tagging_loss=0.01014, over 3049841.17 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:03:12,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1256360.0, ans=0.125
2023-11-20 23:03:26,967 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.476e+01 8.237e+01 9.309e+01 1.004e+02 1.229e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-20 23:03:48,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1256560.0, ans=0.1
2023-11-20 23:03:49,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1256560.0, ans=0.1
2023-11-20 23:03:58,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188500
2023-11-20 23:04:05,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1256693.3333333333, ans=0.125
2023-11-20 23:04:06,654 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8150, loss[loss=0.0919, simple_loss=0.1306, pruned_loss=0.01933, audio_tagging_loss=0.007282, over 16479.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09908, pruned_loss=0.01816, audio_tagging_loss=0.009925, over 3048537.58 frames. ], batch size: 60, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:04:38,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1256826.6666666667, ans=0.025
2023-11-20 23:05:02,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188550
2023-11-20 23:05:08,396 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:05:09,559 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8200, loss[loss=0.09443, simple_loss=0.123, pruned_loss=0.0227, audio_tagging_loss=0.01024, over 14762.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09817, pruned_loss=0.01786, audio_tagging_loss=0.009865, over 3053662.03 frames. ], batch size: 54, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:05:21,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.04 vs. limit=6.0
2023-11-20 23:05:34,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.92 vs. limit=15.0
2023-11-20 23:05:34,611 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.637e+01 8.183e+01 8.865e+01 9.614e+01 1.183e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-20 23:06:00,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1257293.3333333333, ans=0.0
2023-11-20 23:06:04,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1257293.3333333333, ans=0.0
2023-11-20 23:06:07,404 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188600
2023-11-20 23:06:15,086 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8250, loss[loss=0.1078, simple_loss=0.1376, pruned_loss=0.02972, audio_tagging_loss=0.009312, over 15290.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09782, pruned_loss=0.01794, audio_tagging_loss=0.009793, over 3053651.02 frames. ], batch size: 55, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:06:16,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1257360.0, ans=0.125
2023-11-20 23:06:24,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1257360.0, ans=0.015
2023-11-20 23:06:30,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1257426.6666666667, ans=0.125
2023-11-20 23:06:51,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1257493.3333333333, ans=0.05
2023-11-20 23:07:10,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1257626.6666666667, ans=0.0
2023-11-20 23:07:11,209 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188650
2023-11-20 23:07:17,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=5.54 vs. limit=15.0
2023-11-20 23:07:18,994 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8300, loss[loss=0.07959, simple_loss=0.1019, pruned_loss=0.01787, audio_tagging_loss=0.01077, over 14847.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.0973, pruned_loss=0.01759, audio_tagging_loss=0.009867, over 3045971.20 frames. ], batch size: 55, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:07:20,885 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.86 vs. limit=15.0
2023-11-20 23:07:24,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1257693.3333333333, ans=0.1
2023-11-20 23:07:43,183 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.445e+01 8.113e+01 8.858e+01 9.502e+01 1.553e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-20 23:08:08,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1257893.3333333333, ans=0.1
2023-11-20 23:08:09,857 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.88 vs. limit=15.0
2023-11-20 23:08:11,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1257960.0, ans=0.125
2023-11-20 23:08:15,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188700
2023-11-20 23:08:22,687 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8350, loss[loss=0.05503, simple_loss=0.0658, pruned_loss=0.0101, audio_tagging_loss=0.01202, over 17085.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.09753, pruned_loss=0.01762, audio_tagging_loss=0.009802, over 3050406.48 frames. ], batch size: 66, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:08:31,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1258026.6666666667, ans=0.1
2023-11-20 23:08:49,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1258160.0, ans=0.125
2023-11-20 23:08:49,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1258160.0, ans=0.125
2023-11-20 23:08:58,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1258160.0, ans=0.125
2023-11-20 23:09:05,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.37 vs. limit=15.0
2023-11-20 23:09:07,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1258226.6666666667, ans=0.0
2023-11-20 23:09:09,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1258226.6666666667, ans=0.125
2023-11-20 23:09:19,702 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188750
2023-11-20 23:09:23,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1258293.3333333333, ans=0.125
2023-11-20 23:09:27,460 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8400, loss[loss=0.07877, simple_loss=0.09931, pruned_loss=0.02013, audio_tagging_loss=0.008978, over 14626.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.0972, pruned_loss=0.0178, audio_tagging_loss=0.009683, over 3049607.35 frames. ], batch size: 56, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:09:39,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.07 vs. limit=15.0
2023-11-20 23:09:45,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1258426.6666666667, ans=0.09899494936611666
2023-11-20 23:09:51,132 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.077e+01 8.855e+01 9.404e+01 1.397e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-20 23:09:56,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1258493.3333333333, ans=0.0
2023-11-20 23:10:23,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188800
2023-11-20 23:10:31,297 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8450, loss[loss=0.09293, simple_loss=0.117, pruned_loss=0.02737, audio_tagging_loss=0.007062, over 15594.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09607, pruned_loss=0.01757, audio_tagging_loss=0.009772, over 3053256.17 frames. ], batch size: 59, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:10:38,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1258693.3333333333, ans=0.125
2023-11-20 23:10:54,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1258760.0, ans=0.0
2023-11-20 23:11:00,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1258826.6666666667, ans=0.95
2023-11-20 23:11:04,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.39 vs. limit=15.0
2023-11-20 23:11:14,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1258893.3333333333, ans=0.0
2023-11-20 23:11:14,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1258893.3333333333, ans=0.125
2023-11-20 23:11:27,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188850
2023-11-20 23:11:34,532 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8500, loss[loss=0.08029, simple_loss=0.1004, pruned_loss=0.02047, audio_tagging_loss=0.00964, over 14423.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.0962, pruned_loss=0.0177, audio_tagging_loss=0.009864, over 3046874.81 frames. ], batch size: 56, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:11:38,526 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:11:45,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1259026.6666666667, ans=0.125
2023-11-20 23:11:59,750 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.153e+01 8.872e+01 9.707e+01 1.209e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-20 23:12:10,644 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.70 vs. limit=12.0
2023-11-20 23:12:19,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1259226.6666666667, ans=0.125
2023-11-20 23:12:31,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188900
2023-11-20 23:12:39,226 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8550, loss[loss=0.07811, simple_loss=0.09099, pruned_loss=0.01788, audio_tagging_loss=0.01474, over 14891.00 frames. ], tot_loss[loss=0.07639, simple_loss=0.0972, pruned_loss=0.01792, audio_tagging_loss=0.009875, over 3051218.55 frames. ], batch size: 58, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:12:40,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1259360.0, ans=0.0
2023-11-20 23:12:41,099 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.02 vs. limit=15.0
2023-11-20 23:12:57,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.76 vs. limit=15.0
2023-11-20 23:13:24,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1259560.0, ans=0.125
2023-11-20 23:13:27,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1259560.0, ans=0.2
2023-11-20 23:13:29,312 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.56 vs. limit=10.0
2023-11-20 23:13:34,992 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 188950
2023-11-20 23:13:42,788 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8600, loss[loss=0.0906, simple_loss=0.1218, pruned_loss=0.02189, audio_tagging_loss=0.007793, over 14579.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09729, pruned_loss=0.01795, audio_tagging_loss=0.00991, over 3049250.65 frames. ], batch size: 53, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:14:06,714 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.038e+01 8.664e+01 9.287e+01 1.205e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-20 23:14:24,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1259893.3333333333, ans=0.1
2023-11-20 23:14:26,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1259893.3333333333, ans=0.125
2023-11-20 23:14:39,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189000
2023-11-20 23:14:42,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1259960.0, ans=0.2
2023-11-20 23:14:47,234 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8650, loss[loss=0.07056, simple_loss=0.0894, pruned_loss=0.01563, audio_tagging_loss=0.01023, over 16026.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09879, pruned_loss=0.01834, audio_tagging_loss=0.009899, over 3051180.07 frames. ], batch size: 61, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:14:53,844 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.45 vs. limit=6.0
2023-11-20 23:14:55,292 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.68 vs. limit=15.0
2023-11-20 23:15:05,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1260093.3333333333, ans=0.125
2023-11-20 23:15:18,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1260160.0, ans=0.125
2023-11-20 23:15:43,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189050
2023-11-20 23:15:52,070 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8700, loss[loss=0.09635, simple_loss=0.1123, pruned_loss=0.03161, audio_tagging_loss=0.008573, over 14403.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.0992, pruned_loss=0.01842, audio_tagging_loss=0.009879, over 3050680.50 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:15:54,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1260360.0, ans=0.125
2023-11-20 23:16:17,586 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.100e+01 8.533e+01 9.341e+01 1.224e+02, threshold=1.707e+02, percent-clipped=0.0
2023-11-20 23:16:26,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1260493.3333333333, ans=0.0
2023-11-20 23:16:27,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=12.0
2023-11-20 23:16:35,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1260560.0, ans=0.125
2023-11-20 23:16:48,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189100
2023-11-20 23:16:53,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1260626.6666666667, ans=0.2
2023-11-20 23:16:55,193 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8750, loss[loss=0.08939, simple_loss=0.1334, pruned_loss=0.01699, audio_tagging_loss=0.005729, over 14891.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.099, pruned_loss=0.01811, audio_tagging_loss=0.009945, over 3050648.00 frames. ], batch size: 53, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:17:03,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.27 vs. limit=15.0
2023-11-20 23:17:06,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1260693.3333333333, ans=0.125
2023-11-20 23:17:20,970 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.00 vs. limit=15.0
2023-11-20 23:17:27,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1260826.6666666667, ans=0.0
2023-11-20 23:17:29,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1260826.6666666667, ans=0.2
2023-11-20 23:17:48,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1260960.0, ans=0.04949747468305833
2023-11-20 23:17:51,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189150
2023-11-20 23:17:52,770 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=6.571e-02
2023-11-20 23:17:58,598 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8800, loss[loss=0.05704, simple_loss=0.07012, pruned_loss=0.01098, audio_tagging_loss=0.011, over 15188.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09946, pruned_loss=0.01829, audio_tagging_loss=0.01008, over 3058985.52 frames. ], batch size: 59, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:18:11,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1261093.3333333333, ans=0.0
2023-11-20 23:18:16,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1261093.3333333333, ans=0.0
2023-11-20 23:18:17,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1261093.3333333333, ans=0.125
2023-11-20 23:18:18,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1261093.3333333333, ans=0.125
2023-11-20 23:18:24,231 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.201e+01 8.889e+01 9.657e+01 1.880e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-20 23:18:29,705 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.44 vs. limit=15.0
2023-11-20 23:18:38,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1261226.6666666667, ans=0.125
2023-11-20 23:18:38,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1261226.6666666667, ans=0.1
2023-11-20 23:18:38,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1261226.6666666667, ans=0.125
2023-11-20 23:18:54,534 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189200
2023-11-20 23:19:03,174 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8850, loss[loss=0.07205, simple_loss=0.09416, pruned_loss=0.01539, audio_tagging_loss=0.009578, over 15634.00 frames. ], tot_loss[loss=0.07743, simple_loss=0.09877, pruned_loss=0.01803, audio_tagging_loss=0.01001, over 3057243.69 frames. ], batch size: 58, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:19:12,996 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:19:24,491 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.47 vs. limit=15.0
2023-11-20 23:19:51,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1261560.0, ans=0.2
2023-11-20 23:19:58,744 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189250
2023-11-20 23:20:05,956 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8900, loss[loss=0.07371, simple_loss=0.09649, pruned_loss=0.01438, audio_tagging_loss=0.01108, over 15319.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09931, pruned_loss=0.01819, audio_tagging_loss=0.009782, over 3050633.83 frames. ], batch size: 59, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:20:06,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0
2023-11-20 23:20:08,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1261693.3333333333, ans=0.125
2023-11-20 23:20:13,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1261693.3333333333, ans=0.2
2023-11-20 23:20:21,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1261760.0, ans=0.0
2023-11-20 23:20:29,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.36 vs. limit=10.0
2023-11-20 23:20:31,210 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.227e+01 8.677e+01 9.378e+01 1.174e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-20 23:20:55,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1261960.0, ans=0.125
2023-11-20 23:20:57,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=1261960.0, ans=0.02
2023-11-20 23:20:57,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1261960.0, ans=0.125
2023-11-20 23:21:01,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189300
2023-11-20 23:21:03,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1261960.0, ans=0.2
2023-11-20 23:21:08,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.86 vs. limit=6.0
2023-11-20 23:21:09,622 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 8950, loss[loss=0.09819, simple_loss=0.1233, pruned_loss=0.02732, audio_tagging_loss=0.009216, over 14853.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09955, pruned_loss=0.01828, audio_tagging_loss=0.009706, over 3049710.74 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:21:12,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1262026.6666666667, ans=0.125
2023-11-20 23:21:12,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1262026.6666666667, ans=0.125
2023-11-20 23:21:18,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.81 vs. limit=12.0
2023-11-20 23:21:20,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1262093.3333333333, ans=0.125
2023-11-20 23:21:53,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1262226.6666666667, ans=0.0
2023-11-20 23:22:04,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1262293.3333333333, ans=0.2
2023-11-20 23:22:05,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189350
2023-11-20 23:22:13,047 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9000, loss[loss=0.08519, simple_loss=0.1146, pruned_loss=0.02089, audio_tagging_loss=0.006976, over 15208.00 frames. ], tot_loss[loss=0.0786, simple_loss=0.1005, pruned_loss=0.01867, audio_tagging_loss=0.009664, over 3046979.64 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:22:13,047 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-20 23:22:46,873 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.9788, 5.8758, 5.6311, 5.5600], device='cuda:2')
2023-11-20 23:22:55,287 INFO [train_asr.py:1253] (2/4) Epoch 16, validation: loss=0.06115, simple_loss=0.05296, pruned_loss=0.005511, audio_tagging_loss=0.02916, over 4681554.00 frames. 
2023-11-20 23:22:55,288 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-20 23:23:06,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1262360.0, ans=0.0
2023-11-20 23:23:06,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1262360.0, ans=0.125
2023-11-20 23:23:14,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.50 vs. limit=15.0
2023-11-20 23:23:17,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1262426.6666666667, ans=0.09899494936611666
2023-11-20 23:23:22,726 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.262e+01 9.175e+01 9.933e+01 1.311e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-20 23:23:23,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1262493.3333333333, ans=0.125
2023-11-20 23:23:41,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1262560.0, ans=0.125
2023-11-20 23:23:51,966 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189400
2023-11-20 23:23:59,683 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9050, loss[loss=0.06049, simple_loss=0.07177, pruned_loss=0.01359, audio_tagging_loss=0.01101, over 16136.00 frames. ], tot_loss[loss=0.07832, simple_loss=0.09989, pruned_loss=0.0187, audio_tagging_loss=0.009679, over 3047845.18 frames. ], batch size: 61, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:24:04,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1262693.3333333333, ans=0.2
2023-11-20 23:24:07,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1262693.3333333333, ans=0.0
2023-11-20 23:24:26,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1262826.6666666667, ans=0.2
2023-11-20 23:24:27,605 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.49 vs. limit=15.0
2023-11-20 23:24:39,206 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.39 vs. limit=15.0
2023-11-20 23:24:46,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.41 vs. limit=22.5
2023-11-20 23:24:56,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189450
2023-11-20 23:24:56,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1262960.0, ans=0.125
2023-11-20 23:25:04,476 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9100, loss[loss=0.08568, simple_loss=0.1175, pruned_loss=0.0191, audio_tagging_loss=0.007831, over 15990.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09952, pruned_loss=0.01844, audio_tagging_loss=0.009591, over 3042284.23 frames. ], batch size: 60, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:25:04,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1263026.6666666667, ans=0.125
2023-11-20 23:25:14,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1263026.6666666667, ans=0.125
2023-11-20 23:25:25,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1263093.3333333333, ans=0.0
2023-11-20 23:25:30,559 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.518e+01 8.183e+01 8.786e+01 9.651e+01 1.253e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-20 23:25:32,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1263160.0, ans=0.125
2023-11-20 23:26:00,247 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189500
2023-11-20 23:26:07,497 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9150, loss[loss=0.07399, simple_loss=0.08623, pruned_loss=0.02103, audio_tagging_loss=0.00984, over 14640.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09935, pruned_loss=0.01841, audio_tagging_loss=0.009537, over 3054104.55 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:26:14,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1263360.0, ans=0.125
2023-11-20 23:26:21,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1263426.6666666667, ans=0.125
2023-11-20 23:26:53,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1263560.0, ans=0.2
2023-11-20 23:26:54,769 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.36 vs. limit=22.5
2023-11-20 23:27:03,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189550
2023-11-20 23:27:10,371 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9200, loss[loss=0.1007, simple_loss=0.1334, pruned_loss=0.02521, audio_tagging_loss=0.008802, over 16385.00 frames. ], tot_loss[loss=0.07707, simple_loss=0.09845, pruned_loss=0.01828, audio_tagging_loss=0.009564, over 3051924.44 frames. ], batch size: 59, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:27:37,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.197e+01 8.151e+01 8.702e+01 9.490e+01 1.171e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-20 23:27:48,037 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.81 vs. limit=6.0
2023-11-20 23:28:06,289 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189600
2023-11-20 23:28:12,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.07 vs. limit=15.0
2023-11-20 23:28:12,326 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.36 vs. limit=15.0
2023-11-20 23:28:14,507 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9250, loss[loss=0.087, simple_loss=0.1006, pruned_loss=0.02448, audio_tagging_loss=0.01224, over 14470.00 frames. ], tot_loss[loss=0.07647, simple_loss=0.09757, pruned_loss=0.01804, audio_tagging_loss=0.009646, over 3050688.45 frames. ], batch size: 55, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:28:14,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1264026.6666666667, ans=0.0
2023-11-20 23:28:21,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1264026.6666666667, ans=0.125
2023-11-20 23:28:31,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1264093.3333333333, ans=0.025
2023-11-20 23:28:49,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1264160.0, ans=0.125
2023-11-20 23:28:49,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1264160.0, ans=0.0
2023-11-20 23:29:10,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189650
2023-11-20 23:29:18,278 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9300, loss[loss=0.09616, simple_loss=0.1282, pruned_loss=0.02464, audio_tagging_loss=0.00744, over 14670.00 frames. ], tot_loss[loss=0.07658, simple_loss=0.09779, pruned_loss=0.01802, audio_tagging_loss=0.009663, over 3057938.03 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:29:24,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1264360.0, ans=0.125
2023-11-20 23:29:36,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1264426.6666666667, ans=0.0
2023-11-20 23:29:38,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1264426.6666666667, ans=0.0
2023-11-20 23:29:39,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1264426.6666666667, ans=0.125
2023-11-20 23:29:44,507 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.124e+01 8.662e+01 9.573e+01 1.521e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-20 23:30:14,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189700
2023-11-20 23:30:22,410 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9350, loss[loss=0.0674, simple_loss=0.08629, pruned_loss=0.01479, audio_tagging_loss=0.009466, over 14979.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.0985, pruned_loss=0.01819, audio_tagging_loss=0.009683, over 3050768.27 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:30:25,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1264693.3333333333, ans=0.125
2023-11-20 23:30:28,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1264693.3333333333, ans=0.1
2023-11-20 23:30:31,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1264693.3333333333, ans=0.2
2023-11-20 23:30:47,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1264826.6666666667, ans=0.125
2023-11-20 23:31:07,563 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:31:13,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1264960.0, ans=0.0
2023-11-20 23:31:18,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189750
2023-11-20 23:31:19,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1264960.0, ans=0.125
2023-11-20 23:31:21,086 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.68 vs. limit=15.0
2023-11-20 23:31:26,049 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9400, loss[loss=0.08294, simple_loss=0.1073, pruned_loss=0.01969, audio_tagging_loss=0.009625, over 15492.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.0979, pruned_loss=0.01816, audio_tagging_loss=0.009869, over 3046885.07 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:31:36,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1265026.6666666667, ans=0.125
2023-11-20 23:31:47,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1265093.3333333333, ans=0.125
2023-11-20 23:31:53,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1265160.0, ans=0.125
2023-11-20 23:31:54,703 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.398e+01 8.103e+01 8.759e+01 9.361e+01 1.221e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 23:32:04,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1265226.6666666667, ans=0.125
2023-11-20 23:32:22,979 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189800
2023-11-20 23:32:27,727 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:32:31,400 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9450, loss[loss=0.09254, simple_loss=0.11, pruned_loss=0.02681, audio_tagging_loss=0.01073, over 15477.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09717, pruned_loss=0.01796, audio_tagging_loss=0.009892, over 3046399.48 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:32:50,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1265426.6666666667, ans=0.125
2023-11-20 23:33:09,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1265560.0, ans=0.025
2023-11-20 23:33:27,303 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189850
2023-11-20 23:33:30,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1265626.6666666667, ans=0.125
2023-11-20 23:33:32,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1265626.6666666667, ans=0.0
2023-11-20 23:33:34,926 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9500, loss[loss=0.08531, simple_loss=0.1115, pruned_loss=0.02268, audio_tagging_loss=0.006877, over 15093.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09776, pruned_loss=0.01812, audio_tagging_loss=0.01001, over 3047792.54 frames. ], batch size: 54, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:33:56,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1265760.0, ans=0.125
2023-11-20 23:34:02,680 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.779e+01 8.467e+01 9.097e+01 1.010e+02 1.655e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-20 23:34:22,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1265893.3333333333, ans=0.1
2023-11-20 23:34:30,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189900
2023-11-20 23:34:31,139 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:34:38,066 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9550, loss[loss=0.07424, simple_loss=0.0988, pruned_loss=0.01461, audio_tagging_loss=0.01024, over 16487.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09917, pruned_loss=0.01829, audio_tagging_loss=0.01006, over 3049834.14 frames. ], batch size: 59, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:34:56,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1266093.3333333333, ans=0.125
2023-11-20 23:35:10,737 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0
2023-11-20 23:35:12,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1266160.0, ans=15.0
2023-11-20 23:35:15,663 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.92 vs. limit=6.0
2023-11-20 23:35:30,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1266293.3333333333, ans=0.1
2023-11-20 23:35:35,276 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 189950
2023-11-20 23:35:36,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1266293.3333333333, ans=0.1
2023-11-20 23:35:42,338 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9600, loss[loss=0.06538, simple_loss=0.07202, pruned_loss=0.01417, audio_tagging_loss=0.01519, over 14952.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.0982, pruned_loss=0.01805, audio_tagging_loss=0.01021, over 3055670.73 frames. ], batch size: 59, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:35:50,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1266360.0, ans=0.125
2023-11-20 23:36:10,699 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.084e+01 8.487e+01 9.415e+01 1.113e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-20 23:36:24,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1266560.0, ans=0.125
2023-11-20 23:36:25,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1266560.0, ans=0.0
2023-11-20 23:36:37,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1266626.6666666667, ans=0.1
2023-11-20 23:36:39,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190000
2023-11-20 23:36:40,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1266626.6666666667, ans=0.0
2023-11-20 23:36:47,529 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9650, loss[loss=0.07198, simple_loss=0.08682, pruned_loss=0.01794, audio_tagging_loss=0.01063, over 15931.00 frames. ], tot_loss[loss=0.07707, simple_loss=0.09776, pruned_loss=0.01794, audio_tagging_loss=0.01025, over 3046874.53 frames. ], batch size: 61, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:37:12,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1266826.6666666667, ans=0.0
2023-11-20 23:37:20,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1266826.6666666667, ans=0.0
2023-11-20 23:37:24,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1266893.3333333333, ans=0.125
2023-11-20 23:37:44,117 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190050
2023-11-20 23:37:51,311 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9700, loss[loss=0.0766, simple_loss=0.1031, pruned_loss=0.01524, audio_tagging_loss=0.009811, over 14895.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.09704, pruned_loss=0.01763, audio_tagging_loss=0.01008, over 3045249.82 frames. ], batch size: 55, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:38:17,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1267160.0, ans=0.125
2023-11-20 23:38:21,186 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.452e+01 8.015e+01 8.951e+01 9.775e+01 1.301e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-20 23:38:24,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1267160.0, ans=0.125
2023-11-20 23:38:48,747 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190100
2023-11-20 23:38:56,267 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9750, loss[loss=0.06398, simple_loss=0.07674, pruned_loss=0.01497, audio_tagging_loss=0.01063, over 15170.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09698, pruned_loss=0.01771, audio_tagging_loss=0.009906, over 3042744.94 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:39:06,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1267360.0, ans=0.5
2023-11-20 23:39:18,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1267426.6666666667, ans=0.125
2023-11-20 23:39:32,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1267493.3333333333, ans=0.125
2023-11-20 23:39:33,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1267560.0, ans=0.125
2023-11-20 23:39:34,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1267560.0, ans=0.1
2023-11-20 23:39:49,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1267626.6666666667, ans=0.0
2023-11-20 23:39:52,895 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190150
2023-11-20 23:40:00,677 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9800, loss[loss=0.08453, simple_loss=0.1202, pruned_loss=0.01773, audio_tagging_loss=0.006719, over 15239.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09654, pruned_loss=0.01752, audio_tagging_loss=0.009888, over 3045867.82 frames. ], batch size: 53, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:40:17,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1267760.0, ans=0.0
2023-11-20 23:40:30,320 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.597e+01 7.945e+01 8.740e+01 9.752e+01 1.304e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-20 23:40:35,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1267826.6666666667, ans=0.1
2023-11-20 23:40:48,206 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.30 vs. limit=15.0
2023-11-20 23:40:56,338 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:40:57,666 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190200
2023-11-20 23:41:04,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1268026.6666666667, ans=0.0
2023-11-20 23:41:05,209 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9850, loss[loss=0.09493, simple_loss=0.1229, pruned_loss=0.02458, audio_tagging_loss=0.008885, over 16008.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09812, pruned_loss=0.01795, audio_tagging_loss=0.0097, over 3048559.37 frames. ], batch size: 58, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:41:43,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1268226.6666666667, ans=0.0
2023-11-20 23:41:44,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1268226.6666666667, ans=0.125
2023-11-20 23:41:59,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1268293.3333333333, ans=0.125
2023-11-20 23:42:00,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190250
2023-11-20 23:42:03,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1268293.3333333333, ans=0.0
2023-11-20 23:42:05,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1268293.3333333333, ans=0.0
2023-11-20 23:42:08,616 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9900, loss[loss=0.05662, simple_loss=0.06945, pruned_loss=0.01, audio_tagging_loss=0.01189, over 15367.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09853, pruned_loss=0.01777, audio_tagging_loss=0.009625, over 3053954.31 frames. ], batch size: 60, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:42:10,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1268360.0, ans=0.05
2023-11-20 23:42:26,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1268426.6666666667, ans=0.2
2023-11-20 23:42:37,742 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.053e+01 8.764e+01 9.412e+01 1.307e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-20 23:42:43,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1268493.3333333333, ans=0.2
2023-11-20 23:42:45,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1268560.0, ans=0.0
2023-11-20 23:42:59,589 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.12 vs. limit=15.0
2023-11-20 23:43:02,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1268626.6666666667, ans=0.125
2023-11-20 23:43:04,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190300
2023-11-20 23:43:09,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.29 vs. limit=12.0
2023-11-20 23:43:12,091 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 9950, loss[loss=0.06435, simple_loss=0.08724, pruned_loss=0.01378, audio_tagging_loss=0.006948, over 15139.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.09777, pruned_loss=0.01761, audio_tagging_loss=0.009728, over 3043328.96 frames. ], batch size: 59, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:43:29,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1268760.0, ans=0.2
2023-11-20 23:43:39,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1268826.6666666667, ans=0.2
2023-11-20 23:44:08,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190350
2023-11-20 23:44:16,752 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10000, loss[loss=0.1067, simple_loss=0.1399, pruned_loss=0.03186, audio_tagging_loss=0.004877, over 15026.00 frames. ], tot_loss[loss=0.07574, simple_loss=0.09692, pruned_loss=0.01753, audio_tagging_loss=0.009749, over 3047445.22 frames. ], batch size: 54, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:44:40,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1269093.3333333333, ans=0.125
2023-11-20 23:44:45,925 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.173e+01 8.806e+01 9.776e+01 1.486e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-20 23:44:46,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1269160.0, ans=0.1
2023-11-20 23:44:55,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1269226.6666666667, ans=0.0
2023-11-20 23:45:04,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1269226.6666666667, ans=0.0
2023-11-20 23:45:04,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1269226.6666666667, ans=0.125
2023-11-20 23:45:13,490 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190400
2023-11-20 23:45:16,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1269293.3333333333, ans=0.125
2023-11-20 23:45:21,699 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10050, loss[loss=0.07523, simple_loss=0.1017, pruned_loss=0.01502, audio_tagging_loss=0.009348, over 14568.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09787, pruned_loss=0.01773, audio_tagging_loss=0.009745, over 3048514.71 frames. ], batch size: 53, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:45:29,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.94 vs. limit=15.0
2023-11-20 23:45:46,629 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.59 vs. limit=22.5
2023-11-20 23:45:52,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1269493.3333333333, ans=0.125
2023-11-20 23:46:08,686 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.94 vs. limit=15.0
2023-11-20 23:46:17,995 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190450
2023-11-20 23:46:25,269 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10100, loss[loss=0.06409, simple_loss=0.08156, pruned_loss=0.01441, audio_tagging_loss=0.008902, over 15131.00 frames. ], tot_loss[loss=0.07655, simple_loss=0.09807, pruned_loss=0.01778, audio_tagging_loss=0.009734, over 3052566.22 frames. ], batch size: 58, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:46:47,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1269760.0, ans=0.09899494936611666
2023-11-20 23:46:48,130 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.07 vs. limit=22.5
2023-11-20 23:46:56,617 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.119e+01 8.749e+01 9.415e+01 1.269e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 23:47:15,147 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:47:22,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190500
2023-11-20 23:47:23,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1269960.0, ans=0.09899494936611666
2023-11-20 23:47:29,417 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10150, loss[loss=0.06195, simple_loss=0.07476, pruned_loss=0.009998, audio_tagging_loss=0.01457, over 15586.00 frames. ], tot_loss[loss=0.07676, simple_loss=0.09813, pruned_loss=0.01786, audio_tagging_loss=0.009843, over 3053146.81 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:47:31,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1270026.6666666667, ans=0.125
2023-11-20 23:47:36,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1270026.6666666667, ans=0.2
2023-11-20 23:47:45,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1270093.3333333333, ans=0.125
2023-11-20 23:47:57,657 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:48:23,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1270293.3333333333, ans=0.125
2023-11-20 23:48:25,840 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190550
2023-11-20 23:48:33,569 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10200, loss[loss=0.05619, simple_loss=0.06877, pruned_loss=0.01091, audio_tagging_loss=0.01089, over 16349.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09837, pruned_loss=0.01802, audio_tagging_loss=0.009924, over 3058301.79 frames. ], batch size: 63, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:48:47,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1270426.6666666667, ans=0.1
2023-11-20 23:48:51,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1270426.6666666667, ans=0.125
2023-11-20 23:48:54,067 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:48:57,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1270493.3333333333, ans=0.0
2023-11-20 23:49:02,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1270493.3333333333, ans=0.125
2023-11-20 23:49:02,934 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.808e+01 8.167e+01 8.853e+01 9.625e+01 1.935e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-20 23:49:08,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1270493.3333333333, ans=0.125
2023-11-20 23:49:10,567 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:49:28,628 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190600
2023-11-20 23:49:32,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1270626.6666666667, ans=0.125
2023-11-20 23:49:35,988 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10250, loss[loss=0.09906, simple_loss=0.1214, pruned_loss=0.02721, audio_tagging_loss=0.01112, over 15489.00 frames. ], tot_loss[loss=0.0774, simple_loss=0.0984, pruned_loss=0.01816, audio_tagging_loss=0.01005, over 3059053.69 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:50:01,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1270826.6666666667, ans=0.125
2023-11-20 23:50:01,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1270826.6666666667, ans=0.2
2023-11-20 23:50:11,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1270826.6666666667, ans=0.125
2023-11-20 23:50:32,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190650
2023-11-20 23:50:39,993 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10300, loss[loss=0.06171, simple_loss=0.07881, pruned_loss=0.01115, audio_tagging_loss=0.01115, over 15169.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09792, pruned_loss=0.01802, audio_tagging_loss=0.01003, over 3059764.56 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:50:40,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1271026.6666666667, ans=0.0
2023-11-20 23:50:42,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.23 vs. limit=15.0
2023-11-20 23:50:46,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1271026.6666666667, ans=0.125
2023-11-20 23:50:57,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1271093.3333333333, ans=0.0
2023-11-20 23:51:07,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1271160.0, ans=0.1
2023-11-20 23:51:09,509 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.59 vs. limit=15.0
2023-11-20 23:51:11,224 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.201e+01 8.870e+01 9.848e+01 1.361e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-20 23:51:21,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1271226.6666666667, ans=0.125
2023-11-20 23:51:35,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.23 vs. limit=10.0
2023-11-20 23:51:36,419 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190700
2023-11-20 23:51:38,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1271293.3333333333, ans=0.125
2023-11-20 23:51:44,213 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10350, loss[loss=0.07989, simple_loss=0.1072, pruned_loss=0.0144, audio_tagging_loss=0.01188, over 17109.00 frames. ], tot_loss[loss=0.07724, simple_loss=0.09809, pruned_loss=0.01804, audio_tagging_loss=0.01016, over 3056646.29 frames. ], batch size: 63, lr: 4.25e-03, grad_scale: 8.0
2023-11-20 23:52:10,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1271493.3333333333, ans=0.125
2023-11-20 23:52:21,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1271560.0, ans=0.125
2023-11-20 23:52:30,235 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.13 vs. limit=10.0
2023-11-20 23:52:34,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1271626.6666666667, ans=0.035
2023-11-20 23:52:37,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1271626.6666666667, ans=0.1
2023-11-20 23:52:39,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1271626.6666666667, ans=0.1
2023-11-20 23:52:39,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1271626.6666666667, ans=0.125
2023-11-20 23:52:40,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190750
2023-11-20 23:52:47,668 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10400, loss[loss=0.07488, simple_loss=0.09371, pruned_loss=0.01623, audio_tagging_loss=0.01179, over 15153.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09789, pruned_loss=0.01793, audio_tagging_loss=0.01026, over 3050577.32 frames. ], batch size: 56, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:52:59,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1271760.0, ans=0.1
2023-11-20 23:53:18,605 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.195e+01 8.838e+01 9.520e+01 1.388e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 23:53:43,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190800
2023-11-20 23:53:50,991 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10450, loss[loss=0.05692, simple_loss=0.06622, pruned_loss=0.01282, audio_tagging_loss=0.01099, over 14679.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09838, pruned_loss=0.01804, audio_tagging_loss=0.01009, over 3053186.17 frames. ], batch size: 59, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:54:01,057 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:54:10,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1272093.3333333333, ans=0.04949747468305833
2023-11-20 23:54:10,518 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.70 vs. limit=6.0
2023-11-20 23:54:26,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1272160.0, ans=0.07
2023-11-20 23:54:46,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190850
2023-11-20 23:54:51,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1272293.3333333333, ans=0.125
2023-11-20 23:54:53,825 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10500, loss[loss=0.1002, simple_loss=0.1352, pruned_loss=0.02485, audio_tagging_loss=0.007721, over 16417.00 frames. ], tot_loss[loss=0.07731, simple_loss=0.09873, pruned_loss=0.01804, audio_tagging_loss=0.009904, over 3052680.25 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:54:54,092 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:54:55,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1272360.0, ans=0.125
2023-11-20 23:55:04,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1272360.0, ans=0.125
2023-11-20 23:55:15,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1272426.6666666667, ans=0.1
2023-11-20 23:55:26,072 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.127e+01 8.997e+01 9.660e+01 1.233e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-20 23:55:30,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1272493.3333333333, ans=0.125
2023-11-20 23:55:35,380 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.82 vs. limit=15.0
2023-11-20 23:55:50,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190900
2023-11-20 23:55:57,180 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.78 vs. limit=15.0
2023-11-20 23:55:57,803 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10550, loss[loss=0.05924, simple_loss=0.07418, pruned_loss=0.0123, audio_tagging_loss=0.009839, over 16341.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09769, pruned_loss=0.01791, audio_tagging_loss=0.009811, over 3051046.97 frames. ], batch size: 65, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:56:32,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1272826.6666666667, ans=0.125
2023-11-20 23:56:36,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1272893.3333333333, ans=0.0
2023-11-20 23:56:53,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 190950
2023-11-20 23:57:01,284 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10600, loss[loss=0.05973, simple_loss=0.08183, pruned_loss=0.01002, audio_tagging_loss=0.008798, over 14112.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.0973, pruned_loss=0.01783, audio_tagging_loss=0.009804, over 3048314.06 frames. ], batch size: 55, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:57:05,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1273026.6666666667, ans=0.125
2023-11-20 23:57:06,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=15.0
2023-11-20 23:57:07,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1273026.6666666667, ans=0.125
2023-11-20 23:57:08,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.72 vs. limit=12.0
2023-11-20 23:57:15,129 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0
2023-11-20 23:57:20,048 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.90 vs. limit=22.5
2023-11-20 23:57:26,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1273160.0, ans=0.125
2023-11-20 23:57:29,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.52 vs. limit=15.0
2023-11-20 23:57:32,705 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 7.871e+01 8.482e+01 9.357e+01 1.125e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-20 23:57:55,520 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.76 vs. limit=22.5
2023-11-20 23:57:56,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191000
2023-11-20 23:58:03,693 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10650, loss[loss=0.07259, simple_loss=0.09202, pruned_loss=0.01475, audio_tagging_loss=0.01183, over 15399.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09786, pruned_loss=0.01774, audio_tagging_loss=0.009735, over 3049737.71 frames. ], batch size: 56, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:58:55,877 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:59:00,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191050
2023-11-20 23:59:07,270 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10700, loss[loss=0.07578, simple_loss=0.08428, pruned_loss=0.02179, audio_tagging_loss=0.01186, over 15461.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.09759, pruned_loss=0.01766, audio_tagging_loss=0.009733, over 3050067.02 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:59:09,075 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.83 vs. limit=15.0
2023-11-20 23:59:20,688 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.89 vs. limit=15.0
2023-11-20 23:59:38,723 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 7.953e+01 8.635e+01 9.470e+01 1.499e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-21 00:00:01,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1273960.0, ans=0.0
2023-11-21 00:00:03,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191100
2023-11-21 00:00:10,873 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10750, loss[loss=0.08455, simple_loss=0.1118, pruned_loss=0.02175, audio_tagging_loss=0.006921, over 14802.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09763, pruned_loss=0.01772, audio_tagging_loss=0.009663, over 3048604.88 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 16.0
2023-11-21 00:00:13,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1274026.6666666667, ans=0.0
2023-11-21 00:00:16,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1274026.6666666667, ans=0.125
2023-11-21 00:00:28,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1274093.3333333333, ans=0.2
2023-11-21 00:00:30,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1274093.3333333333, ans=0.1
2023-11-21 00:00:42,434 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=23.43 vs. limit=22.5
2023-11-21 00:01:06,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191150
2023-11-21 00:01:13,354 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10800, loss[loss=0.07216, simple_loss=0.09283, pruned_loss=0.01514, audio_tagging_loss=0.0106, over 15785.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09824, pruned_loss=0.01784, audio_tagging_loss=0.009675, over 3052075.78 frames. ], batch size: 60, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:01:19,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1274360.0, ans=0.95
2023-11-21 00:01:21,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1274360.0, ans=0.125
2023-11-21 00:01:30,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1274426.6666666667, ans=0.125
2023-11-21 00:01:30,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.50 vs. limit=15.0
2023-11-21 00:01:31,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1274426.6666666667, ans=0.0
2023-11-21 00:01:41,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1274493.3333333333, ans=0.125
2023-11-21 00:01:44,974 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.243e+01 8.007e+01 8.718e+01 9.213e+01 1.138e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 00:02:02,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1274626.6666666667, ans=0.0
2023-11-21 00:02:08,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191200
2023-11-21 00:02:17,039 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10850, loss[loss=0.07478, simple_loss=0.09526, pruned_loss=0.01824, audio_tagging_loss=0.0089, over 14911.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.0973, pruned_loss=0.01766, audio_tagging_loss=0.009716, over 3047039.72 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:02:17,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1274693.3333333333, ans=0.0
2023-11-21 00:02:19,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1274693.3333333333, ans=0.2
2023-11-21 00:02:20,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1274693.3333333333, ans=0.125
2023-11-21 00:02:22,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1274693.3333333333, ans=0.0
2023-11-21 00:02:24,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1274693.3333333333, ans=0.0
2023-11-21 00:02:39,904 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.75 vs. limit=15.0
2023-11-21 00:02:41,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1274826.6666666667, ans=0.1
2023-11-21 00:02:41,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1274826.6666666667, ans=0.0
2023-11-21 00:02:54,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1274893.3333333333, ans=0.1
2023-11-21 00:03:13,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191250
2023-11-21 00:03:15,949 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:03:21,473 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10900, loss[loss=0.06752, simple_loss=0.07922, pruned_loss=0.01395, audio_tagging_loss=0.01396, over 15189.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09733, pruned_loss=0.01765, audio_tagging_loss=0.009817, over 3054521.83 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:03:41,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1275093.3333333333, ans=0.125
2023-11-21 00:03:48,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1275160.0, ans=0.125
2023-11-21 00:03:52,725 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.158e+01 8.756e+01 9.674e+01 1.348e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 00:03:56,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1275160.0, ans=0.1
2023-11-21 00:03:59,460 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.74 vs. limit=15.0
2023-11-21 00:04:06,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1275226.6666666667, ans=0.125
2023-11-21 00:04:16,703 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191300
2023-11-21 00:04:19,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1275293.3333333333, ans=0.0
2023-11-21 00:04:23,768 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 10950, loss[loss=0.07324, simple_loss=0.09094, pruned_loss=0.01665, audio_tagging_loss=0.01112, over 16524.00 frames. ], tot_loss[loss=0.07661, simple_loss=0.09792, pruned_loss=0.01784, audio_tagging_loss=0.00981, over 3051374.22 frames. ], batch size: 62, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:04:30,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1275360.0, ans=0.125
2023-11-21 00:04:41,344 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.39 vs. limit=15.0
2023-11-21 00:05:15,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1275626.6666666667, ans=0.125
2023-11-21 00:05:19,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191350
2023-11-21 00:05:27,996 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11000, loss[loss=0.06707, simple_loss=0.08672, pruned_loss=0.01428, audio_tagging_loss=0.009436, over 15268.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.0991, pruned_loss=0.01805, audio_tagging_loss=0.009815, over 3055647.71 frames. ], batch size: 59, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:05:29,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1275693.3333333333, ans=0.1
2023-11-21 00:05:35,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1275693.3333333333, ans=0.2
2023-11-21 00:05:36,734 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:05:59,924 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.037e+01 8.729e+01 9.395e+01 1.274e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 00:06:02,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1275826.6666666667, ans=0.125
2023-11-21 00:06:17,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1275893.3333333333, ans=0.125
2023-11-21 00:06:17,914 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.26 vs. limit=10.0
2023-11-21 00:06:24,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191400
2023-11-21 00:06:30,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1275960.0, ans=0.2
2023-11-21 00:06:32,581 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11050, loss[loss=0.07404, simple_loss=0.1019, pruned_loss=0.01391, audio_tagging_loss=0.00918, over 15699.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09915, pruned_loss=0.01803, audio_tagging_loss=0.009856, over 3055411.24 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:06:36,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1276026.6666666667, ans=0.125
2023-11-21 00:06:51,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.82 vs. limit=12.0
2023-11-21 00:06:54,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1276093.3333333333, ans=0.125
2023-11-21 00:06:56,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1276093.3333333333, ans=0.1
2023-11-21 00:07:03,188 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.50 vs. limit=15.0
2023-11-21 00:07:17,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1276226.6666666667, ans=0.125
2023-11-21 00:07:29,118 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191450
2023-11-21 00:07:34,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1276293.3333333333, ans=0.125
2023-11-21 00:07:36,870 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11100, loss[loss=0.07026, simple_loss=0.0896, pruned_loss=0.01674, audio_tagging_loss=0.008721, over 15311.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.09883, pruned_loss=0.01817, audio_tagging_loss=0.009875, over 3050169.52 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:07:37,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1276360.0, ans=0.2
2023-11-21 00:07:39,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1276360.0, ans=0.0
2023-11-21 00:08:09,713 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.351e+01 9.133e+01 9.830e+01 1.252e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-21 00:08:21,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1276560.0, ans=0.1
2023-11-21 00:08:32,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191500
2023-11-21 00:08:40,572 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11150, loss[loss=0.09241, simple_loss=0.1221, pruned_loss=0.02393, audio_tagging_loss=0.007406, over 14812.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.0982, pruned_loss=0.01808, audio_tagging_loss=0.01001, over 3053178.78 frames. ], batch size: 55, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:08:40,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1276693.3333333333, ans=0.125
2023-11-21 00:08:46,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1276693.3333333333, ans=0.125
2023-11-21 00:08:52,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.59 vs. limit=10.0
2023-11-21 00:09:03,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1276760.0, ans=0.0
2023-11-21 00:09:06,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1276826.6666666667, ans=0.0
2023-11-21 00:09:09,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1276826.6666666667, ans=0.0
2023-11-21 00:09:17,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1276893.3333333333, ans=0.1
2023-11-21 00:09:21,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1276893.3333333333, ans=0.125
2023-11-21 00:09:26,211 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:09:28,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.19 vs. limit=22.5
2023-11-21 00:09:37,015 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191550
2023-11-21 00:09:38,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.03 vs. limit=15.0
2023-11-21 00:09:43,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.98 vs. limit=15.0
2023-11-21 00:09:44,265 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11200, loss[loss=0.09478, simple_loss=0.1228, pruned_loss=0.02354, audio_tagging_loss=0.009818, over 15731.00 frames. ], tot_loss[loss=0.077, simple_loss=0.09791, pruned_loss=0.01797, audio_tagging_loss=0.01007, over 3057121.29 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:10:18,320 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.318e+01 8.964e+01 9.909e+01 1.504e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 00:10:27,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1277226.6666666667, ans=0.0
2023-11-21 00:10:41,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191600
2023-11-21 00:10:48,730 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11250, loss[loss=0.08275, simple_loss=0.1056, pruned_loss=0.02231, audio_tagging_loss=0.007627, over 15979.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.0966, pruned_loss=0.01774, audio_tagging_loss=0.01015, over 3055109.77 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:10:48,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1277360.0, ans=0.0
2023-11-21 00:11:00,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1277426.6666666667, ans=0.0
2023-11-21 00:11:04,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1277426.6666666667, ans=0.125
2023-11-21 00:11:06,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1277426.6666666667, ans=10.0
2023-11-21 00:11:15,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1277493.3333333333, ans=0.09899494936611666
2023-11-21 00:11:16,891 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.09 vs. limit=22.5
2023-11-21 00:11:26,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1277560.0, ans=0.0
2023-11-21 00:11:46,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191650
2023-11-21 00:11:53,461 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11300, loss[loss=0.07698, simple_loss=0.1072, pruned_loss=0.01598, audio_tagging_loss=0.00738, over 15812.00 frames. ], tot_loss[loss=0.07608, simple_loss=0.09666, pruned_loss=0.01777, audio_tagging_loss=0.009974, over 3049412.82 frames. ], batch size: 60, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:11:53,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1277693.3333333333, ans=0.125
2023-11-21 00:11:59,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1277693.3333333333, ans=0.125
2023-11-21 00:12:19,313 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.07 vs. limit=6.0
2023-11-21 00:12:26,965 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.170e+01 7.843e+01 8.601e+01 9.135e+01 1.178e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-21 00:12:38,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.42 vs. limit=22.5
2023-11-21 00:12:50,751 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191700
2023-11-21 00:12:52,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1277960.0, ans=0.1
2023-11-21 00:12:57,886 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11350, loss[loss=0.09696, simple_loss=0.1312, pruned_loss=0.02365, audio_tagging_loss=0.007705, over 15587.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.0971, pruned_loss=0.01783, audio_tagging_loss=0.009806, over 3049018.38 frames. ], batch size: 59, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:12:59,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1278026.6666666667, ans=0.125
2023-11-21 00:13:02,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1278026.6666666667, ans=0.125
2023-11-21 00:13:15,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1278093.3333333333, ans=0.0
2023-11-21 00:13:53,021 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191750
2023-11-21 00:13:59,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1278360.0, ans=0.125
2023-11-21 00:14:00,783 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11400, loss[loss=0.1016, simple_loss=0.1249, pruned_loss=0.02902, audio_tagging_loss=0.01009, over 14846.00 frames. ], tot_loss[loss=0.07633, simple_loss=0.09739, pruned_loss=0.01789, audio_tagging_loss=0.009743, over 3040363.80 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:14:35,216 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.048e+01 8.638e+01 9.436e+01 1.129e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 00:14:50,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1278560.0, ans=0.125
2023-11-21 00:14:55,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1278626.6666666667, ans=0.125
2023-11-21 00:14:57,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191800
2023-11-21 00:14:57,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1278626.6666666667, ans=0.1
2023-11-21 00:15:00,473 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.61 vs. limit=15.0
2023-11-21 00:15:05,468 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11450, loss[loss=0.0496, simple_loss=0.06375, pruned_loss=0.008565, audio_tagging_loss=0.009163, over 16256.00 frames. ], tot_loss[loss=0.07635, simple_loss=0.09747, pruned_loss=0.01797, audio_tagging_loss=0.009643, over 3042704.64 frames. ], batch size: 63, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:15:16,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1278693.3333333333, ans=0.125
2023-11-21 00:15:22,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1278760.0, ans=0.95
2023-11-21 00:15:26,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.89 vs. limit=15.0
2023-11-21 00:15:28,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1278760.0, ans=0.125
2023-11-21 00:15:39,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1278826.6666666667, ans=0.07
2023-11-21 00:15:46,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1278893.3333333333, ans=0.125
2023-11-21 00:15:53,360 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.26 vs. limit=6.0
2023-11-21 00:15:58,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1278960.0, ans=0.1
2023-11-21 00:16:02,265 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191850
2023-11-21 00:16:09,425 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11500, loss[loss=0.08165, simple_loss=0.1038, pruned_loss=0.02101, audio_tagging_loss=0.008729, over 15054.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09753, pruned_loss=0.01795, audio_tagging_loss=0.009708, over 3037643.16 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:16:41,280 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.61 vs. limit=12.0
2023-11-21 00:16:41,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1279160.0, ans=0.2
2023-11-21 00:16:42,857 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 7.974e+01 8.630e+01 9.256e+01 1.145e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 00:16:55,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0
2023-11-21 00:17:05,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191900
2023-11-21 00:17:05,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1279293.3333333333, ans=0.0
2023-11-21 00:17:12,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1279360.0, ans=0.125
2023-11-21 00:17:13,189 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11550, loss[loss=0.08988, simple_loss=0.1178, pruned_loss=0.01966, audio_tagging_loss=0.01134, over 14983.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.09772, pruned_loss=0.01805, audio_tagging_loss=0.009727, over 3043004.61 frames. ], batch size: 53, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:17:36,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1279426.6666666667, ans=0.05
2023-11-21 00:17:40,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.60 vs. limit=15.0
2023-11-21 00:17:42,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1279493.3333333333, ans=0.0
2023-11-21 00:17:44,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1279493.3333333333, ans=0.05
2023-11-21 00:17:51,068 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:17:52,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.91 vs. limit=6.0
2023-11-21 00:17:53,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1279560.0, ans=0.0
2023-11-21 00:17:54,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1279560.0, ans=0.0
2023-11-21 00:18:00,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1279560.0, ans=0.125
2023-11-21 00:18:08,735 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 191950
2023-11-21 00:18:10,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1279626.6666666667, ans=0.0
2023-11-21 00:18:16,019 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11600, loss[loss=0.06684, simple_loss=0.0925, pruned_loss=0.01194, audio_tagging_loss=0.008644, over 16112.00 frames. ], tot_loss[loss=0.07777, simple_loss=0.09954, pruned_loss=0.01844, audio_tagging_loss=0.009557, over 3047580.08 frames. ], batch size: 61, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:18:51,116 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.722e+01 8.133e+01 8.974e+01 9.770e+01 1.687e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-21 00:18:59,042 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.71 vs. limit=22.5
2023-11-21 00:19:03,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1279893.3333333333, ans=0.125
2023-11-21 00:19:13,327 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192000
2023-11-21 00:19:24,919 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11650, loss[loss=0.09331, simple_loss=0.1249, pruned_loss=0.021, audio_tagging_loss=0.00987, over 14858.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09948, pruned_loss=0.01834, audio_tagging_loss=0.009547, over 3039338.59 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:19:25,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1280026.6666666667, ans=0.2
2023-11-21 00:19:31,266 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:19:33,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0
2023-11-21 00:19:44,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1280093.3333333333, ans=0.1
2023-11-21 00:19:47,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1280093.3333333333, ans=0.125
2023-11-21 00:19:52,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1280160.0, ans=0.2
2023-11-21 00:20:21,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192050
2023-11-21 00:20:25,241 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.60 vs. limit=15.0
2023-11-21 00:20:25,433 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.52 vs. limit=15.0
2023-11-21 00:20:28,817 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11700, loss[loss=0.08619, simple_loss=0.1109, pruned_loss=0.01982, audio_tagging_loss=0.01092, over 15926.00 frames. ], tot_loss[loss=0.07756, simple_loss=0.09927, pruned_loss=0.01827, audio_tagging_loss=0.009658, over 3044680.31 frames. ], batch size: 59, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:20:57,830 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:21:03,126 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.317e+01 8.852e+01 9.641e+01 1.228e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 00:21:05,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1280560.0, ans=0.1
2023-11-21 00:21:17,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1280560.0, ans=0.125
2023-11-21 00:21:24,875 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192100
2023-11-21 00:21:25,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1280626.6666666667, ans=0.125
2023-11-21 00:21:32,263 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11750, loss[loss=0.08491, simple_loss=0.1079, pruned_loss=0.02349, audio_tagging_loss=0.007485, over 15031.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09793, pruned_loss=0.01809, audio_tagging_loss=0.00976, over 3042294.20 frames. ], batch size: 56, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:21:42,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1280693.3333333333, ans=0.0
2023-11-21 00:21:56,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1280826.6666666667, ans=0.1
2023-11-21 00:22:09,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1280893.3333333333, ans=0.025
2023-11-21 00:22:16,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.65 vs. limit=15.0
2023-11-21 00:22:27,437 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192150
2023-11-21 00:22:35,283 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11800, loss[loss=0.09035, simple_loss=0.1198, pruned_loss=0.02265, audio_tagging_loss=0.007812, over 16300.00 frames. ], tot_loss[loss=0.07667, simple_loss=0.09754, pruned_loss=0.01805, audio_tagging_loss=0.009852, over 3044374.09 frames. ], batch size: 62, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:22:41,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1281026.6666666667, ans=0.2
2023-11-21 00:22:49,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1281093.3333333333, ans=0.0
2023-11-21 00:23:00,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1281160.0, ans=0.125
2023-11-21 00:23:09,491 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.064e+01 8.621e+01 9.481e+01 1.242e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-21 00:23:14,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1281226.6666666667, ans=0.0
2023-11-21 00:23:28,196 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.50 vs. limit=5.0
2023-11-21 00:23:31,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192200
2023-11-21 00:23:34,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1281293.3333333333, ans=0.125
2023-11-21 00:23:39,816 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11850, loss[loss=0.09299, simple_loss=0.1158, pruned_loss=0.0254, audio_tagging_loss=0.009721, over 14695.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.0981, pruned_loss=0.0181, audio_tagging_loss=0.009841, over 3050149.96 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:23:46,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1281360.0, ans=0.0
2023-11-21 00:23:51,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1281426.6666666667, ans=0.2
2023-11-21 00:23:54,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1281426.6666666667, ans=0.125
2023-11-21 00:23:56,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1281426.6666666667, ans=0.2
2023-11-21 00:23:59,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1281426.6666666667, ans=0.2
2023-11-21 00:24:13,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1281493.3333333333, ans=0.125
2023-11-21 00:24:15,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1281560.0, ans=0.07
2023-11-21 00:24:18,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1281560.0, ans=0.1
2023-11-21 00:24:35,073 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192250
2023-11-21 00:24:39,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1281626.6666666667, ans=0.0
2023-11-21 00:24:42,315 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11900, loss[loss=0.07178, simple_loss=0.09051, pruned_loss=0.01364, audio_tagging_loss=0.01289, over 16737.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09798, pruned_loss=0.01819, audio_tagging_loss=0.009952, over 3053233.77 frames. ], batch size: 60, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:25:13,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1281826.6666666667, ans=0.125
2023-11-21 00:25:15,445 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.08 vs. limit=12.0
2023-11-21 00:25:17,109 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.295e+01 9.404e+01 1.030e+02 1.674e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-21 00:25:34,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1281960.0, ans=0.2
2023-11-21 00:25:37,983 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192300
2023-11-21 00:25:45,943 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 11950, loss[loss=0.08999, simple_loss=0.1165, pruned_loss=0.02281, audio_tagging_loss=0.008947, over 14761.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09772, pruned_loss=0.01811, audio_tagging_loss=0.01011, over 3061339.19 frames. ], batch size: 57, lr: 4.23e-03, grad_scale: 32.0
2023-11-21 00:26:00,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1282093.3333333333, ans=0.125
2023-11-21 00:26:16,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1282160.0, ans=0.125
2023-11-21 00:26:25,894 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.64 vs. limit=6.0
2023-11-21 00:26:29,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.36 vs. limit=12.0
2023-11-21 00:26:31,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1282226.6666666667, ans=0.125
2023-11-21 00:26:39,417 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192350
2023-11-21 00:26:44,269 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:26:44,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1282293.3333333333, ans=0.125
2023-11-21 00:26:46,466 INFO [train_asr.py:1221] (2/4) Epoch 16, batch 12000, loss[loss=0.07651, simple_loss=0.09106, pruned_loss=0.02253, audio_tagging_loss=0.008452, over 14142.00 frames. ], tot_loss[loss=0.07718, simple_loss=0.09805, pruned_loss=0.01811, audio_tagging_loss=0.01004, over 3052834.59 frames. ], batch size: 56, lr: 4.23e-03, grad_scale: 32.0
2023-11-21 00:26:46,467 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 00:27:11,134 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8564, 3.4250, 4.8077, 4.4647], device='cuda:2')
2023-11-21 00:27:30,245 INFO [train_asr.py:1253] (2/4) Epoch 16, validation: loss=0.06114, simple_loss=0.05299, pruned_loss=0.005583, audio_tagging_loss=0.02906, over 4681554.00 frames. 
2023-11-21 00:27:30,246 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 00:27:36,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.98 vs. limit=22.5
2023-11-21 00:27:47,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1282426.6666666667, ans=0.125
2023-11-21 00:27:48,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1282426.6666666667, ans=0.125
2023-11-21 00:28:33,017 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 0, loss[loss=0.08359, simple_loss=0.08458, pruned_loss=0.01503, audio_tagging_loss=0.02628, over 15624.00 frames. ], tot_loss[loss=0.08359, simple_loss=0.08458, pruned_loss=0.01503, audio_tagging_loss=0.02628, over 15624.00 frames. ], batch size: 60, lr: 4.11e-03, grad_scale: 32.0
2023-11-21 00:28:33,018 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 00:28:55,981 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.6856, 3.7967, 4.8738, 4.0469], device='cuda:2')
2023-11-21 00:29:04,531 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8328, 4.9202, 4.9250, 4.9013], device='cuda:2')
2023-11-21 00:29:12,205 INFO [train_asr.py:1253] (2/4) Epoch 17, validation: loss=0.06074, simple_loss=0.05295, pruned_loss=0.005487, audio_tagging_loss=0.02878, over 4681554.00 frames. 
2023-11-21 00:29:12,206 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 00:29:18,854 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.043e+01 8.765e+01 9.548e+01 1.252e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 00:29:24,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1282580.0, ans=0.95
2023-11-21 00:29:25,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1282580.0, ans=0.0
2023-11-21 00:29:39,187 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192400
2023-11-21 00:29:42,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1282646.6666666667, ans=0.125
2023-11-21 00:29:46,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1282646.6666666667, ans=0.0
2023-11-21 00:29:52,263 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:30:16,443 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 50, loss[loss=0.07917, simple_loss=0.0969, pruned_loss=0.01512, audio_tagging_loss=0.0156, over 15328.00 frames. ], tot_loss[loss=0.08752, simple_loss=0.1006, pruned_loss=0.01855, audio_tagging_loss=0.01864, over 692584.04 frames. ], batch size: 56, lr: 4.11e-03, grad_scale: 32.0
2023-11-21 00:30:16,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1282846.6666666667, ans=0.0
2023-11-21 00:30:18,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.50 vs. limit=22.5
2023-11-21 00:30:29,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1282913.3333333333, ans=0.95
2023-11-21 00:30:35,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1282913.3333333333, ans=0.1
2023-11-21 00:30:42,573 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192450
2023-11-21 00:31:00,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.40 vs. limit=15.0
2023-11-21 00:31:18,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1283180.0, ans=0.0
2023-11-21 00:31:20,477 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 100, loss[loss=0.0999, simple_loss=0.1254, pruned_loss=0.02273, audio_tagging_loss=0.01448, over 15834.00 frames. ], tot_loss[loss=0.08364, simple_loss=0.09642, pruned_loss=0.01728, audio_tagging_loss=0.01815, over 1223964.67 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:31:20,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1283180.0, ans=0.125
2023-11-21 00:31:24,695 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:31:27,988 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.701e+01 8.625e+01 9.421e+01 9.913e+01 1.432e+02, threshold=1.884e+02, percent-clipped=0.0
2023-11-21 00:31:31,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.27 vs. limit=15.0
2023-11-21 00:31:47,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192500
2023-11-21 00:32:06,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1283380.0, ans=0.125
2023-11-21 00:32:24,248 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 150, loss[loss=0.07969, simple_loss=0.1036, pruned_loss=0.01964, audio_tagging_loss=0.008228, over 13926.00 frames. ], tot_loss[loss=0.08371, simple_loss=0.09937, pruned_loss=0.01799, audio_tagging_loss=0.01603, over 1631639.95 frames. ], batch size: 54, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:32:51,325 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192550
2023-11-21 00:32:53,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1283646.6666666667, ans=0.125
2023-11-21 00:32:57,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1283646.6666666667, ans=0.0
2023-11-21 00:33:02,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1283713.3333333333, ans=0.125
2023-11-21 00:33:12,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1283713.3333333333, ans=0.2
2023-11-21 00:33:16,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1283780.0, ans=0.2
2023-11-21 00:33:29,868 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 200, loss[loss=0.06397, simple_loss=0.08058, pruned_loss=0.01379, audio_tagging_loss=0.009885, over 15197.00 frames. ], tot_loss[loss=0.08211, simple_loss=0.09954, pruned_loss=0.01812, audio_tagging_loss=0.01422, over 1947430.92 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:33:30,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1283846.6666666667, ans=0.05
2023-11-21 00:33:35,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.64 vs. limit=22.5
2023-11-21 00:33:37,143 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 8.165e+01 8.890e+01 9.863e+01 2.020e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-21 00:33:37,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1283846.6666666667, ans=0.95
2023-11-21 00:33:37,840 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.49 vs. limit=15.0
2023-11-21 00:33:56,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192600
2023-11-21 00:34:13,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1284046.6666666667, ans=0.1
2023-11-21 00:34:16,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1284046.6666666667, ans=0.125
2023-11-21 00:34:21,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1284113.3333333333, ans=0.1
2023-11-21 00:34:33,591 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 250, loss[loss=0.1087, simple_loss=0.1472, pruned_loss=0.02971, audio_tagging_loss=0.005359, over 15182.00 frames. ], tot_loss[loss=0.08165, simple_loss=0.1005, pruned_loss=0.01869, audio_tagging_loss=0.0127, over 2193425.73 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:34:47,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1284246.6666666667, ans=0.125
2023-11-21 00:35:00,410 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192650
2023-11-21 00:35:26,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1284446.6666666667, ans=0.2
2023-11-21 00:35:37,464 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 300, loss[loss=0.0806, simple_loss=0.1023, pruned_loss=0.02011, audio_tagging_loss=0.009351, over 13659.00 frames. ], tot_loss[loss=0.08061, simple_loss=0.1007, pruned_loss=0.01854, audio_tagging_loss=0.01174, over 2379381.46 frames. ], batch size: 50, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:35:44,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.58 vs. limit=15.0
2023-11-21 00:35:45,417 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.137e+01 8.919e+01 9.657e+01 1.268e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 00:36:03,707 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192700
2023-11-21 00:36:17,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0
2023-11-21 00:36:30,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1284780.0, ans=0.125
2023-11-21 00:36:35,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.72 vs. limit=22.5
2023-11-21 00:36:40,666 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 350, loss[loss=0.07156, simple_loss=0.09298, pruned_loss=0.01761, audio_tagging_loss=0.007452, over 15395.00 frames. ], tot_loss[loss=0.07913, simple_loss=0.09936, pruned_loss=0.01828, audio_tagging_loss=0.01117, over 2534499.23 frames. ], batch size: 59, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:36:42,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1284846.6666666667, ans=0.2
2023-11-21 00:37:00,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.82 vs. limit=15.0
2023-11-21 00:37:05,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.97 vs. limit=10.0
2023-11-21 00:37:07,009 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192750
2023-11-21 00:37:10,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1284980.0, ans=0.0
2023-11-21 00:37:20,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1285046.6666666667, ans=0.09899494936611666
2023-11-21 00:37:28,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1285046.6666666667, ans=0.2
2023-11-21 00:37:28,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1285046.6666666667, ans=0.0
2023-11-21 00:37:44,026 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 400, loss[loss=0.08463, simple_loss=0.1139, pruned_loss=0.01973, audio_tagging_loss=0.007966, over 15679.00 frames. ], tot_loss[loss=0.07778, simple_loss=0.09827, pruned_loss=0.01778, audio_tagging_loss=0.01086, over 2646812.87 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 32.0
2023-11-21 00:37:44,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1285180.0, ans=0.0
2023-11-21 00:37:51,990 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.176e+01 8.863e+01 9.795e+01 2.108e+02, threshold=1.773e+02, percent-clipped=1.0
2023-11-21 00:37:59,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1285246.6666666667, ans=0.125
2023-11-21 00:38:11,641 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192800
2023-11-21 00:38:17,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1285313.3333333333, ans=0.125
2023-11-21 00:38:22,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1285380.0, ans=0.125
2023-11-21 00:38:26,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.52 vs. limit=15.0
2023-11-21 00:38:35,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1285446.6666666667, ans=0.1
2023-11-21 00:38:42,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1285446.6666666667, ans=0.125
2023-11-21 00:38:45,874 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.45 vs. limit=15.0
2023-11-21 00:38:47,679 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 450, loss[loss=0.09024, simple_loss=0.1053, pruned_loss=0.02491, audio_tagging_loss=0.01268, over 15630.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09742, pruned_loss=0.01782, audio_tagging_loss=0.01068, over 2740089.59 frames. ], batch size: 58, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:39:06,864 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.47 vs. limit=12.0
2023-11-21 00:39:14,745 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192850
2023-11-21 00:39:29,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.82 vs. limit=15.0
2023-11-21 00:39:35,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1285713.3333333333, ans=0.1
2023-11-21 00:39:51,822 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 500, loss[loss=0.06979, simple_loss=0.08248, pruned_loss=0.01813, audio_tagging_loss=0.01043, over 15280.00 frames. ], tot_loss[loss=0.07753, simple_loss=0.09818, pruned_loss=0.01802, audio_tagging_loss=0.01042, over 2805409.71 frames. ], batch size: 59, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:40:00,411 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 7.999e+01 8.766e+01 9.500e+01 1.232e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 00:40:04,876 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.53 vs. limit=6.0
2023-11-21 00:40:18,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192900
2023-11-21 00:40:28,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1286046.6666666667, ans=0.1
2023-11-21 00:40:41,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1286113.3333333333, ans=0.125
2023-11-21 00:40:55,088 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 550, loss[loss=0.05864, simple_loss=0.07585, pruned_loss=0.008959, audio_tagging_loss=0.01175, over 14793.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.09809, pruned_loss=0.01809, audio_tagging_loss=0.01033, over 2853040.37 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:40:58,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1286180.0, ans=0.015
2023-11-21 00:41:01,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1286180.0, ans=0.2
2023-11-21 00:41:20,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1286313.3333333333, ans=0.125
2023-11-21 00:41:22,232 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 192950
2023-11-21 00:41:35,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1286380.0, ans=0.125
2023-11-21 00:41:49,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1286446.6666666667, ans=0.2
2023-11-21 00:41:50,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1286446.6666666667, ans=0.125
2023-11-21 00:41:50,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1286446.6666666667, ans=0.125
2023-11-21 00:41:58,726 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 600, loss[loss=0.07803, simple_loss=0.09645, pruned_loss=0.0191, audio_tagging_loss=0.01071, over 14503.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09799, pruned_loss=0.01803, audio_tagging_loss=0.01023, over 2891007.33 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:42:05,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1286513.3333333333, ans=0.0
2023-11-21 00:42:07,182 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.459e+01 7.723e+01 8.573e+01 9.272e+01 1.267e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 00:42:24,983 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193000
2023-11-21 00:42:34,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1286646.6666666667, ans=0.1
2023-11-21 00:42:38,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1286713.3333333333, ans=0.125
2023-11-21 00:42:50,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1286780.0, ans=0.125
2023-11-21 00:43:02,803 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 650, loss[loss=0.05225, simple_loss=0.05903, pruned_loss=0.0123, audio_tagging_loss=0.01043, over 15089.00 frames. ], tot_loss[loss=0.077, simple_loss=0.09787, pruned_loss=0.0179, audio_tagging_loss=0.01017, over 2921473.55 frames. ], batch size: 58, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:43:08,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.56 vs. limit=12.0
2023-11-21 00:43:14,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1286913.3333333333, ans=0.1
2023-11-21 00:43:16,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.11 vs. limit=15.0
2023-11-21 00:43:17,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1286913.3333333333, ans=0.0
2023-11-21 00:43:29,232 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193050
2023-11-21 00:43:41,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1287046.6666666667, ans=0.125
2023-11-21 00:43:52,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.70 vs. limit=10.0
2023-11-21 00:44:00,082 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:44:05,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1287180.0, ans=0.1
2023-11-21 00:44:05,984 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 700, loss[loss=0.0579, simple_loss=0.07023, pruned_loss=0.008626, audio_tagging_loss=0.01416, over 15481.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09873, pruned_loss=0.01804, audio_tagging_loss=0.0101, over 2954903.56 frames. ], batch size: 60, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:44:10,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1287180.0, ans=0.125
2023-11-21 00:44:14,962 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.94 vs. limit=15.0
2023-11-21 00:44:15,341 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 7.999e+01 8.466e+01 9.113e+01 1.191e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-21 00:44:28,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1287246.6666666667, ans=0.0
2023-11-21 00:44:32,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193100
2023-11-21 00:44:41,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1287313.3333333333, ans=0.035
2023-11-21 00:45:09,617 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.79 vs. limit=22.5
2023-11-21 00:45:10,272 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 750, loss[loss=0.07238, simple_loss=0.09493, pruned_loss=0.01806, audio_tagging_loss=0.006853, over 15183.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09907, pruned_loss=0.01831, audio_tagging_loss=0.01004, over 2979995.32 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:45:12,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1287513.3333333333, ans=15.0
2023-11-21 00:45:37,828 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193150
2023-11-21 00:45:58,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1287713.3333333333, ans=0.125
2023-11-21 00:46:14,610 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 800, loss[loss=0.05032, simple_loss=0.06326, pruned_loss=0.0101, audio_tagging_loss=0.008592, over 14462.00 frames. ], tot_loss[loss=0.07826, simple_loss=0.09948, pruned_loss=0.01847, audio_tagging_loss=0.01005, over 2993723.30 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 32.0
2023-11-21 00:46:24,327 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.140e+01 8.941e+01 9.385e+01 1.183e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 00:46:27,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1287913.3333333333, ans=0.125
2023-11-21 00:46:32,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1287913.3333333333, ans=0.125
2023-11-21 00:46:34,505 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:46:38,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1287913.3333333333, ans=0.125
2023-11-21 00:46:42,138 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193200
2023-11-21 00:46:46,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1287980.0, ans=0.125
2023-11-21 00:46:48,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1287980.0, ans=0.125
2023-11-21 00:47:19,547 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 850, loss[loss=0.06544, simple_loss=0.08549, pruned_loss=0.01129, audio_tagging_loss=0.0114, over 15386.00 frames. ], tot_loss[loss=0.0772, simple_loss=0.09804, pruned_loss=0.01806, audio_tagging_loss=0.01012, over 3001765.06 frames. ], batch size: 60, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:47:35,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1288246.6666666667, ans=0.0
2023-11-21 00:47:45,977 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193250
2023-11-21 00:47:57,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1288380.0, ans=0.125
2023-11-21 00:47:58,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288380.0, ans=0.1
2023-11-21 00:48:05,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1288380.0, ans=0.125
2023-11-21 00:48:12,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1288446.6666666667, ans=0.125
2023-11-21 00:48:23,673 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 900, loss[loss=0.07486, simple_loss=0.105, pruned_loss=0.01564, audio_tagging_loss=0.006693, over 14892.00 frames. ], tot_loss[loss=0.0768, simple_loss=0.09749, pruned_loss=0.01788, audio_tagging_loss=0.01017, over 3013501.17 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:48:33,407 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.920e+01 8.040e+01 8.568e+01 9.329e+01 1.303e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 00:48:46,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288580.0, ans=0.1
2023-11-21 00:48:50,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193300
2023-11-21 00:48:55,538 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=12.0
2023-11-21 00:48:57,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288646.6666666667, ans=0.1
2023-11-21 00:49:00,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1288646.6666666667, ans=0.125
2023-11-21 00:49:13,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1288713.3333333333, ans=0.2
2023-11-21 00:49:27,453 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 950, loss[loss=0.09069, simple_loss=0.1212, pruned_loss=0.02286, audio_tagging_loss=0.007206, over 15168.00 frames. ], tot_loss[loss=0.07649, simple_loss=0.09714, pruned_loss=0.01783, audio_tagging_loss=0.01009, over 3014281.83 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:49:39,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1288913.3333333333, ans=0.125
2023-11-21 00:49:52,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1288980.0, ans=0.125
2023-11-21 00:49:55,850 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193350
2023-11-21 00:49:59,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1288980.0, ans=0.1
2023-11-21 00:50:03,995 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.33 vs. limit=6.0
2023-11-21 00:50:04,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1288980.0, ans=0.2
2023-11-21 00:50:04,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1288980.0, ans=0.0
2023-11-21 00:50:16,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1289046.6666666667, ans=0.125
2023-11-21 00:50:29,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1289113.3333333333, ans=0.0
2023-11-21 00:50:32,433 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1000, loss[loss=0.0802, simple_loss=0.1044, pruned_loss=0.01758, audio_tagging_loss=0.01042, over 16605.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09789, pruned_loss=0.01789, audio_tagging_loss=0.009857, over 3028917.92 frames. ], batch size: 64, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:50:42,902 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.002e+01 8.372e+01 8.934e+01 9.683e+01 2.784e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-21 00:50:51,394 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:50:56,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1289246.6666666667, ans=0.1
2023-11-21 00:50:58,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1289313.3333333333, ans=0.125
2023-11-21 00:50:58,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1289313.3333333333, ans=0.125
2023-11-21 00:50:59,520 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:50:59,595 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193400
2023-11-21 00:51:03,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1289313.3333333333, ans=0.0
2023-11-21 00:51:24,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1289446.6666666667, ans=0.2
2023-11-21 00:51:31,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1289446.6666666667, ans=0.125
2023-11-21 00:51:38,124 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1050, loss[loss=0.08928, simple_loss=0.1161, pruned_loss=0.0236, audio_tagging_loss=0.007609, over 15037.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.09752, pruned_loss=0.01786, audio_tagging_loss=0.009707, over 3032664.32 frames. ], batch size: 55, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 00:51:54,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1289580.0, ans=0.125
2023-11-21 00:52:05,436 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193450
2023-11-21 00:52:35,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1289780.0, ans=0.1
2023-11-21 00:52:37,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1289780.0, ans=0.125
2023-11-21 00:52:40,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1289846.6666666667, ans=0.0
2023-11-21 00:52:41,786 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1100, loss[loss=0.07654, simple_loss=0.1033, pruned_loss=0.01531, audio_tagging_loss=0.009569, over 15027.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.09801, pruned_loss=0.01797, audio_tagging_loss=0.00965, over 3035856.89 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 00:52:44,391 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:52:47,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1289846.6666666667, ans=0.125
2023-11-21 00:52:52,953 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.642e+01 8.168e+01 8.741e+01 9.235e+01 1.137e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 00:52:55,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1289913.3333333333, ans=0.125
2023-11-21 00:53:09,432 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193500
2023-11-21 00:53:46,391 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1150, loss[loss=0.05849, simple_loss=0.07178, pruned_loss=0.01145, audio_tagging_loss=0.01115, over 16532.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09737, pruned_loss=0.0176, audio_tagging_loss=0.009632, over 3036511.66 frames. ], batch size: 64, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 00:53:52,009 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.00 vs. limit=15.0
2023-11-21 00:54:09,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1290246.6666666667, ans=10.0
2023-11-21 00:54:13,287 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193550
2023-11-21 00:54:23,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1290380.0, ans=0.125
2023-11-21 00:54:26,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=1290380.0, ans=0.02
2023-11-21 00:54:26,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1290380.0, ans=0.125
2023-11-21 00:54:35,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.35 vs. limit=6.0
2023-11-21 00:54:40,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.83 vs. limit=15.0
2023-11-21 00:54:51,220 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1200, loss[loss=0.07691, simple_loss=0.09634, pruned_loss=0.0212, audio_tagging_loss=0.007542, over 15848.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09688, pruned_loss=0.01759, audio_tagging_loss=0.009683, over 3038482.75 frames. ], batch size: 58, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:55:00,969 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 7.968e+01 8.692e+01 9.308e+01 1.785e+02, threshold=1.738e+02, percent-clipped=1.0
2023-11-21 00:55:02,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=12.0
2023-11-21 00:55:18,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193600
2023-11-21 00:55:55,001 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1250, loss[loss=0.06133, simple_loss=0.07268, pruned_loss=0.01493, audio_tagging_loss=0.01005, over 14899.00 frames. ], tot_loss[loss=0.07622, simple_loss=0.09749, pruned_loss=0.01777, audio_tagging_loss=0.00971, over 3042625.04 frames. ], batch size: 55, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:56:10,456 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-21 00:56:12,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1290913.3333333333, ans=0.1
2023-11-21 00:56:22,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193650
2023-11-21 00:56:59,679 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1300, loss[loss=0.06479, simple_loss=0.08363, pruned_loss=0.01237, audio_tagging_loss=0.01061, over 15671.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09734, pruned_loss=0.01778, audio_tagging_loss=0.009699, over 3046521.33 frames. ], batch size: 57, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:57:06,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1291180.0, ans=0.0
2023-11-21 00:57:09,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.443e+01 7.829e+01 8.638e+01 9.326e+01 1.163e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 00:57:12,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1291246.6666666667, ans=0.125
2023-11-21 00:57:17,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1291246.6666666667, ans=0.125
2023-11-21 00:57:22,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1291246.6666666667, ans=0.125
2023-11-21 00:57:26,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193700
2023-11-21 00:58:03,402 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1350, loss[loss=0.09419, simple_loss=0.1247, pruned_loss=0.02361, audio_tagging_loss=0.008245, over 15790.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09782, pruned_loss=0.01797, audio_tagging_loss=0.009626, over 3051570.84 frames. ], batch size: 58, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:58:03,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1291513.3333333333, ans=0.1
2023-11-21 00:58:08,484 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:58:22,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1291580.0, ans=0.125
2023-11-21 00:58:29,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.27 vs. limit=22.5
2023-11-21 00:58:30,938 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193750
2023-11-21 00:58:49,289 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:59:07,692 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1400, loss[loss=0.05814, simple_loss=0.06716, pruned_loss=0.01402, audio_tagging_loss=0.01054, over 14728.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09703, pruned_loss=0.01786, audio_tagging_loss=0.009758, over 3057889.56 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:59:08,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1291846.6666666667, ans=0.0
2023-11-21 00:59:11,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1291846.6666666667, ans=0.1
2023-11-21 00:59:18,071 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.121e+01 7.749e+01 8.445e+01 9.345e+01 1.750e+02, threshold=1.689e+02, percent-clipped=1.0
2023-11-21 00:59:31,335 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.89 vs. limit=15.0
2023-11-21 00:59:34,630 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193800
2023-11-21 00:59:49,493 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.79 vs. limit=15.0
2023-11-21 01:00:12,429 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1450, loss[loss=0.07379, simple_loss=0.09206, pruned_loss=0.01487, audio_tagging_loss=0.01289, over 14946.00 frames. ], tot_loss[loss=0.07594, simple_loss=0.097, pruned_loss=0.01767, audio_tagging_loss=0.009772, over 3057357.26 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:00:21,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1292180.0, ans=0.1
2023-11-21 01:00:23,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1292246.6666666667, ans=0.1
2023-11-21 01:00:38,750 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193850
2023-11-21 01:00:56,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1292380.0, ans=15.0
2023-11-21 01:01:16,047 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1500, loss[loss=0.09251, simple_loss=0.1146, pruned_loss=0.02517, audio_tagging_loss=0.01003, over 15227.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09706, pruned_loss=0.01769, audio_tagging_loss=0.009824, over 3057647.21 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:01:19,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1292513.3333333333, ans=0.125
2023-11-21 01:01:19,378 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:01:23,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1292513.3333333333, ans=0.1
2023-11-21 01:01:27,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.416e+01 9.067e+01 9.838e+01 1.487e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-21 01:01:42,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193900
2023-11-21 01:02:15,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1292780.0, ans=0.0
2023-11-21 01:02:15,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1292780.0, ans=0.125
2023-11-21 01:02:20,429 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1550, loss[loss=0.08556, simple_loss=0.1132, pruned_loss=0.02153, audio_tagging_loss=0.007409, over 15596.00 frames. ], tot_loss[loss=0.07696, simple_loss=0.09816, pruned_loss=0.01796, audio_tagging_loss=0.009921, over 3059361.92 frames. ], batch size: 55, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:02:32,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1292913.3333333333, ans=0.2
2023-11-21 01:02:34,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1292913.3333333333, ans=0.125
2023-11-21 01:02:47,736 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 193950
2023-11-21 01:03:03,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1293046.6666666667, ans=0.125
2023-11-21 01:03:05,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1293046.6666666667, ans=0.0
2023-11-21 01:03:06,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1293046.6666666667, ans=0.1
2023-11-21 01:03:09,293 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:03:25,729 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1600, loss[loss=0.05611, simple_loss=0.07134, pruned_loss=0.009934, audio_tagging_loss=0.0105, over 14809.00 frames. ], tot_loss[loss=0.0772, simple_loss=0.09842, pruned_loss=0.01797, audio_tagging_loss=0.01001, over 3057349.30 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:03:26,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.14 vs. limit=15.0
2023-11-21 01:03:36,993 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.454e+01 8.126e+01 8.877e+01 9.641e+01 1.501e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 01:03:37,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1293246.6666666667, ans=0.1
2023-11-21 01:03:38,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1293246.6666666667, ans=0.125
2023-11-21 01:03:50,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1293313.3333333333, ans=0.1
2023-11-21 01:03:52,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194000
2023-11-21 01:03:59,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1293313.3333333333, ans=0.2
2023-11-21 01:04:28,723 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.18 vs. limit=15.0
2023-11-21 01:04:30,484 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1650, loss[loss=0.06016, simple_loss=0.0635, pruned_loss=0.01407, audio_tagging_loss=0.01433, over 15264.00 frames. ], tot_loss[loss=0.07741, simple_loss=0.09889, pruned_loss=0.01803, audio_tagging_loss=0.009936, over 3055906.33 frames. ], batch size: 59, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:04:37,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1293513.3333333333, ans=0.125
2023-11-21 01:04:38,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1293513.3333333333, ans=0.125
2023-11-21 01:04:44,122 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0
2023-11-21 01:04:44,750 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:04:47,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1293580.0, ans=0.0
2023-11-21 01:04:56,795 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194050
2023-11-21 01:05:07,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1293646.6666666667, ans=0.0
2023-11-21 01:05:35,457 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1700, loss[loss=0.07377, simple_loss=0.09767, pruned_loss=0.0152, audio_tagging_loss=0.009733, over 15833.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.09949, pruned_loss=0.01816, audio_tagging_loss=0.009901, over 3054001.15 frames. ], batch size: 58, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:05:37,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1293846.6666666667, ans=0.125
2023-11-21 01:05:41,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1293846.6666666667, ans=0.2
2023-11-21 01:05:44,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1293846.6666666667, ans=0.1
2023-11-21 01:05:45,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1293846.6666666667, ans=0.125
2023-11-21 01:05:46,330 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.167e+01 8.614e+01 9.209e+01 1.216e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 01:06:02,886 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194100
2023-11-21 01:06:32,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1294113.3333333333, ans=0.125
2023-11-21 01:06:40,574 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1750, loss[loss=0.0849, simple_loss=0.09797, pruned_loss=0.02511, audio_tagging_loss=0.0108, over 16366.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09876, pruned_loss=0.01793, audio_tagging_loss=0.009861, over 3057825.16 frames. ], batch size: 63, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:06:42,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1294180.0, ans=0.125
2023-11-21 01:06:48,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1294180.0, ans=0.125
2023-11-21 01:07:02,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2.whitening_limit, batch_count=1294246.6666666667, ans=15.0
2023-11-21 01:07:07,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194150
2023-11-21 01:07:14,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1294313.3333333333, ans=0.0
2023-11-21 01:07:29,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.74 vs. limit=12.0
2023-11-21 01:07:35,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1294446.6666666667, ans=0.0
2023-11-21 01:07:41,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1294446.6666666667, ans=0.0
2023-11-21 01:07:43,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1294513.3333333333, ans=0.125
2023-11-21 01:07:44,352 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1800, loss[loss=0.08316, simple_loss=0.1107, pruned_loss=0.02189, audio_tagging_loss=0.005926, over 14941.00 frames. ], tot_loss[loss=0.07734, simple_loss=0.09911, pruned_loss=0.01802, audio_tagging_loss=0.009766, over 3057087.56 frames. ], batch size: 55, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:07:49,459 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:07:55,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1294513.3333333333, ans=0.0
2023-11-21 01:07:55,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1294513.3333333333, ans=0.125
2023-11-21 01:07:55,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1294513.3333333333, ans=0.125
2023-11-21 01:07:57,247 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.340e+01 8.982e+01 9.918e+01 1.410e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 01:08:10,790 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194200
2023-11-21 01:08:27,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.42 vs. limit=22.5
2023-11-21 01:08:48,991 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1850, loss[loss=0.08091, simple_loss=0.1021, pruned_loss=0.02041, audio_tagging_loss=0.009418, over 16400.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09909, pruned_loss=0.01785, audio_tagging_loss=0.009734, over 3058608.34 frames. ], batch size: 61, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:09:15,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1294980.0, ans=0.0
2023-11-21 01:09:16,890 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194250
2023-11-21 01:09:20,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1294980.0, ans=0.0
2023-11-21 01:09:23,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.92 vs. limit=15.0
2023-11-21 01:09:31,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1295046.6666666667, ans=0.1
2023-11-21 01:09:50,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1295113.3333333333, ans=0.125
2023-11-21 01:09:52,803 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1900, loss[loss=0.08207, simple_loss=0.1073, pruned_loss=0.02069, audio_tagging_loss=0.007742, over 16155.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.09944, pruned_loss=0.01791, audio_tagging_loss=0.009597, over 3058777.36 frames. ], batch size: 60, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:09:56,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1295180.0, ans=0.125
2023-11-21 01:10:02,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1295180.0, ans=0.2
2023-11-21 01:10:06,890 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.112e+01 7.944e+01 8.677e+01 9.299e+01 1.429e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 01:10:21,037 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194300
2023-11-21 01:10:21,191 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:10:21,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.43 vs. limit=22.5
2023-11-21 01:10:37,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1295380.0, ans=0.125
2023-11-21 01:10:54,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1295446.6666666667, ans=0.0
2023-11-21 01:10:58,317 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 1950, loss[loss=0.0809, simple_loss=0.1076, pruned_loss=0.01798, audio_tagging_loss=0.009127, over 14755.00 frames. ], tot_loss[loss=0.07687, simple_loss=0.09882, pruned_loss=0.01773, audio_tagging_loss=0.00973, over 3057379.51 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:11:02,630 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.27 vs. limit=22.5
2023-11-21 01:11:11,928 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.79 vs. limit=22.5
2023-11-21 01:11:24,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194350
2023-11-21 01:11:24,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1295646.6666666667, ans=0.1
2023-11-21 01:11:36,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1295713.3333333333, ans=0.125
2023-11-21 01:11:37,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1295713.3333333333, ans=0.125
2023-11-21 01:12:02,468 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2000, loss[loss=0.07852, simple_loss=0.1041, pruned_loss=0.01783, audio_tagging_loss=0.008629, over 14026.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09932, pruned_loss=0.01805, audio_tagging_loss=0.009676, over 3048622.75 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:12:08,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1295846.6666666667, ans=0.0
2023-11-21 01:12:14,647 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.274e+01 9.033e+01 9.827e+01 1.298e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 01:12:21,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1295913.3333333333, ans=0.125
2023-11-21 01:12:25,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1295913.3333333333, ans=0.05
2023-11-21 01:12:29,391 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194400
2023-11-21 01:12:45,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.94 vs. limit=15.0
2023-11-21 01:12:53,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1296113.3333333333, ans=0.125
2023-11-21 01:12:56,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1296113.3333333333, ans=0.0
2023-11-21 01:13:06,218 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2050, loss[loss=0.07412, simple_loss=0.09369, pruned_loss=0.01686, audio_tagging_loss=0.01042, over 17052.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09848, pruned_loss=0.01786, audio_tagging_loss=0.009608, over 3043847.63 frames. ], batch size: 65, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:13:14,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1296180.0, ans=0.0
2023-11-21 01:13:33,598 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194450
2023-11-21 01:13:45,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1296380.0, ans=0.125
2023-11-21 01:13:47,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1296380.0, ans=0.125
2023-11-21 01:13:49,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1296380.0, ans=0.1
2023-11-21 01:13:51,011 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.87 vs. limit=15.0
2023-11-21 01:14:06,794 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.01 vs. limit=6.0
2023-11-21 01:14:08,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1296446.6666666667, ans=0.125
2023-11-21 01:14:10,840 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2100, loss[loss=0.07711, simple_loss=0.09389, pruned_loss=0.01779, audio_tagging_loss=0.01237, over 14676.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.09749, pruned_loss=0.01778, audio_tagging_loss=0.009605, over 3042143.97 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:14:12,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1296513.3333333333, ans=0.125
2023-11-21 01:14:24,179 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.053e+01 8.707e+01 9.415e+01 1.178e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-21 01:14:24,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1296580.0, ans=0.0
2023-11-21 01:14:37,790 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194500
2023-11-21 01:14:48,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1296713.3333333333, ans=0.0
2023-11-21 01:15:05,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.03 vs. limit=22.5
2023-11-21 01:15:11,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1296780.0, ans=0.0
2023-11-21 01:15:15,749 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2150, loss[loss=0.1036, simple_loss=0.1308, pruned_loss=0.0266, audio_tagging_loss=0.01159, over 16151.00 frames. ], tot_loss[loss=0.07622, simple_loss=0.09772, pruned_loss=0.01774, audio_tagging_loss=0.009618, over 3050624.46 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:15:17,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1296846.6666666667, ans=0.1
2023-11-21 01:15:30,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1296913.3333333333, ans=0.0
2023-11-21 01:15:42,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194550
2023-11-21 01:15:52,152 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:15:59,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=15.0
2023-11-21 01:16:18,810 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2200, loss[loss=0.04237, simple_loss=0.04559, pruned_loss=0.006198, audio_tagging_loss=0.01338, over 15368.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09775, pruned_loss=0.01761, audio_tagging_loss=0.009645, over 3051694.12 frames. ], batch size: 60, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:16:32,228 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.162e+01 8.764e+01 9.413e+01 1.161e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 01:16:45,755 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194600
2023-11-21 01:16:47,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1297313.3333333333, ans=0.125
2023-11-21 01:17:06,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1297380.0, ans=0.1
2023-11-21 01:17:16,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1297446.6666666667, ans=0.125
2023-11-21 01:17:23,470 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2250, loss[loss=0.062, simple_loss=0.07913, pruned_loss=0.01377, audio_tagging_loss=0.00867, over 15387.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09729, pruned_loss=0.01751, audio_tagging_loss=0.009565, over 3042680.87 frames. ], batch size: 60, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:17:42,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1297580.0, ans=0.0
2023-11-21 01:17:50,716 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194650
2023-11-21 01:17:59,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1297646.6666666667, ans=0.2
2023-11-21 01:18:27,466 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2300, loss[loss=0.05584, simple_loss=0.06688, pruned_loss=0.01023, audio_tagging_loss=0.01218, over 14214.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.09697, pruned_loss=0.0174, audio_tagging_loss=0.00966, over 3042022.56 frames. ], batch size: 54, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:18:38,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1297846.6666666667, ans=0.2
2023-11-21 01:18:38,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1297846.6666666667, ans=0.125
2023-11-21 01:18:38,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1297846.6666666667, ans=0.1
2023-11-21 01:18:40,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.91 vs. limit=15.0
2023-11-21 01:18:41,611 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.557e+01 8.363e+01 8.884e+01 1.011e+02 1.300e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 01:18:46,002 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0
2023-11-21 01:18:55,310 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194700
2023-11-21 01:18:56,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1297980.0, ans=0.125
2023-11-21 01:18:59,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1297980.0, ans=0.125
2023-11-21 01:19:23,577 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:19:23,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1298113.3333333333, ans=0.1
2023-11-21 01:19:26,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1298113.3333333333, ans=10.0
2023-11-21 01:19:27,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1298113.3333333333, ans=0.1
2023-11-21 01:19:32,003 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2350, loss[loss=0.05825, simple_loss=0.074, pruned_loss=0.01068, audio_tagging_loss=0.01058, over 15044.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.09637, pruned_loss=0.01723, audio_tagging_loss=0.009834, over 3041593.83 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:19:48,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1298246.6666666667, ans=0.2
2023-11-21 01:19:52,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1298246.6666666667, ans=0.125
2023-11-21 01:19:59,051 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194750
2023-11-21 01:20:15,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_na.min_abs, batch_count=1298380.0, ans=0.02
2023-11-21 01:20:15,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1298380.0, ans=0.125
2023-11-21 01:20:17,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1298380.0, ans=0.125
2023-11-21 01:20:19,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.41 vs. limit=6.0
2023-11-21 01:20:36,884 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2400, loss[loss=0.08213, simple_loss=0.1045, pruned_loss=0.01796, audio_tagging_loss=0.01191, over 15091.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.0969, pruned_loss=0.0175, audio_tagging_loss=0.009883, over 3035284.51 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:20:50,276 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.420e+01 8.164e+01 8.845e+01 9.386e+01 1.132e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 01:21:03,840 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194800
2023-11-21 01:21:27,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1298780.0, ans=0.04949747468305833
2023-11-21 01:21:36,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1298780.0, ans=0.0
2023-11-21 01:21:40,671 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2450, loss[loss=0.06936, simple_loss=0.08353, pruned_loss=0.01567, audio_tagging_loss=0.01193, over 14692.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09632, pruned_loss=0.01741, audio_tagging_loss=0.01007, over 3032074.34 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:22:08,385 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194850
2023-11-21 01:22:11,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1298980.0, ans=0.2
2023-11-21 01:22:22,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1299046.6666666667, ans=0.1
2023-11-21 01:22:36,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1299113.3333333333, ans=10.0
2023-11-21 01:22:44,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1299180.0, ans=0.125
2023-11-21 01:22:45,483 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2500, loss[loss=0.07651, simple_loss=0.0952, pruned_loss=0.0205, audio_tagging_loss=0.008408, over 15906.00 frames. ], tot_loss[loss=0.07636, simple_loss=0.09732, pruned_loss=0.01775, audio_tagging_loss=0.009949, over 3044287.87 frames. ], batch size: 61, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:22:50,793 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:22:54,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1299180.0, ans=0.1
2023-11-21 01:23:01,471 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.181e+01 8.676e+01 9.218e+01 1.349e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 01:23:10,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1299313.3333333333, ans=0.1
2023-11-21 01:23:12,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194900
2023-11-21 01:23:42,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1299446.6666666667, ans=0.125
2023-11-21 01:23:50,124 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2550, loss[loss=0.09956, simple_loss=0.1223, pruned_loss=0.02938, audio_tagging_loss=0.009049, over 15468.00 frames. ], tot_loss[loss=0.07639, simple_loss=0.09714, pruned_loss=0.01796, audio_tagging_loss=0.009855, over 3039549.62 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:23:55,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1299513.3333333333, ans=0.2
2023-11-21 01:24:09,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1299580.0, ans=0.125
2023-11-21 01:24:16,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 194950
2023-11-21 01:24:24,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1299646.6666666667, ans=0.125
2023-11-21 01:24:27,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1299713.3333333333, ans=0.1
2023-11-21 01:24:50,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1299780.0, ans=0.04949747468305833
2023-11-21 01:24:51,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1299780.0, ans=0.2
2023-11-21 01:24:53,771 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2600, loss[loss=0.08784, simple_loss=0.1232, pruned_loss=0.01804, audio_tagging_loss=0.008172, over 14905.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09769, pruned_loss=0.01804, audio_tagging_loss=0.009715, over 3038348.21 frames. ], batch size: 58, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:24:54,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.71 vs. limit=22.5
2023-11-21 01:24:58,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1299846.6666666667, ans=0.125
2023-11-21 01:25:09,165 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.407e+01 7.907e+01 8.660e+01 9.294e+01 1.206e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 01:25:12,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1299913.3333333333, ans=0.125
2023-11-21 01:25:20,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195000
2023-11-21 01:25:34,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1300046.6666666667, ans=0.125
2023-11-21 01:25:39,433 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.05 vs. limit=12.0
2023-11-21 01:25:48,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1300113.3333333333, ans=0.125
2023-11-21 01:25:58,417 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2650, loss[loss=0.08223, simple_loss=0.1051, pruned_loss=0.01931, audio_tagging_loss=0.01039, over 15606.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09758, pruned_loss=0.01786, audio_tagging_loss=0.009618, over 3038125.06 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:26:13,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1300246.6666666667, ans=0.125
2023-11-21 01:26:18,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1300246.6666666667, ans=0.0
2023-11-21 01:26:26,009 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195050
2023-11-21 01:26:27,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1300313.3333333333, ans=0.125
2023-11-21 01:26:28,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1300313.3333333333, ans=0.125
2023-11-21 01:26:31,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1300313.3333333333, ans=0.125
2023-11-21 01:26:36,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1300380.0, ans=0.05
2023-11-21 01:26:38,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1300380.0, ans=15.0
2023-11-21 01:26:48,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1300446.6666666667, ans=0.2
2023-11-21 01:26:51,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1300446.6666666667, ans=0.125
2023-11-21 01:27:03,574 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2700, loss[loss=0.06664, simple_loss=0.08021, pruned_loss=0.01398, audio_tagging_loss=0.01255, over 14784.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.09579, pruned_loss=0.0174, audio_tagging_loss=0.009702, over 3040539.45 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:27:10,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1300513.3333333333, ans=0.015
2023-11-21 01:27:18,977 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 7.841e+01 8.793e+01 9.412e+01 1.255e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 01:27:24,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1300580.0, ans=0.1
2023-11-21 01:27:30,624 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195100
2023-11-21 01:27:35,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1300646.6666666667, ans=0.125
2023-11-21 01:28:08,080 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2750, loss[loss=0.06957, simple_loss=0.07586, pruned_loss=0.02009, audio_tagging_loss=0.01155, over 14733.00 frames. ], tot_loss[loss=0.07468, simple_loss=0.09525, pruned_loss=0.01725, audio_tagging_loss=0.009808, over 3043966.85 frames. ], batch size: 58, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:28:12,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1300846.6666666667, ans=0.125
2023-11-21 01:28:15,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1300846.6666666667, ans=0.0
2023-11-21 01:28:18,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1300846.6666666667, ans=0.05
2023-11-21 01:28:34,574 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195150
2023-11-21 01:29:02,053 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:29:03,102 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:29:03,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1301113.3333333333, ans=0.0
2023-11-21 01:29:11,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1301180.0, ans=0.1
2023-11-21 01:29:12,465 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2800, loss[loss=0.06897, simple_loss=0.09494, pruned_loss=0.01391, audio_tagging_loss=0.007595, over 15753.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09627, pruned_loss=0.01731, audio_tagging_loss=0.009709, over 3052810.19 frames. ], batch size: 61, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:29:16,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=12.0
2023-11-21 01:29:18,170 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.81 vs. limit=15.0
2023-11-21 01:29:27,663 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 8.111e+01 8.659e+01 9.368e+01 1.203e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 01:29:32,292 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.48 vs. limit=15.0
2023-11-21 01:29:36,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1301246.6666666667, ans=0.125
2023-11-21 01:29:39,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195200
2023-11-21 01:29:49,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1301380.0, ans=0.125
2023-11-21 01:29:49,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1301380.0, ans=0.125
2023-11-21 01:30:00,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.26 vs. limit=22.5
2023-11-21 01:30:02,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1301446.6666666667, ans=0.1
2023-11-21 01:30:16,070 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2850, loss[loss=0.07813, simple_loss=0.107, pruned_loss=0.01545, audio_tagging_loss=0.009198, over 16456.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09628, pruned_loss=0.01742, audio_tagging_loss=0.009645, over 3050150.14 frames. ], batch size: 63, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:30:24,037 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.63 vs. limit=22.5
2023-11-21 01:30:43,716 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195250
2023-11-21 01:30:44,161 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.37 vs. limit=15.0
2023-11-21 01:31:14,575 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=12.0
2023-11-21 01:31:15,678 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0
2023-11-21 01:31:16,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1301780.0, ans=0.125
2023-11-21 01:31:21,297 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2900, loss[loss=0.0695, simple_loss=0.09345, pruned_loss=0.01134, audio_tagging_loss=0.01144, over 15188.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.0961, pruned_loss=0.01736, audio_tagging_loss=0.009661, over 3044390.09 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:31:27,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1301846.6666666667, ans=0.1
2023-11-21 01:31:30,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.18 vs. limit=15.0
2023-11-21 01:31:32,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1301846.6666666667, ans=0.125
2023-11-21 01:31:34,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1301913.3333333333, ans=0.125
2023-11-21 01:31:36,825 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.167e+01 8.153e+01 9.015e+01 9.723e+01 1.263e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-21 01:31:38,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.04 vs. limit=15.0
2023-11-21 01:31:48,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195300
2023-11-21 01:31:52,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1301980.0, ans=0.1
2023-11-21 01:31:55,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1301980.0, ans=0.125
2023-11-21 01:32:03,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1302046.6666666667, ans=0.125
2023-11-21 01:32:20,683 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.97 vs. limit=22.5
2023-11-21 01:32:26,137 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 2950, loss[loss=0.07783, simple_loss=0.0949, pruned_loss=0.01871, audio_tagging_loss=0.01168, over 14609.00 frames. ], tot_loss[loss=0.07607, simple_loss=0.09732, pruned_loss=0.01761, audio_tagging_loss=0.009791, over 3050027.31 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:32:27,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1302180.0, ans=0.0
2023-11-21 01:32:53,732 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195350
2023-11-21 01:33:01,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1302313.3333333333, ans=0.2
2023-11-21 01:33:26,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1302446.6666666667, ans=0.07
2023-11-21 01:33:29,915 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3000, loss[loss=0.07858, simple_loss=0.09106, pruned_loss=0.02083, audio_tagging_loss=0.01222, over 14515.00 frames. ], tot_loss[loss=0.07645, simple_loss=0.09769, pruned_loss=0.01787, audio_tagging_loss=0.009734, over 3054419.64 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:33:29,916 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 01:34:10,817 INFO [train_asr.py:1253] (2/4) Epoch 17, validation: loss=0.06009, simple_loss=0.05276, pruned_loss=0.005332, audio_tagging_loss=0.02838, over 4681554.00 frames. 
2023-11-21 01:34:10,819 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 01:34:27,234 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.276e+01 8.086e+01 8.879e+01 9.716e+01 1.329e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 01:34:30,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1302580.0, ans=0.5
2023-11-21 01:34:37,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195400
2023-11-21 01:34:47,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.45 vs. limit=10.0
2023-11-21 01:35:15,249 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3050, loss[loss=0.07346, simple_loss=0.09025, pruned_loss=0.0166, audio_tagging_loss=0.01174, over 15679.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.09836, pruned_loss=0.01788, audio_tagging_loss=0.009732, over 3048829.61 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:35:17,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1302846.6666666667, ans=0.1
2023-11-21 01:35:25,532 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.55 vs. limit=22.5
2023-11-21 01:35:33,515 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.58 vs. limit=15.0
2023-11-21 01:35:41,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195450
2023-11-21 01:35:52,271 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:35:56,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1303046.6666666667, ans=0.125
2023-11-21 01:36:04,175 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:36:10,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=1303113.3333333333, ans=15.0
2023-11-21 01:36:18,589 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3100, loss[loss=0.06579, simple_loss=0.08466, pruned_loss=0.01227, audio_tagging_loss=0.0112, over 14506.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09904, pruned_loss=0.01788, audio_tagging_loss=0.00973, over 3044038.75 frames. ], batch size: 55, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:36:20,192 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:36:36,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.34 vs. limit=15.0
2023-11-21 01:36:36,455 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.588e+01 8.198e+01 8.813e+01 9.373e+01 1.166e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 01:36:39,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1303246.6666666667, ans=0.1
2023-11-21 01:36:46,855 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195500
2023-11-21 01:36:48,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1303313.3333333333, ans=0.0
2023-11-21 01:36:48,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.88 vs. limit=6.0
2023-11-21 01:36:50,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1303313.3333333333, ans=0.0
2023-11-21 01:36:54,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1303313.3333333333, ans=0.125
2023-11-21 01:36:55,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1303313.3333333333, ans=0.0
2023-11-21 01:36:58,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1303380.0, ans=0.0
2023-11-21 01:36:59,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1303380.0, ans=0.125
2023-11-21 01:37:18,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1303446.6666666667, ans=10.0
2023-11-21 01:37:20,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1303446.6666666667, ans=0.125
2023-11-21 01:37:22,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0
2023-11-21 01:37:23,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.49 vs. limit=22.5
2023-11-21 01:37:23,932 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3150, loss[loss=0.07158, simple_loss=0.08565, pruned_loss=0.01663, audio_tagging_loss=0.01211, over 15462.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09813, pruned_loss=0.01782, audio_tagging_loss=0.009881, over 3045321.27 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:37:27,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1303513.3333333333, ans=0.1
2023-11-21 01:37:50,702 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195550
2023-11-21 01:38:03,020 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.26 vs. limit=10.0
2023-11-21 01:38:16,914 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.42 vs. limit=10.0
2023-11-21 01:38:19,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1303780.0, ans=0.125
2023-11-21 01:38:25,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1303780.0, ans=0.125
2023-11-21 01:38:28,850 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3200, loss[loss=0.06403, simple_loss=0.08181, pruned_loss=0.01352, audio_tagging_loss=0.009611, over 14993.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09874, pruned_loss=0.01804, audio_tagging_loss=0.009955, over 3039665.08 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:38:31,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1303846.6666666667, ans=0.2
2023-11-21 01:38:33,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1303846.6666666667, ans=0.0
2023-11-21 01:38:44,759 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.303e+01 8.338e+01 8.946e+01 9.583e+01 3.907e+02, threshold=1.789e+02, percent-clipped=1.0
2023-11-21 01:38:46,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.95 vs. limit=15.0
2023-11-21 01:38:55,493 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195600
2023-11-21 01:39:16,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1304046.6666666667, ans=0.125
2023-11-21 01:39:32,127 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:39:33,040 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3250, loss[loss=0.08704, simple_loss=0.1196, pruned_loss=0.01871, audio_tagging_loss=0.008514, over 15375.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09862, pruned_loss=0.01792, audio_tagging_loss=0.009973, over 3041958.17 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:39:34,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1304180.0, ans=0.125
2023-11-21 01:39:42,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1304180.0, ans=0.95
2023-11-21 01:39:50,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1304246.6666666667, ans=0.125
2023-11-21 01:40:00,390 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195650
2023-11-21 01:40:06,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.79 vs. limit=22.5
2023-11-21 01:40:12,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1304380.0, ans=0.0
2023-11-21 01:40:35,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1304446.6666666667, ans=0.1
2023-11-21 01:40:37,901 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3300, loss[loss=0.0776, simple_loss=0.09924, pruned_loss=0.0166, audio_tagging_loss=0.01138, over 16648.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09919, pruned_loss=0.01798, audio_tagging_loss=0.01007, over 3048577.93 frames. ], batch size: 62, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:40:45,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.17 vs. limit=15.0
2023-11-21 01:40:56,306 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.228e+01 8.124e+01 8.901e+01 9.520e+01 1.377e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 01:41:05,086 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195700
2023-11-21 01:41:25,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1304713.3333333333, ans=0.125
2023-11-21 01:41:33,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1304780.0, ans=0.0
2023-11-21 01:41:35,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1304780.0, ans=0.0
2023-11-21 01:41:37,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1304780.0, ans=0.125
2023-11-21 01:41:40,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.74 vs. limit=12.0
2023-11-21 01:41:41,961 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3350, loss[loss=0.07264, simple_loss=0.08929, pruned_loss=0.01841, audio_tagging_loss=0.009586, over 15138.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09926, pruned_loss=0.01799, audio_tagging_loss=0.009891, over 3051408.68 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:42:08,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195750
2023-11-21 01:42:16,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1304980.0, ans=0.125
2023-11-21 01:42:19,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1305046.6666666667, ans=0.0
2023-11-21 01:42:21,757 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.49 vs. limit=15.0
2023-11-21 01:42:45,552 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3400, loss[loss=0.07368, simple_loss=0.09407, pruned_loss=0.01833, audio_tagging_loss=0.008313, over 15521.00 frames. ], tot_loss[loss=0.0771, simple_loss=0.09871, pruned_loss=0.01795, audio_tagging_loss=0.009789, over 3050414.73 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:42:56,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1305180.0, ans=0.125
2023-11-21 01:43:01,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1305246.6666666667, ans=0.125
2023-11-21 01:43:03,980 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.029e+01 8.814e+01 9.645e+01 1.986e+02, threshold=1.763e+02, percent-clipped=1.0
2023-11-21 01:43:05,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1305246.6666666667, ans=0.125
2023-11-21 01:43:10,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1305313.3333333333, ans=0.0
2023-11-21 01:43:12,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195800
2023-11-21 01:43:27,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1305380.0, ans=0.0
2023-11-21 01:43:49,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1305513.3333333333, ans=0.125
2023-11-21 01:43:51,467 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3450, loss[loss=0.0557, simple_loss=0.06551, pruned_loss=0.01216, audio_tagging_loss=0.01078, over 15267.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.09818, pruned_loss=0.01786, audio_tagging_loss=0.009849, over 3047022.80 frames. ], batch size: 62, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:44:08,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1305580.0, ans=0.2
2023-11-21 01:44:16,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1305646.6666666667, ans=0.0
2023-11-21 01:44:18,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195850
2023-11-21 01:44:26,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.90 vs. limit=15.0
2023-11-21 01:44:29,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1305713.3333333333, ans=0.0
2023-11-21 01:44:31,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1305713.3333333333, ans=0.125
2023-11-21 01:44:54,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1305846.6666666667, ans=0.0
2023-11-21 01:44:56,171 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3500, loss[loss=0.07316, simple_loss=0.09699, pruned_loss=0.01454, audio_tagging_loss=0.01012, over 15357.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.0976, pruned_loss=0.01764, audio_tagging_loss=0.009832, over 3037499.29 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:45:12,965 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:45:13,929 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.440e+01 8.307e+01 8.800e+01 9.894e+01 1.360e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 01:45:23,399 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195900
2023-11-21 01:45:29,946 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:45:37,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1306046.6666666667, ans=6.0
2023-11-21 01:45:54,664 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:45:58,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1306113.3333333333, ans=0.0
2023-11-21 01:46:00,382 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3550, loss[loss=0.06702, simple_loss=0.07534, pruned_loss=0.01545, audio_tagging_loss=0.0139, over 15483.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09712, pruned_loss=0.01749, audio_tagging_loss=0.009802, over 3041666.38 frames. ], batch size: 59, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:46:15,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1306246.6666666667, ans=0.1
2023-11-21 01:46:19,414 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.23 vs. limit=15.0
2023-11-21 01:46:27,494 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 195950
2023-11-21 01:46:35,805 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.49 vs. limit=15.0
2023-11-21 01:46:36,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1306313.3333333333, ans=0.125
2023-11-21 01:46:41,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.80 vs. limit=15.0
2023-11-21 01:47:02,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1306446.6666666667, ans=0.2
2023-11-21 01:47:04,472 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3600, loss[loss=0.0678, simple_loss=0.09153, pruned_loss=0.01341, audio_tagging_loss=0.008622, over 14328.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09586, pruned_loss=0.01712, audio_tagging_loss=0.009845, over 3045974.47 frames. ], batch size: 54, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:47:16,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1306580.0, ans=0.0
2023-11-21 01:47:17,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1306580.0, ans=0.1
2023-11-21 01:47:22,588 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.078e+01 8.527e+01 9.382e+01 1.423e+02, threshold=1.705e+02, percent-clipped=0.0
2023-11-21 01:47:30,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1306646.6666666667, ans=0.0
2023-11-21 01:47:31,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196000
2023-11-21 01:47:43,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1306646.6666666667, ans=0.1
2023-11-21 01:47:55,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1306713.3333333333, ans=0.125
2023-11-21 01:48:11,998 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3650, loss[loss=0.07357, simple_loss=0.09392, pruned_loss=0.02009, audio_tagging_loss=0.006518, over 15618.00 frames. ], tot_loss[loss=0.07537, simple_loss=0.09659, pruned_loss=0.01736, audio_tagging_loss=0.009715, over 3051779.97 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:48:21,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1306846.6666666667, ans=0.2
2023-11-21 01:48:38,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196050
2023-11-21 01:48:39,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1306980.0, ans=0.2
2023-11-21 01:49:08,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1307113.3333333333, ans=0.125
2023-11-21 01:49:15,771 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3700, loss[loss=0.07893, simple_loss=0.09531, pruned_loss=0.02097, audio_tagging_loss=0.01031, over 14071.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09694, pruned_loss=0.01731, audio_tagging_loss=0.009699, over 3048282.65 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:49:36,161 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.153e+01 8.365e+01 9.163e+01 1.051e+02 1.464e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-21 01:49:43,715 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196100
2023-11-21 01:49:46,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1307313.3333333333, ans=0.125
2023-11-21 01:50:04,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.57 vs. limit=6.0
2023-11-21 01:50:08,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1307446.6666666667, ans=0.0
2023-11-21 01:50:21,140 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3750, loss[loss=0.05489, simple_loss=0.06766, pruned_loss=0.007937, audio_tagging_loss=0.01313, over 15129.00 frames. ], tot_loss[loss=0.07574, simple_loss=0.09685, pruned_loss=0.01748, audio_tagging_loss=0.009834, over 3048466.62 frames. ], batch size: 59, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:50:22,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1307513.3333333333, ans=0.2
2023-11-21 01:50:24,495 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.77 vs. limit=10.0
2023-11-21 01:50:25,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1307513.3333333333, ans=0.0
2023-11-21 01:50:29,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1307513.3333333333, ans=22.5
2023-11-21 01:50:42,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1307580.0, ans=0.125
2023-11-21 01:50:48,159 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196150
2023-11-21 01:50:54,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1307646.6666666667, ans=0.125
2023-11-21 01:51:05,797 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:51:25,129 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3800, loss[loss=0.05835, simple_loss=0.07189, pruned_loss=0.01236, audio_tagging_loss=0.01005, over 15511.00 frames. ], tot_loss[loss=0.07519, simple_loss=0.09605, pruned_loss=0.01723, audio_tagging_loss=0.009931, over 3054144.53 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:51:44,157 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.062e+01 7.915e+01 8.603e+01 9.483e+01 1.210e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 01:51:52,370 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196200
2023-11-21 01:52:11,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.71 vs. limit=12.0
2023-11-21 01:52:22,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1308113.3333333333, ans=0.0
2023-11-21 01:52:29,389 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3850, loss[loss=0.0881, simple_loss=0.1219, pruned_loss=0.02099, audio_tagging_loss=0.006169, over 14613.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.0972, pruned_loss=0.01743, audio_tagging_loss=0.00988, over 3062158.48 frames. ], batch size: 52, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:52:43,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1308246.6666666667, ans=0.125
2023-11-21 01:52:45,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1308246.6666666667, ans=0.1
2023-11-21 01:52:47,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1308246.6666666667, ans=0.125
2023-11-21 01:52:50,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1308246.6666666667, ans=0.2
2023-11-21 01:52:50,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1308246.6666666667, ans=0.0
2023-11-21 01:52:54,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1308313.3333333333, ans=0.0
2023-11-21 01:52:56,917 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196250
2023-11-21 01:53:03,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1308313.3333333333, ans=0.125
2023-11-21 01:53:21,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1308446.6666666667, ans=0.2
2023-11-21 01:53:33,574 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3900, loss[loss=0.09133, simple_loss=0.118, pruned_loss=0.02273, audio_tagging_loss=0.009621, over 14959.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09768, pruned_loss=0.01746, audio_tagging_loss=0.009906, over 3062712.30 frames. ], batch size: 55, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 01:53:35,372 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.82 vs. limit=12.0
2023-11-21 01:53:52,213 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.160e+01 8.803e+01 9.797e+01 1.334e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 01:53:52,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1308580.0, ans=0.0
2023-11-21 01:53:53,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1308580.0, ans=0.0
2023-11-21 01:54:00,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196300
2023-11-21 01:54:13,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1308713.3333333333, ans=0.125
2023-11-21 01:54:15,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1308713.3333333333, ans=0.125
2023-11-21 01:54:37,445 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 3950, loss[loss=0.07246, simple_loss=0.0938, pruned_loss=0.01475, audio_tagging_loss=0.01081, over 15637.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.0984, pruned_loss=0.0176, audio_tagging_loss=0.009963, over 3061455.43 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 01:54:52,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1308913.3333333333, ans=0.04949747468305833
2023-11-21 01:54:57,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1308913.3333333333, ans=0.125
2023-11-21 01:55:01,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1308980.0, ans=0.07
2023-11-21 01:55:03,833 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196350
2023-11-21 01:55:06,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1308980.0, ans=0.125
2023-11-21 01:55:07,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1308980.0, ans=0.125
2023-11-21 01:55:19,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1309046.6666666667, ans=0.2
2023-11-21 01:55:31,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1309113.3333333333, ans=0.0
2023-11-21 01:55:32,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.97 vs. limit=15.0
2023-11-21 01:55:41,770 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4000, loss[loss=0.08891, simple_loss=0.114, pruned_loss=0.02219, audio_tagging_loss=0.009701, over 15188.00 frames. ], tot_loss[loss=0.07743, simple_loss=0.09901, pruned_loss=0.01793, audio_tagging_loss=0.009998, over 3066470.72 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:55:43,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1309180.0, ans=0.125
2023-11-21 01:55:47,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1309180.0, ans=0.125
2023-11-21 01:55:51,116 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.42 vs. limit=6.0
2023-11-21 01:56:00,786 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.301e+01 8.874e+01 9.780e+01 1.152e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 01:56:07,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1309313.3333333333, ans=0.0
2023-11-21 01:56:09,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196400
2023-11-21 01:56:42,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1309446.6666666667, ans=0.125
2023-11-21 01:56:45,655 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4050, loss[loss=0.07805, simple_loss=0.101, pruned_loss=0.01874, audio_tagging_loss=0.008797, over 14989.00 frames. ], tot_loss[loss=0.07718, simple_loss=0.09887, pruned_loss=0.01782, audio_tagging_loss=0.009926, over 3059861.49 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:56:49,892 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:56:54,630 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.87 vs. limit=22.5
2023-11-21 01:56:57,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1309513.3333333333, ans=0.125
2023-11-21 01:57:07,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.32 vs. limit=22.5
2023-11-21 01:57:13,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196450
2023-11-21 01:57:20,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1309646.6666666667, ans=0.125
2023-11-21 01:57:23,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1309646.6666666667, ans=0.125
2023-11-21 01:57:34,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=15.0
2023-11-21 01:57:37,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1309780.0, ans=0.0
2023-11-21 01:57:40,092 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.09 vs. limit=22.5
2023-11-21 01:57:50,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1309846.6666666667, ans=0.1
2023-11-21 01:57:51,804 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4100, loss[loss=0.08199, simple_loss=0.1155, pruned_loss=0.01746, audio_tagging_loss=0.006808, over 14732.00 frames. ], tot_loss[loss=0.07805, simple_loss=0.1001, pruned_loss=0.01812, audio_tagging_loss=0.00988, over 3055449.06 frames. ], batch size: 53, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:57:53,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1309846.6666666667, ans=0.04949747468305833
2023-11-21 01:57:53,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.50 vs. limit=15.0
2023-11-21 01:58:04,173 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.02 vs. limit=6.0
2023-11-21 01:58:10,684 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.112e+01 8.710e+01 9.440e+01 1.199e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 01:58:18,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196500
2023-11-21 01:58:34,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1310046.6666666667, ans=0.0
2023-11-21 01:58:38,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1310046.6666666667, ans=0.125
2023-11-21 01:58:40,410 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.70 vs. limit=10.0
2023-11-21 01:58:42,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.98 vs. limit=15.0
2023-11-21 01:58:46,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1310113.3333333333, ans=0.125
2023-11-21 01:58:50,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1310113.3333333333, ans=0.125
2023-11-21 01:58:53,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1310113.3333333333, ans=0.125
2023-11-21 01:58:56,158 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4150, loss[loss=0.05932, simple_loss=0.07547, pruned_loss=0.01263, audio_tagging_loss=0.008959, over 15542.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09946, pruned_loss=0.01802, audio_tagging_loss=0.009739, over 3053317.94 frames. ], batch size: 59, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:58:58,991 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:59:00,420 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.37 vs. limit=15.0
2023-11-21 01:59:14,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1310246.6666666667, ans=0.125
2023-11-21 01:59:23,466 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196550
2023-11-21 01:59:23,950 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=22.5
2023-11-21 01:59:32,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1310313.3333333333, ans=0.125
2023-11-21 01:59:33,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1310313.3333333333, ans=0.0
2023-11-21 01:59:39,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1310380.0, ans=0.125
2023-11-21 01:59:43,275 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:00:00,197 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4200, loss[loss=0.09778, simple_loss=0.13, pruned_loss=0.02409, audio_tagging_loss=0.008672, over 16545.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09988, pruned_loss=0.01805, audio_tagging_loss=0.009578, over 3055255.62 frames. ], batch size: 60, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:00:17,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1310580.0, ans=0.1
2023-11-21 02:00:17,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1310580.0, ans=0.125
2023-11-21 02:00:21,783 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.242e+01 8.811e+01 9.654e+01 1.216e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 02:00:23,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1310580.0, ans=0.125
2023-11-21 02:00:27,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1310646.6666666667, ans=0.125
2023-11-21 02:00:28,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196600
2023-11-21 02:00:32,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1310646.6666666667, ans=0.09899494936611666
2023-11-21 02:00:35,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1310646.6666666667, ans=0.125
2023-11-21 02:00:46,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1310713.3333333333, ans=0.2
2023-11-21 02:01:05,041 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4250, loss[loss=0.07994, simple_loss=0.09783, pruned_loss=0.02319, audio_tagging_loss=0.007833, over 14500.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.09999, pruned_loss=0.01818, audio_tagging_loss=0.0095, over 3049900.71 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:01:05,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1310846.6666666667, ans=0.0
2023-11-21 02:01:20,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1310913.3333333333, ans=0.125
2023-11-21 02:01:32,457 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196650
2023-11-21 02:01:39,442 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.15 vs. limit=6.0
2023-11-21 02:01:39,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1310980.0, ans=0.125
2023-11-21 02:01:47,760 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:01:55,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1311113.3333333333, ans=0.2
2023-11-21 02:02:05,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.26 vs. limit=22.5
2023-11-21 02:02:09,889 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4300, loss[loss=0.06024, simple_loss=0.07628, pruned_loss=0.01182, audio_tagging_loss=0.01027, over 15164.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.1001, pruned_loss=0.01827, audio_tagging_loss=0.009553, over 3047254.01 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:02:27,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1311246.6666666667, ans=0.125
2023-11-21 02:02:28,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1311246.6666666667, ans=0.5
2023-11-21 02:02:29,313 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.098e+01 8.656e+01 9.268e+01 1.139e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 02:02:36,095 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196700
2023-11-21 02:02:55,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1311380.0, ans=0.0
2023-11-21 02:03:05,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1311446.6666666667, ans=0.125
2023-11-21 02:03:11,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1311446.6666666667, ans=0.125
2023-11-21 02:03:11,632 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.63 vs. limit=22.5
2023-11-21 02:03:13,558 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4350, loss[loss=0.07296, simple_loss=0.09731, pruned_loss=0.01487, audio_tagging_loss=0.00944, over 15363.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.1002, pruned_loss=0.01804, audio_tagging_loss=0.00955, over 3050358.12 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:03:13,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1311513.3333333333, ans=0.0
2023-11-21 02:03:17,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1311513.3333333333, ans=0.125
2023-11-21 02:03:28,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1311580.0, ans=0.1
2023-11-21 02:03:38,658 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.64 vs. limit=22.5
2023-11-21 02:03:40,655 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196750
2023-11-21 02:03:43,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1311646.6666666667, ans=0.125
2023-11-21 02:03:46,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1311646.6666666667, ans=0.125
2023-11-21 02:03:53,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1311713.3333333333, ans=0.125
2023-11-21 02:03:54,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1311713.3333333333, ans=0.125
2023-11-21 02:04:01,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1311713.3333333333, ans=0.125
2023-11-21 02:04:05,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1311780.0, ans=0.2
2023-11-21 02:04:17,773 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4400, loss[loss=0.0717, simple_loss=0.0906, pruned_loss=0.01755, audio_tagging_loss=0.008845, over 15180.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.09874, pruned_loss=0.0179, audio_tagging_loss=0.009673, over 3049497.43 frames. ], batch size: 60, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:04:25,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1311846.6666666667, ans=0.0
2023-11-21 02:04:38,506 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 7.827e+01 8.393e+01 8.998e+01 1.090e+02, threshold=1.679e+02, percent-clipped=0.0
2023-11-21 02:04:44,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196800
2023-11-21 02:04:46,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1311980.0, ans=0.0
2023-11-21 02:04:57,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1312046.6666666667, ans=0.125
2023-11-21 02:05:13,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1312113.3333333333, ans=0.0
2023-11-21 02:05:22,839 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4450, loss[loss=0.07495, simple_loss=0.09873, pruned_loss=0.01729, audio_tagging_loss=0.008299, over 14836.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09883, pruned_loss=0.01795, audio_tagging_loss=0.009654, over 3048585.07 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:05:28,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1312180.0, ans=0.125
2023-11-21 02:05:49,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196850
2023-11-21 02:05:54,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1312313.3333333333, ans=0.125
2023-11-21 02:06:05,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1312380.0, ans=0.1
2023-11-21 02:06:07,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1312380.0, ans=0.1
2023-11-21 02:06:16,184 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=15.0
2023-11-21 02:06:24,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1312446.6666666667, ans=0.125
2023-11-21 02:06:24,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1312446.6666666667, ans=0.0
2023-11-21 02:06:26,540 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4500, loss[loss=0.0954, simple_loss=0.136, pruned_loss=0.02104, audio_tagging_loss=0.006371, over 15446.00 frames. ], tot_loss[loss=0.07753, simple_loss=0.0996, pruned_loss=0.01817, audio_tagging_loss=0.009568, over 3046067.62 frames. ], batch size: 55, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:06:48,062 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.230e+01 8.666e+01 9.490e+01 1.272e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 02:06:53,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196900
2023-11-21 02:06:57,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1312646.6666666667, ans=0.2
2023-11-21 02:06:58,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1312646.6666666667, ans=0.1
2023-11-21 02:06:59,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1312646.6666666667, ans=0.125
2023-11-21 02:07:30,279 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4550, loss[loss=0.0534, simple_loss=0.07243, pruned_loss=0.01032, audio_tagging_loss=0.006864, over 14493.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09819, pruned_loss=0.01767, audio_tagging_loss=0.009657, over 3041155.62 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:07:57,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 196950
2023-11-21 02:08:02,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1312980.0, ans=0.125
2023-11-21 02:08:03,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1312980.0, ans=0.2
2023-11-21 02:08:10,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1313046.6666666667, ans=0.125
2023-11-21 02:08:19,058 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:08:30,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1313113.3333333333, ans=0.125
2023-11-21 02:08:34,857 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4600, loss[loss=0.08183, simple_loss=0.1044, pruned_loss=0.02196, audio_tagging_loss=0.007669, over 14449.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09827, pruned_loss=0.01773, audio_tagging_loss=0.009694, over 3045047.65 frames. ], batch size: 54, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:08:40,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1313180.0, ans=0.07
2023-11-21 02:08:56,243 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.061e+01 8.654e+01 9.730e+01 1.161e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 02:09:01,171 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197000
2023-11-21 02:09:03,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.76 vs. limit=22.5
2023-11-21 02:09:38,871 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4650, loss[loss=0.05036, simple_loss=0.06121, pruned_loss=0.007922, audio_tagging_loss=0.01184, over 15004.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.0976, pruned_loss=0.01758, audio_tagging_loss=0.009778, over 3053576.93 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:09:50,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.31 vs. limit=15.0
2023-11-21 02:09:53,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1313580.0, ans=0.0
2023-11-21 02:09:53,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1313580.0, ans=0.0
2023-11-21 02:10:05,725 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197050
2023-11-21 02:10:10,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.96 vs. limit=22.5
2023-11-21 02:10:12,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1313646.6666666667, ans=0.0
2023-11-21 02:10:21,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1313713.3333333333, ans=0.125
2023-11-21 02:10:31,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1313780.0, ans=0.2
2023-11-21 02:10:40,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1313780.0, ans=0.0
2023-11-21 02:10:42,953 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4700, loss[loss=0.06687, simple_loss=0.08222, pruned_loss=0.01592, audio_tagging_loss=0.009846, over 14566.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09733, pruned_loss=0.01743, audio_tagging_loss=0.009754, over 3047850.62 frames. ], batch size: 55, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:11:04,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.79 vs. limit=10.0
2023-11-21 02:11:04,736 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.068e+01 8.673e+01 9.283e+01 1.102e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 02:11:05,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.91 vs. limit=15.0
2023-11-21 02:11:09,795 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197100
2023-11-21 02:11:19,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1313980.0, ans=0.125
2023-11-21 02:11:47,010 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4750, loss[loss=0.07404, simple_loss=0.09209, pruned_loss=0.01802, audio_tagging_loss=0.009982, over 15195.00 frames. ], tot_loss[loss=0.07523, simple_loss=0.09597, pruned_loss=0.01727, audio_tagging_loss=0.009971, over 3046922.37 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:12:06,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1314246.6666666667, ans=0.1
2023-11-21 02:12:13,863 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197150
2023-11-21 02:12:40,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1314446.6666666667, ans=0.1
2023-11-21 02:12:47,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1314446.6666666667, ans=0.125
2023-11-21 02:12:50,789 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4800, loss[loss=0.0738, simple_loss=0.08502, pruned_loss=0.01858, audio_tagging_loss=0.01272, over 14942.00 frames. ], tot_loss[loss=0.07636, simple_loss=0.09754, pruned_loss=0.01762, audio_tagging_loss=0.009968, over 3049378.78 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:12:50,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1314513.3333333333, ans=0.125
2023-11-21 02:13:12,700 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.143e+01 8.825e+01 9.696e+01 1.559e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 02:13:18,365 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197200
2023-11-21 02:13:19,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1314646.6666666667, ans=10.0
2023-11-21 02:13:36,162 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.79 vs. limit=15.0
2023-11-21 02:13:55,413 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4850, loss[loss=0.06836, simple_loss=0.089, pruned_loss=0.01414, audio_tagging_loss=0.009719, over 15652.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09672, pruned_loss=0.01746, audio_tagging_loss=0.01009, over 3044639.54 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:13:56,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1314846.6666666667, ans=0.2
2023-11-21 02:14:09,618 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=12.19 vs. limit=15.0
2023-11-21 02:14:21,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1314980.0, ans=0.1
2023-11-21 02:14:22,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197250
2023-11-21 02:14:36,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1315046.6666666667, ans=0.125
2023-11-21 02:14:49,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1315113.3333333333, ans=0.0
2023-11-21 02:14:49,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1315113.3333333333, ans=0.0
2023-11-21 02:14:59,942 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4900, loss[loss=0.06973, simple_loss=0.08113, pruned_loss=0.0186, audio_tagging_loss=0.01057, over 16578.00 frames. ], tot_loss[loss=0.07624, simple_loss=0.09745, pruned_loss=0.01754, audio_tagging_loss=0.009974, over 3045564.77 frames. ], batch size: 64, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:15:21,160 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.125e+01 8.648e+01 9.394e+01 1.252e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 02:15:26,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197300
2023-11-21 02:15:31,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.72 vs. limit=8.0
2023-11-21 02:15:33,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1315313.3333333333, ans=0.1
2023-11-21 02:15:44,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1315380.0, ans=0.0
2023-11-21 02:15:46,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1315380.0, ans=0.125
2023-11-21 02:16:03,675 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 4950, loss[loss=0.06452, simple_loss=0.08764, pruned_loss=0.01174, audio_tagging_loss=0.008963, over 15952.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09775, pruned_loss=0.01748, audio_tagging_loss=0.009815, over 3051424.94 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:16:24,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1315580.0, ans=0.125
2023-11-21 02:16:31,173 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197350
2023-11-21 02:16:32,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1315646.6666666667, ans=0.125
2023-11-21 02:16:33,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1315646.6666666667, ans=0.0
2023-11-21 02:16:51,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1315713.3333333333, ans=0.125
2023-11-21 02:17:07,625 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5000, loss[loss=0.09011, simple_loss=0.121, pruned_loss=0.02067, audio_tagging_loss=0.008931, over 15632.00 frames. ], tot_loss[loss=0.07551, simple_loss=0.0969, pruned_loss=0.01731, audio_tagging_loss=0.009742, over 3054075.97 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:17:14,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1315846.6666666667, ans=0.125
2023-11-21 02:17:18,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=1315846.6666666667, ans=0.025
2023-11-21 02:17:29,321 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.139e+01 8.846e+01 9.798e+01 1.314e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 02:17:34,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197400
2023-11-21 02:17:39,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1315980.0, ans=0.2
2023-11-21 02:17:45,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1316046.6666666667, ans=0.025
2023-11-21 02:17:50,146 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.37 vs. limit=15.0
2023-11-21 02:17:58,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1316113.3333333333, ans=0.0
2023-11-21 02:18:03,560 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=15.0
2023-11-21 02:18:09,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1316113.3333333333, ans=0.125
2023-11-21 02:18:12,061 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5050, loss[loss=0.1107, simple_loss=0.1443, pruned_loss=0.03164, audio_tagging_loss=0.006913, over 16748.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.097, pruned_loss=0.01737, audio_tagging_loss=0.009664, over 3041955.54 frames. ], batch size: 59, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:18:21,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1316180.0, ans=0.1
2023-11-21 02:18:38,401 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197450
2023-11-21 02:18:45,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1316313.3333333333, ans=0.0
2023-11-21 02:19:03,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1316446.6666666667, ans=0.125
2023-11-21 02:19:16,006 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5100, loss[loss=0.08535, simple_loss=0.1064, pruned_loss=0.02134, audio_tagging_loss=0.01081, over 14890.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09711, pruned_loss=0.01738, audio_tagging_loss=0.009635, over 3039946.00 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:19:26,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1316513.3333333333, ans=0.2
2023-11-21 02:19:31,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1316580.0, ans=0.0
2023-11-21 02:19:34,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1316580.0, ans=0.125
2023-11-21 02:19:37,080 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.469e+01 7.939e+01 8.776e+01 9.689e+01 1.456e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 02:19:42,565 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197500
2023-11-21 02:20:18,818 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5150, loss[loss=0.09652, simple_loss=0.1303, pruned_loss=0.02158, audio_tagging_loss=0.00979, over 14735.00 frames. ], tot_loss[loss=0.07564, simple_loss=0.09716, pruned_loss=0.0175, audio_tagging_loss=0.009557, over 3040590.92 frames. ], batch size: 54, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:20:46,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197550
2023-11-21 02:20:50,685 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.16 vs. limit=22.5
2023-11-21 02:21:23,743 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5200, loss[loss=0.06757, simple_loss=0.09114, pruned_loss=0.01456, audio_tagging_loss=0.00744, over 14723.00 frames. ], tot_loss[loss=0.07607, simple_loss=0.09774, pruned_loss=0.01774, audio_tagging_loss=0.009468, over 3047434.57 frames. ], batch size: 55, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:21:35,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1317246.6666666667, ans=0.125
2023-11-21 02:21:36,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1317246.6666666667, ans=0.125
2023-11-21 02:21:41,733 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.00 vs. limit=15.0
2023-11-21 02:21:44,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.249e+01 8.824e+01 9.467e+01 1.175e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 02:21:49,703 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197600
2023-11-21 02:21:53,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1317313.3333333333, ans=0.0
2023-11-21 02:21:55,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1317313.3333333333, ans=0.0
2023-11-21 02:21:58,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1317313.3333333333, ans=0.125
2023-11-21 02:22:02,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1317380.0, ans=0.1
2023-11-21 02:22:08,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1317380.0, ans=0.125
2023-11-21 02:22:25,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1317446.6666666667, ans=0.0
2023-11-21 02:22:25,933 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.79 vs. limit=15.0
2023-11-21 02:22:27,576 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5250, loss[loss=0.08273, simple_loss=0.1092, pruned_loss=0.01861, audio_tagging_loss=0.009504, over 14456.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09804, pruned_loss=0.01789, audio_tagging_loss=0.009506, over 3043009.73 frames. ], batch size: 55, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:22:37,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1317513.3333333333, ans=0.035
2023-11-21 02:22:38,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1317580.0, ans=0.0
2023-11-21 02:22:54,326 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197650
2023-11-21 02:23:05,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1317713.3333333333, ans=0.125
2023-11-21 02:23:08,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1317713.3333333333, ans=0.0
2023-11-21 02:23:30,863 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5300, loss[loss=0.05197, simple_loss=0.05718, pruned_loss=0.0115, audio_tagging_loss=0.01188, over 16021.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09765, pruned_loss=0.01776, audio_tagging_loss=0.009544, over 3051424.75 frames. ], batch size: 65, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:23:32,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1317846.6666666667, ans=0.0
2023-11-21 02:23:36,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1317846.6666666667, ans=0.2
2023-11-21 02:23:43,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1317913.3333333333, ans=0.0
2023-11-21 02:23:46,974 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:23:53,260 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.065e+01 8.722e+01 9.594e+01 1.316e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 02:23:58,316 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197700
2023-11-21 02:24:25,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1318113.3333333333, ans=0.125
2023-11-21 02:24:34,805 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5350, loss[loss=0.07823, simple_loss=0.1028, pruned_loss=0.01837, audio_tagging_loss=0.008471, over 15215.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09705, pruned_loss=0.01769, audio_tagging_loss=0.009686, over 3048531.84 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:24:39,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.33 vs. limit=12.0
2023-11-21 02:24:55,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1318246.6666666667, ans=0.2
2023-11-21 02:25:01,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.10 vs. limit=22.5
2023-11-21 02:25:02,206 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197750
2023-11-21 02:25:11,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1318380.0, ans=0.125
2023-11-21 02:25:14,757 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.67 vs. limit=15.0
2023-11-21 02:25:17,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.16 vs. limit=22.5
2023-11-21 02:25:28,111 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.46 vs. limit=6.0
2023-11-21 02:25:38,681 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5400, loss[loss=0.06473, simple_loss=0.08167, pruned_loss=0.01412, audio_tagging_loss=0.00977, over 16200.00 frames. ], tot_loss[loss=0.07582, simple_loss=0.0968, pruned_loss=0.01773, audio_tagging_loss=0.009687, over 3046456.08 frames. ], batch size: 58, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:25:48,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1318513.3333333333, ans=0.0
2023-11-21 02:25:50,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1318580.0, ans=0.2
2023-11-21 02:25:56,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.33 vs. limit=22.5
2023-11-21 02:26:00,531 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.677e+01 8.214e+01 8.750e+01 9.356e+01 1.169e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 02:26:04,400 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197800
2023-11-21 02:26:38,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1318780.0, ans=0.025
2023-11-21 02:26:39,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1318780.0, ans=0.125
2023-11-21 02:26:41,775 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5450, loss[loss=0.08115, simple_loss=0.1007, pruned_loss=0.02064, audio_tagging_loss=0.01015, over 15699.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09754, pruned_loss=0.01795, audio_tagging_loss=0.0097, over 3049420.72 frames. ], batch size: 59, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:26:43,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.19 vs. limit=15.0
2023-11-21 02:26:47,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1318846.6666666667, ans=0.125
2023-11-21 02:26:53,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1318913.3333333333, ans=0.125
2023-11-21 02:26:58,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1318913.3333333333, ans=0.2
2023-11-21 02:27:09,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197850
2023-11-21 02:27:09,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1318980.0, ans=0.0
2023-11-21 02:27:10,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1318980.0, ans=0.125
2023-11-21 02:27:22,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.95 vs. limit=15.0
2023-11-21 02:27:25,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1319046.6666666667, ans=0.1
2023-11-21 02:27:35,106 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:27:41,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1319113.3333333333, ans=0.0
2023-11-21 02:27:41,556 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.96 vs. limit=10.0
2023-11-21 02:27:45,086 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5500, loss[loss=0.07401, simple_loss=0.1001, pruned_loss=0.01586, audio_tagging_loss=0.008091, over 13896.00 frames. ], tot_loss[loss=0.07689, simple_loss=0.0983, pruned_loss=0.01804, audio_tagging_loss=0.009697, over 3049427.31 frames. ], batch size: 54, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:28:00,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1319246.6666666667, ans=0.125
2023-11-21 02:28:08,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.510e+01 8.155e+01 8.783e+01 9.511e+01 1.245e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 02:28:12,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197900
2023-11-21 02:28:20,217 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-21 02:28:23,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1319380.0, ans=0.0
2023-11-21 02:28:23,992 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.18 vs. limit=15.0
2023-11-21 02:28:27,324 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:28:32,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.56 vs. limit=22.5
2023-11-21 02:28:35,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1319446.6666666667, ans=0.1
2023-11-21 02:28:46,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.84 vs. limit=22.5
2023-11-21 02:28:49,898 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5550, loss[loss=0.07135, simple_loss=0.08422, pruned_loss=0.02078, audio_tagging_loss=0.008457, over 14555.00 frames. ], tot_loss[loss=0.07665, simple_loss=0.09763, pruned_loss=0.01797, audio_tagging_loss=0.009872, over 3038892.66 frames. ], batch size: 55, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:28:53,767 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.805e-01
2023-11-21 02:28:54,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1319513.3333333333, ans=0.0
2023-11-21 02:29:09,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1319580.0, ans=0.0
2023-11-21 02:29:11,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1319580.0, ans=0.035
2023-11-21 02:29:15,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 197950
2023-11-21 02:29:38,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1319713.3333333333, ans=0.95
2023-11-21 02:29:45,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1319780.0, ans=0.2
2023-11-21 02:29:49,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1319780.0, ans=0.125
2023-11-21 02:29:49,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1319780.0, ans=0.1
2023-11-21 02:29:52,778 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5600, loss[loss=0.06345, simple_loss=0.08065, pruned_loss=0.01289, audio_tagging_loss=0.01024, over 16928.00 frames. ], tot_loss[loss=0.07687, simple_loss=0.09815, pruned_loss=0.01785, audio_tagging_loss=0.009942, over 3050132.86 frames. ], batch size: 63, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:29:54,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1319846.6666666667, ans=0.1
2023-11-21 02:30:05,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1319913.3333333333, ans=0.125
2023-11-21 02:30:05,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1319913.3333333333, ans=0.125
2023-11-21 02:30:17,153 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.123e+01 8.788e+01 9.495e+01 1.515e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 02:30:17,966 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0
2023-11-21 02:30:19,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198000
2023-11-21 02:30:25,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1319980.0, ans=0.125
2023-11-21 02:30:37,693 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:30:39,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1320046.6666666667, ans=0.125
2023-11-21 02:30:43,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1320113.3333333333, ans=0.125
2023-11-21 02:30:55,888 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5650, loss[loss=0.1027, simple_loss=0.1382, pruned_loss=0.02775, audio_tagging_loss=0.005837, over 16234.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.09899, pruned_loss=0.01799, audio_tagging_loss=0.009975, over 3057725.78 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:30:59,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1320180.0, ans=0.0
2023-11-21 02:31:09,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=10.02 vs. limit=10.0
2023-11-21 02:31:23,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198050
2023-11-21 02:31:26,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1320313.3333333333, ans=0.125
2023-11-21 02:31:26,850 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.36 vs. limit=15.0
2023-11-21 02:31:56,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1320446.6666666667, ans=0.2
2023-11-21 02:31:59,782 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5700, loss[loss=0.07798, simple_loss=0.09802, pruned_loss=0.01835, audio_tagging_loss=0.01062, over 15138.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09753, pruned_loss=0.0177, audio_tagging_loss=0.009938, over 3057875.29 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:32:19,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1320580.0, ans=0.125
2023-11-21 02:32:23,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1320580.0, ans=0.125
2023-11-21 02:32:23,988 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.139e+01 8.085e+01 8.943e+01 9.516e+01 1.355e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 02:32:26,634 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198100
2023-11-21 02:32:28,785 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0
2023-11-21 02:33:03,879 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5750, loss[loss=0.06128, simple_loss=0.07224, pruned_loss=0.01494, audio_tagging_loss=0.01021, over 14707.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09752, pruned_loss=0.01773, audio_tagging_loss=0.009802, over 3056682.62 frames. ], batch size: 60, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:33:13,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1320846.6666666667, ans=0.2
2023-11-21 02:33:30,769 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198150
2023-11-21 02:33:39,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1320980.0, ans=0.1
2023-11-21 02:33:54,427 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.57 vs. limit=15.0
2023-11-21 02:34:01,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1321113.3333333333, ans=0.1
2023-11-21 02:34:06,823 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5800, loss[loss=0.0935, simple_loss=0.1114, pruned_loss=0.02865, audio_tagging_loss=0.009155, over 14846.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.09764, pruned_loss=0.01782, audio_tagging_loss=0.009673, over 3050109.27 frames. ], batch size: 56, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:34:14,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1321180.0, ans=0.125
2023-11-21 02:34:21,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1321246.6666666667, ans=0.125
2023-11-21 02:34:28,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1321246.6666666667, ans=0.2
2023-11-21 02:34:31,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.88 vs. limit=15.0
2023-11-21 02:34:31,416 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.619e+01 8.028e+01 8.750e+01 9.322e+01 1.418e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 02:34:34,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198200
2023-11-21 02:34:42,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1321313.3333333333, ans=0.2
2023-11-21 02:35:11,405 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5850, loss[loss=0.0771, simple_loss=0.1047, pruned_loss=0.01704, audio_tagging_loss=0.00771, over 15383.00 frames. ], tot_loss[loss=0.07598, simple_loss=0.09756, pruned_loss=0.01764, audio_tagging_loss=0.009557, over 3052260.01 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:35:12,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1321513.3333333333, ans=0.2
2023-11-21 02:35:21,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1321513.3333333333, ans=0.0
2023-11-21 02:35:28,801 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=3.662e-02
2023-11-21 02:35:38,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198250
2023-11-21 02:35:52,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1321713.3333333333, ans=0.125
2023-11-21 02:36:15,432 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5900, loss[loss=0.06774, simple_loss=0.08406, pruned_loss=0.01628, audio_tagging_loss=0.009423, over 14940.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09734, pruned_loss=0.01755, audio_tagging_loss=0.009593, over 3050016.18 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:36:24,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1321846.6666666667, ans=0.125
2023-11-21 02:36:39,026 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.990e+01 8.061e+01 8.738e+01 9.398e+01 2.090e+02, threshold=1.748e+02, percent-clipped=1.0
2023-11-21 02:36:41,560 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198300
2023-11-21 02:37:18,285 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 5950, loss[loss=0.07883, simple_loss=0.09829, pruned_loss=0.02189, audio_tagging_loss=0.007801, over 14712.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.098, pruned_loss=0.01768, audio_tagging_loss=0.00943, over 3050788.30 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:37:22,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1322180.0, ans=0.125
2023-11-21 02:37:34,770 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0
2023-11-21 02:37:45,617 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198350
2023-11-21 02:37:58,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1322380.0, ans=0.0
2023-11-21 02:38:02,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1322380.0, ans=0.1
2023-11-21 02:38:22,702 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6000, loss[loss=0.04987, simple_loss=0.06244, pruned_loss=0.007134, audio_tagging_loss=0.01152, over 16036.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09764, pruned_loss=0.01763, audio_tagging_loss=0.009552, over 3051384.64 frames. ], batch size: 63, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:38:22,703 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 02:39:04,194 INFO [train_asr.py:1253] (2/4) Epoch 17, validation: loss=0.06056, simple_loss=0.05273, pruned_loss=0.005281, audio_tagging_loss=0.02892, over 4681554.00 frames. 
2023-11-21 02:39:04,195 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 02:39:13,961 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:39:28,367 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.213e+01 8.088e+01 8.691e+01 9.569e+01 1.376e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 02:39:31,146 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198400
2023-11-21 02:39:48,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1322713.3333333333, ans=0.125
2023-11-21 02:39:50,596 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:39:57,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.97 vs. limit=15.0
2023-11-21 02:39:59,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.50 vs. limit=15.0
2023-11-21 02:40:08,176 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6050, loss[loss=0.08176, simple_loss=0.1102, pruned_loss=0.02085, audio_tagging_loss=0.005822, over 16145.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09832, pruned_loss=0.01774, audio_tagging_loss=0.009525, over 3046004.39 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:40:35,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198450
2023-11-21 02:40:36,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.42 vs. limit=22.5
2023-11-21 02:40:44,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1322980.0, ans=0.125
2023-11-21 02:40:55,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1323046.6666666667, ans=0.0
2023-11-21 02:41:12,333 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6100, loss[loss=0.09681, simple_loss=0.124, pruned_loss=0.02612, audio_tagging_loss=0.008693, over 15997.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.09769, pruned_loss=0.01758, audio_tagging_loss=0.009666, over 3051212.09 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:41:12,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1323180.0, ans=0.125
2023-11-21 02:41:16,628 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.28 vs. limit=15.0
2023-11-21 02:41:21,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1323180.0, ans=0.1
2023-11-21 02:41:36,815 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.186e+01 8.873e+01 9.701e+01 1.239e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 02:41:38,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1323313.3333333333, ans=0.0
2023-11-21 02:41:39,342 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198500
2023-11-21 02:41:59,257 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.18 vs. limit=12.0
2023-11-21 02:42:14,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1323446.6666666667, ans=0.0
2023-11-21 02:42:16,016 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6150, loss[loss=0.05999, simple_loss=0.07837, pruned_loss=0.009955, audio_tagging_loss=0.01085, over 15974.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09655, pruned_loss=0.01731, audio_tagging_loss=0.009706, over 3045255.06 frames. ], batch size: 61, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:42:28,220 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.53 vs. limit=15.0
2023-11-21 02:42:38,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1323580.0, ans=0.0
2023-11-21 02:42:40,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1323646.6666666667, ans=0.125
2023-11-21 02:42:43,478 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198550
2023-11-21 02:42:52,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.47 vs. limit=15.0
2023-11-21 02:43:01,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.35 vs. limit=15.0
2023-11-21 02:43:09,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1323780.0, ans=0.125
2023-11-21 02:43:16,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1323780.0, ans=0.125
2023-11-21 02:43:19,601 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6200, loss[loss=0.07914, simple_loss=0.1034, pruned_loss=0.01858, audio_tagging_loss=0.008855, over 14522.00 frames. ], tot_loss[loss=0.07597, simple_loss=0.09742, pruned_loss=0.01752, audio_tagging_loss=0.009742, over 3039299.45 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:43:23,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1323846.6666666667, ans=0.125
2023-11-21 02:43:35,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1323913.3333333333, ans=0.125
2023-11-21 02:43:42,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.33 vs. limit=10.0
2023-11-21 02:43:45,294 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.093e+01 8.158e+01 8.699e+01 9.470e+01 1.213e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 02:43:47,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198600
2023-11-21 02:43:55,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.36 vs. limit=15.0
2023-11-21 02:44:06,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1324046.6666666667, ans=0.1
2023-11-21 02:44:09,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=12.0
2023-11-21 02:44:24,705 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6250, loss[loss=0.08322, simple_loss=0.1104, pruned_loss=0.01947, audio_tagging_loss=0.008564, over 15173.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.0975, pruned_loss=0.01752, audio_tagging_loss=0.009826, over 3039593.56 frames. ], batch size: 59, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:44:25,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1324180.0, ans=0.2
2023-11-21 02:44:26,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.44 vs. limit=6.0
2023-11-21 02:44:40,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.73 vs. limit=22.5
2023-11-21 02:44:50,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198650
2023-11-21 02:44:58,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1324313.3333333333, ans=0.2
2023-11-21 02:45:02,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1324380.0, ans=0.0
2023-11-21 02:45:14,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1324446.6666666667, ans=0.125
2023-11-21 02:45:15,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.00 vs. limit=10.0
2023-11-21 02:45:24,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1324446.6666666667, ans=0.125
2023-11-21 02:45:27,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1324513.3333333333, ans=0.05
2023-11-21 02:45:28,201 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6300, loss[loss=0.0691, simple_loss=0.08168, pruned_loss=0.01782, audio_tagging_loss=0.01044, over 16027.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09727, pruned_loss=0.01753, audio_tagging_loss=0.009871, over 3041820.79 frames. ], batch size: 61, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:45:34,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1324513.3333333333, ans=0.0
2023-11-21 02:45:45,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1324580.0, ans=0.0
2023-11-21 02:45:50,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1324580.0, ans=0.125
2023-11-21 02:45:53,587 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.806e+01 8.207e+01 8.844e+01 9.523e+01 1.211e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 02:45:55,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198700
2023-11-21 02:46:10,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1324713.3333333333, ans=0.09899494936611666
2023-11-21 02:46:11,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1324713.3333333333, ans=0.125
2023-11-21 02:46:18,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1324780.0, ans=0.1
2023-11-21 02:46:22,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1324780.0, ans=0.2
2023-11-21 02:46:31,999 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6350, loss[loss=0.0501, simple_loss=0.06031, pruned_loss=0.01032, audio_tagging_loss=0.009625, over 14212.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09729, pruned_loss=0.01763, audio_tagging_loss=0.009839, over 3050154.04 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:46:46,714 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.63 vs. limit=10.0
2023-11-21 02:46:59,710 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198750
2023-11-21 02:47:23,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1325113.3333333333, ans=0.1
2023-11-21 02:47:25,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.86 vs. limit=15.0
2023-11-21 02:47:37,366 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6400, loss[loss=0.101, simple_loss=0.1257, pruned_loss=0.02831, audio_tagging_loss=0.00983, over 16309.00 frames. ], tot_loss[loss=0.07598, simple_loss=0.09683, pruned_loss=0.01759, audio_tagging_loss=0.00997, over 3049545.27 frames. ], batch size: 60, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:47:45,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1325180.0, ans=0.1
2023-11-21 02:47:54,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.31 vs. limit=15.0
2023-11-21 02:48:02,548 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 8.215e+01 8.788e+01 9.513e+01 1.963e+02, threshold=1.758e+02, percent-clipped=1.0
2023-11-21 02:48:03,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198800
2023-11-21 02:48:06,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1325313.3333333333, ans=0.0
2023-11-21 02:48:36,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1325446.6666666667, ans=0.0
2023-11-21 02:48:37,045 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.33 vs. limit=6.0
2023-11-21 02:48:41,905 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6450, loss[loss=0.07041, simple_loss=0.09261, pruned_loss=0.01482, audio_tagging_loss=0.009279, over 15274.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09775, pruned_loss=0.0179, audio_tagging_loss=0.01007, over 3045248.74 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:49:07,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198850
2023-11-21 02:49:32,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1325780.0, ans=0.04949747468305833
2023-11-21 02:49:32,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1325780.0, ans=0.125
2023-11-21 02:49:34,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1325780.0, ans=0.125
2023-11-21 02:49:45,182 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6500, loss[loss=0.05148, simple_loss=0.06268, pruned_loss=0.007168, audio_tagging_loss=0.01297, over 15089.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.09676, pruned_loss=0.01764, audio_tagging_loss=0.01009, over 3047776.21 frames. ], batch size: 60, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:49:57,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1325913.3333333333, ans=0.0
2023-11-21 02:50:10,554 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.009e+01 8.684e+01 9.310e+01 1.221e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 02:50:11,870 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198900
2023-11-21 02:50:24,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.26 vs. limit=15.0
2023-11-21 02:50:27,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1326046.6666666667, ans=0.125
2023-11-21 02:50:40,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.55 vs. limit=15.0
2023-11-21 02:50:48,742 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6550, loss[loss=0.0838, simple_loss=0.1189, pruned_loss=0.01659, audio_tagging_loss=0.007761, over 15419.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09762, pruned_loss=0.01774, audio_tagging_loss=0.009968, over 3048745.21 frames. ], batch size: 55, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:50:50,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1326180.0, ans=0.125
2023-11-21 02:51:03,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1326246.6666666667, ans=0.125
2023-11-21 02:51:09,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1326246.6666666667, ans=0.125
2023-11-21 02:51:12,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1326246.6666666667, ans=0.0
2023-11-21 02:51:15,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 198950
2023-11-21 02:51:17,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1326313.3333333333, ans=0.125
2023-11-21 02:51:20,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0
2023-11-21 02:51:22,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1326313.3333333333, ans=0.1
2023-11-21 02:51:45,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1326446.6666666667, ans=0.125
2023-11-21 02:51:52,726 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6600, loss[loss=0.08963, simple_loss=0.113, pruned_loss=0.02357, audio_tagging_loss=0.009553, over 15562.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09843, pruned_loss=0.01803, audio_tagging_loss=0.009887, over 3051219.17 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:51:55,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.36 vs. limit=8.0
2023-11-21 02:52:17,930 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.252e+01 8.813e+01 9.521e+01 1.214e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 02:52:19,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199000
2023-11-21 02:52:45,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1326780.0, ans=0.2
2023-11-21 02:52:48,230 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.52 vs. limit=6.0
2023-11-21 02:52:57,341 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6650, loss[loss=0.06135, simple_loss=0.08268, pruned_loss=0.01111, audio_tagging_loss=0.008897, over 15956.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09873, pruned_loss=0.01806, audio_tagging_loss=0.009685, over 3050666.36 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:52:58,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1326846.6666666667, ans=0.0
2023-11-21 02:53:01,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten.whitening_limit, batch_count=1326846.6666666667, ans=15.0
2023-11-21 02:53:25,137 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199050
2023-11-21 02:53:33,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1326980.0, ans=0.125
2023-11-21 02:54:01,186 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6700, loss[loss=0.08326, simple_loss=0.1082, pruned_loss=0.01928, audio_tagging_loss=0.009881, over 15735.00 frames. ], tot_loss[loss=0.0769, simple_loss=0.09834, pruned_loss=0.01801, audio_tagging_loss=0.009715, over 3053176.51 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:54:03,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1327180.0, ans=0.1
2023-11-21 02:54:04,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1327180.0, ans=0.125
2023-11-21 02:54:10,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1327180.0, ans=0.125
2023-11-21 02:54:11,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1327180.0, ans=0.125
2023-11-21 02:54:27,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1327313.3333333333, ans=0.025
2023-11-21 02:54:28,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1327313.3333333333, ans=0.2
2023-11-21 02:54:29,386 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 7.825e+01 8.466e+01 9.524e+01 1.131e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-21 02:54:29,533 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199100
2023-11-21 02:54:33,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1327313.3333333333, ans=0.125
2023-11-21 02:54:33,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1327313.3333333333, ans=0.0
2023-11-21 02:54:48,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1327380.0, ans=0.125
2023-11-21 02:54:50,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1327380.0, ans=0.2
2023-11-21 02:54:58,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.82 vs. limit=15.0
2023-11-21 02:55:05,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1327513.3333333333, ans=0.125
2023-11-21 02:55:06,854 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6750, loss[loss=0.06684, simple_loss=0.08679, pruned_loss=0.01548, audio_tagging_loss=0.007969, over 14144.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09836, pruned_loss=0.01795, audio_tagging_loss=0.009724, over 3043592.89 frames. ], batch size: 55, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:55:15,659 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.55 vs. limit=15.0
2023-11-21 02:55:32,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199150
2023-11-21 02:56:10,695 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6800, loss[loss=0.08555, simple_loss=0.1151, pruned_loss=0.02158, audio_tagging_loss=0.006408, over 16071.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09815, pruned_loss=0.01793, audio_tagging_loss=0.00968, over 3044324.40 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:56:13,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1327846.6666666667, ans=0.1
2023-11-21 02:56:16,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1327846.6666666667, ans=0.125
2023-11-21 02:56:19,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1327846.6666666667, ans=0.1
2023-11-21 02:56:26,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1327913.3333333333, ans=0.125
2023-11-21 02:56:33,523 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.92 vs. limit=10.0
2023-11-21 02:56:37,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199200
2023-11-21 02:56:38,911 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.071e+01 8.833e+01 9.608e+01 1.176e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 02:57:05,631 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.12 vs. limit=15.0
2023-11-21 02:57:13,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1328180.0, ans=0.125
2023-11-21 02:57:14,625 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6850, loss[loss=0.07756, simple_loss=0.1011, pruned_loss=0.01581, audio_tagging_loss=0.01118, over 15218.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09808, pruned_loss=0.01779, audio_tagging_loss=0.009597, over 3043805.04 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 02:57:19,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1328180.0, ans=0.125
2023-11-21 02:57:37,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1328246.6666666667, ans=0.015
2023-11-21 02:57:42,698 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199250
2023-11-21 02:57:47,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1328313.3333333333, ans=0.1
2023-11-21 02:57:56,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1328380.0, ans=0.125
2023-11-21 02:57:58,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1328380.0, ans=0.2
2023-11-21 02:58:11,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1328446.6666666667, ans=0.09899494936611666
2023-11-21 02:58:19,383 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6900, loss[loss=0.0702, simple_loss=0.09311, pruned_loss=0.0137, audio_tagging_loss=0.009948, over 15857.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09796, pruned_loss=0.01773, audio_tagging_loss=0.009592, over 3046057.05 frames. ], batch size: 60, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 02:58:28,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1328513.3333333333, ans=0.5
2023-11-21 02:58:46,560 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199300
2023-11-21 02:58:47,623 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.846e+01 8.588e+01 9.245e+01 1.215e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 02:59:06,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1328713.3333333333, ans=0.125
2023-11-21 02:59:08,584 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:59:14,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1328780.0, ans=0.125
2023-11-21 02:59:23,854 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 6950, loss[loss=0.08104, simple_loss=0.1091, pruned_loss=0.01843, audio_tagging_loss=0.008064, over 14597.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09798, pruned_loss=0.01783, audio_tagging_loss=0.009597, over 3043980.40 frames. ], batch size: 54, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 02:59:42,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1328913.3333333333, ans=0.0
2023-11-21 02:59:50,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1328980.0, ans=0.125
2023-11-21 02:59:51,114 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199350
2023-11-21 03:00:04,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1329046.6666666667, ans=0.125
2023-11-21 03:00:05,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1329046.6666666667, ans=0.125
2023-11-21 03:00:10,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1329046.6666666667, ans=0.1
2023-11-21 03:00:16,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1329113.3333333333, ans=0.2
2023-11-21 03:00:18,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1329113.3333333333, ans=0.125
2023-11-21 03:00:27,830 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7000, loss[loss=0.06968, simple_loss=0.08021, pruned_loss=0.01676, audio_tagging_loss=0.01282, over 14460.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09764, pruned_loss=0.01773, audio_tagging_loss=0.009699, over 3043323.09 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:00:45,668 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-21 03:00:54,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1329313.3333333333, ans=0.125
2023-11-21 03:00:55,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199400
2023-11-21 03:00:57,286 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.260e+01 8.901e+01 9.434e+01 1.193e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 03:01:02,025 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.73 vs. limit=10.0
2023-11-21 03:01:06,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1329380.0, ans=0.125
2023-11-21 03:01:32,979 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7050, loss[loss=0.07849, simple_loss=0.1093, pruned_loss=0.01457, audio_tagging_loss=0.009277, over 15229.00 frames. ], tot_loss[loss=0.07649, simple_loss=0.09796, pruned_loss=0.01776, audio_tagging_loss=0.00975, over 3048822.48 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:01:40,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1329513.3333333333, ans=0.0
2023-11-21 03:02:00,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199450
2023-11-21 03:02:06,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1329646.6666666667, ans=0.125
2023-11-21 03:02:07,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1329646.6666666667, ans=0.2
2023-11-21 03:02:30,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1329780.0, ans=0.125
2023-11-21 03:02:32,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1329780.0, ans=0.0
2023-11-21 03:02:36,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1329846.6666666667, ans=0.0
2023-11-21 03:02:38,024 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7100, loss[loss=0.08141, simple_loss=0.1091, pruned_loss=0.01957, audio_tagging_loss=0.007276, over 15217.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.09813, pruned_loss=0.01779, audio_tagging_loss=0.009781, over 3047602.46 frames. ], batch size: 57, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:03:02,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1329980.0, ans=0.125
2023-11-21 03:03:05,030 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199500
2023-11-21 03:03:06,066 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 7.897e+01 8.673e+01 9.642e+01 1.180e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 03:03:42,059 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7150, loss[loss=0.07212, simple_loss=0.09103, pruned_loss=0.01825, audio_tagging_loss=0.008347, over 16023.00 frames. ], tot_loss[loss=0.07696, simple_loss=0.09838, pruned_loss=0.01796, audio_tagging_loss=0.009816, over 3046165.93 frames. ], batch size: 62, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:03:43,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1330180.0, ans=0.125
2023-11-21 03:03:46,026 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:04:09,797 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199550
2023-11-21 03:04:26,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1330380.0, ans=0.2
2023-11-21 03:04:41,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1330446.6666666667, ans=0.1
2023-11-21 03:04:46,670 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7200, loss[loss=0.0881, simple_loss=0.1169, pruned_loss=0.02059, audio_tagging_loss=0.009044, over 15523.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09822, pruned_loss=0.01806, audio_tagging_loss=0.009938, over 3051510.50 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:05:13,758 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199600
2023-11-21 03:05:16,287 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 7.951e+01 8.767e+01 9.502e+01 1.229e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 03:05:21,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1330646.6666666667, ans=0.1
2023-11-21 03:05:21,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1330646.6666666667, ans=0.125
2023-11-21 03:05:32,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1330713.3333333333, ans=0.0
2023-11-21 03:05:48,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1330780.0, ans=0.0
2023-11-21 03:05:50,866 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7250, loss[loss=0.06637, simple_loss=0.076, pruned_loss=0.01538, audio_tagging_loss=0.01299, over 14235.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09849, pruned_loss=0.018, audio_tagging_loss=0.009898, over 3044217.14 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:05:55,299 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.00 vs. limit=22.5
2023-11-21 03:06:04,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1330913.3333333333, ans=0.125
2023-11-21 03:06:05,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1330913.3333333333, ans=0.2
2023-11-21 03:06:17,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199650
2023-11-21 03:06:22,134 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.56 vs. limit=6.0
2023-11-21 03:06:33,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.02 vs. limit=12.0
2023-11-21 03:06:34,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1331046.6666666667, ans=0.125
2023-11-21 03:06:54,324 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7300, loss[loss=0.07863, simple_loss=0.09247, pruned_loss=0.02066, audio_tagging_loss=0.01173, over 15117.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.09856, pruned_loss=0.01809, audio_tagging_loss=0.009818, over 3046733.28 frames. ], batch size: 55, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:07:15,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1331246.6666666667, ans=0.125
2023-11-21 03:07:17,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1331246.6666666667, ans=0.125
2023-11-21 03:07:20,998 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199700
2023-11-21 03:07:23,853 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.086e+01 8.011e+01 8.733e+01 9.607e+01 1.282e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 03:07:29,299 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.01 vs. limit=12.0
2023-11-21 03:07:31,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1331380.0, ans=0.1
2023-11-21 03:07:34,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.74 vs. limit=15.0
2023-11-21 03:07:42,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1331380.0, ans=0.0
2023-11-21 03:07:58,467 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7350, loss[loss=0.04689, simple_loss=0.05373, pruned_loss=0.008498, audio_tagging_loss=0.01152, over 14711.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09783, pruned_loss=0.01798, audio_tagging_loss=0.009661, over 3040414.36 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:08:24,786 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199750
2023-11-21 03:08:37,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1331713.3333333333, ans=0.1
2023-11-21 03:08:41,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1331713.3333333333, ans=0.125
2023-11-21 03:08:46,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.30 vs. limit=15.0
2023-11-21 03:08:46,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.47 vs. limit=15.0
2023-11-21 03:08:52,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1331780.0, ans=10.0
2023-11-21 03:08:53,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1331780.0, ans=0.125
2023-11-21 03:08:56,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1331780.0, ans=0.125
2023-11-21 03:09:01,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1331846.6666666667, ans=0.0
2023-11-21 03:09:02,783 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7400, loss[loss=0.08057, simple_loss=0.1117, pruned_loss=0.01878, audio_tagging_loss=0.005967, over 14296.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09729, pruned_loss=0.01766, audio_tagging_loss=0.009677, over 3033930.75 frames. ], batch size: 54, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:09:29,543 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199800
2023-11-21 03:09:32,472 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.394e+01 8.010e+01 8.731e+01 9.621e+01 1.537e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 03:09:38,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1331980.0, ans=0.125
2023-11-21 03:09:43,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.54 vs. limit=5.0
2023-11-21 03:09:44,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1332046.6666666667, ans=0.95
2023-11-21 03:09:50,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1332046.6666666667, ans=0.125
2023-11-21 03:10:06,593 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7450, loss[loss=0.09023, simple_loss=0.1158, pruned_loss=0.02192, audio_tagging_loss=0.0104, over 15181.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.0966, pruned_loss=0.01765, audio_tagging_loss=0.009696, over 3038652.68 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:10:08,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1332180.0, ans=0.0
2023-11-21 03:10:32,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1332313.3333333333, ans=0.125
2023-11-21 03:10:33,737 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199850
2023-11-21 03:10:41,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1332313.3333333333, ans=0.2
2023-11-21 03:10:41,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1332313.3333333333, ans=0.0
2023-11-21 03:10:47,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1332380.0, ans=0.125
2023-11-21 03:11:01,899 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.63 vs. limit=12.0
2023-11-21 03:11:10,948 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7500, loss[loss=0.08392, simple_loss=0.1152, pruned_loss=0.01961, audio_tagging_loss=0.006717, over 15751.00 frames. ], tot_loss[loss=0.07594, simple_loss=0.09713, pruned_loss=0.01773, audio_tagging_loss=0.009639, over 3038906.80 frames. ], batch size: 59, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:11:18,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1332513.3333333333, ans=0.125
2023-11-21 03:11:22,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1332580.0, ans=0.0
2023-11-21 03:11:34,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1332646.6666666667, ans=0.2
2023-11-21 03:11:37,138 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199900
2023-11-21 03:11:39,368 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.979e+01 8.350e+01 8.922e+01 9.540e+01 1.272e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 03:12:10,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1332780.0, ans=0.125
2023-11-21 03:12:14,026 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7550, loss[loss=0.07385, simple_loss=0.1025, pruned_loss=0.01512, audio_tagging_loss=0.007465, over 15690.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09726, pruned_loss=0.01782, audio_tagging_loss=0.009535, over 3045972.65 frames. ], batch size: 59, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:12:29,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1332913.3333333333, ans=0.125
2023-11-21 03:12:35,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0
2023-11-21 03:12:38,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.97 vs. limit=15.0
2023-11-21 03:12:40,267 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 199950
2023-11-21 03:12:54,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1333046.6666666667, ans=0.2
2023-11-21 03:13:05,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1333113.3333333333, ans=0.2
2023-11-21 03:13:17,620 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7600, loss[loss=0.05442, simple_loss=0.06748, pruned_loss=0.0111, audio_tagging_loss=0.009584, over 15088.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09688, pruned_loss=0.01767, audio_tagging_loss=0.009591, over 3045081.63 frames. ], batch size: 57, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:13:22,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1333180.0, ans=0.125
2023-11-21 03:13:23,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1333180.0, ans=0.125
2023-11-21 03:13:28,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1333246.6666666667, ans=0.1
2023-11-21 03:13:29,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.77 vs. limit=15.0
2023-11-21 03:13:39,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1333246.6666666667, ans=0.0
2023-11-21 03:13:45,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200000
2023-11-21 03:13:50,922 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.857e+01 8.060e+01 8.807e+01 9.387e+01 1.277e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 03:13:51,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1333313.3333333333, ans=0.125
2023-11-21 03:13:52,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.70 vs. limit=15.0
2023-11-21 03:13:53,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1333313.3333333333, ans=0.1
2023-11-21 03:14:01,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1333380.0, ans=0.1
2023-11-21 03:14:04,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1333380.0, ans=0.125
2023-11-21 03:14:25,262 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7650, loss[loss=0.07429, simple_loss=0.09261, pruned_loss=0.01753, audio_tagging_loss=0.01046, over 15411.00 frames. ], tot_loss[loss=0.07564, simple_loss=0.09641, pruned_loss=0.0177, audio_tagging_loss=0.00973, over 3047176.19 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:14:25,821 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.97 vs. limit=15.0
2023-11-21 03:14:37,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1333580.0, ans=0.035
2023-11-21 03:14:50,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1333646.6666666667, ans=0.125
2023-11-21 03:14:52,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200050
2023-11-21 03:15:18,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1333780.0, ans=0.0
2023-11-21 03:15:26,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1333780.0, ans=0.125
2023-11-21 03:15:29,948 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7700, loss[loss=0.08463, simple_loss=0.114, pruned_loss=0.01845, audio_tagging_loss=0.009185, over 15558.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09589, pruned_loss=0.0173, audio_tagging_loss=0.009734, over 3049778.23 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:15:42,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1333913.3333333333, ans=0.125
2023-11-21 03:15:43,581 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.49 vs. limit=15.0
2023-11-21 03:15:55,887 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200100
2023-11-21 03:15:58,170 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.520e+01 8.098e+01 8.698e+01 9.708e+01 1.361e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 03:16:23,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1334113.3333333333, ans=0.125
2023-11-21 03:16:26,101 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:16:33,107 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7750, loss[loss=0.08213, simple_loss=0.1056, pruned_loss=0.01897, audio_tagging_loss=0.01038, over 15882.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09674, pruned_loss=0.01751, audio_tagging_loss=0.009722, over 3052088.23 frames. ], batch size: 60, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:16:33,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1334180.0, ans=0.125
2023-11-21 03:16:34,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1334180.0, ans=0.125
2023-11-21 03:16:45,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1334246.6666666667, ans=0.125
2023-11-21 03:16:52,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1334246.6666666667, ans=0.1
2023-11-21 03:17:00,601 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200150
2023-11-21 03:17:02,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1334313.3333333333, ans=0.1
2023-11-21 03:17:25,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1334446.6666666667, ans=0.1
2023-11-21 03:17:25,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1334446.6666666667, ans=0.04949747468305833
2023-11-21 03:17:36,756 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7800, loss[loss=0.07, simple_loss=0.09526, pruned_loss=0.01639, audio_tagging_loss=0.005984, over 14517.00 frames. ], tot_loss[loss=0.07594, simple_loss=0.0975, pruned_loss=0.01753, audio_tagging_loss=0.009659, over 3044005.29 frames. ], batch size: 54, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:17:39,811 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.18 vs. limit=22.5
2023-11-21 03:18:04,895 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200200
2023-11-21 03:18:07,464 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.821e+01 7.982e+01 8.620e+01 9.218e+01 1.876e+02, threshold=1.724e+02, percent-clipped=1.0
2023-11-21 03:18:27,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1334780.0, ans=0.025
2023-11-21 03:18:28,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1334780.0, ans=0.0
2023-11-21 03:18:34,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1334780.0, ans=0.125
2023-11-21 03:18:37,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1334780.0, ans=0.125
2023-11-21 03:18:42,230 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7850, loss[loss=0.06072, simple_loss=0.07798, pruned_loss=0.01297, audio_tagging_loss=0.008761, over 14913.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.09771, pruned_loss=0.0175, audio_tagging_loss=0.009764, over 3051734.53 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:18:57,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1334913.3333333333, ans=0.0
2023-11-21 03:19:08,671 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200250
2023-11-21 03:19:14,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1334980.0, ans=0.1
2023-11-21 03:19:17,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.77 vs. limit=12.0
2023-11-21 03:19:23,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1335046.6666666667, ans=0.125
2023-11-21 03:19:30,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1335046.6666666667, ans=0.1
2023-11-21 03:19:37,349 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.42 vs. limit=15.0
2023-11-21 03:19:45,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1335180.0, ans=0.125
2023-11-21 03:19:46,654 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7900, loss[loss=0.08453, simple_loss=0.1041, pruned_loss=0.02231, audio_tagging_loss=0.01017, over 14851.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09951, pruned_loss=0.01811, audio_tagging_loss=0.009705, over 3052676.64 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:19:50,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1335180.0, ans=0.5
2023-11-21 03:19:53,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1335180.0, ans=0.125
2023-11-21 03:20:13,779 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200300
2023-11-21 03:20:16,603 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.124e+01 8.233e+01 9.036e+01 9.766e+01 1.372e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 03:20:20,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1335313.3333333333, ans=0.1
2023-11-21 03:20:43,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1335446.6666666667, ans=0.125
2023-11-21 03:20:49,704 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 7950, loss[loss=0.057, simple_loss=0.06597, pruned_loss=0.01369, audio_tagging_loss=0.01032, over 16532.00 frames. ], tot_loss[loss=0.07769, simple_loss=0.09947, pruned_loss=0.01816, audio_tagging_loss=0.009797, over 3050421.08 frames. ], batch size: 64, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:20:49,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1335513.3333333333, ans=0.125
2023-11-21 03:21:05,448 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:21:07,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1335580.0, ans=0.2
2023-11-21 03:21:17,871 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200350
2023-11-21 03:21:41,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1335780.0, ans=0.0
2023-11-21 03:21:53,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1335846.6666666667, ans=0.04949747468305833
2023-11-21 03:21:54,633 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8000, loss[loss=0.06003, simple_loss=0.07111, pruned_loss=0.01367, audio_tagging_loss=0.01081, over 14895.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09832, pruned_loss=0.01795, audio_tagging_loss=0.009965, over 3052932.81 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:21:59,221 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=8.716e-02
2023-11-21 03:22:10,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1335913.3333333333, ans=0.1
2023-11-21 03:22:18,538 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0
2023-11-21 03:22:20,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1335980.0, ans=0.0
2023-11-21 03:22:21,534 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200400
2023-11-21 03:22:22,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1335980.0, ans=0.2
2023-11-21 03:22:23,525 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.35 vs. limit=15.0
2023-11-21 03:22:24,191 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.612e+01 7.996e+01 8.659e+01 9.313e+01 1.449e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 03:22:31,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1336046.6666666667, ans=0.125
2023-11-21 03:22:59,634 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8050, loss[loss=0.07502, simple_loss=0.1001, pruned_loss=0.01557, audio_tagging_loss=0.009421, over 15404.00 frames. ], tot_loss[loss=0.07675, simple_loss=0.09795, pruned_loss=0.01773, audio_tagging_loss=0.01004, over 3060586.81 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:23:26,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200450
2023-11-21 03:23:32,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1336313.3333333333, ans=0.125
2023-11-21 03:23:49,560 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.92 vs. limit=12.0
2023-11-21 03:23:55,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1336446.6666666667, ans=0.1
2023-11-21 03:24:01,644 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:24:02,437 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8100, loss[loss=0.08656, simple_loss=0.1114, pruned_loss=0.02002, audio_tagging_loss=0.01084, over 14615.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09879, pruned_loss=0.01781, audio_tagging_loss=0.009918, over 3064806.23 frames. ], batch size: 54, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:24:05,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.27 vs. limit=15.0
2023-11-21 03:24:12,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1336513.3333333333, ans=0.05
2023-11-21 03:24:17,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1336580.0, ans=0.1
2023-11-21 03:24:20,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1336580.0, ans=0.125
2023-11-21 03:24:29,760 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200500
2023-11-21 03:24:32,115 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.135e+01 8.794e+01 9.348e+01 1.385e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 03:24:32,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1336646.6666666667, ans=0.2
2023-11-21 03:24:46,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1336713.3333333333, ans=0.0
2023-11-21 03:24:49,036 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.12 vs. limit=15.0
2023-11-21 03:24:53,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.18 vs. limit=15.0
2023-11-21 03:25:06,600 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8150, loss[loss=0.06138, simple_loss=0.07431, pruned_loss=0.01388, audio_tagging_loss=0.01035, over 15326.00 frames. ], tot_loss[loss=0.07695, simple_loss=0.09859, pruned_loss=0.01788, audio_tagging_loss=0.009781, over 3058828.81 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:25:18,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1336913.3333333333, ans=0.125
2023-11-21 03:25:20,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1336913.3333333333, ans=0.0
2023-11-21 03:25:23,327 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=22.5
2023-11-21 03:25:29,334 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.74 vs. limit=22.5
2023-11-21 03:25:33,564 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200550
2023-11-21 03:25:34,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1336980.0, ans=0.05
2023-11-21 03:25:44,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1337046.6666666667, ans=0.0
2023-11-21 03:26:11,136 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8200, loss[loss=0.08597, simple_loss=0.1116, pruned_loss=0.01883, audio_tagging_loss=0.01132, over 15767.00 frames. ], tot_loss[loss=0.0772, simple_loss=0.09923, pruned_loss=0.01802, audio_tagging_loss=0.009569, over 3057111.67 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:26:11,166 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:26:30,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0
2023-11-21 03:26:32,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1337246.6666666667, ans=0.0
2023-11-21 03:26:37,469 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200600
2023-11-21 03:26:40,663 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.735e+01 8.383e+01 9.269e+01 1.042e+02 1.722e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-21 03:27:02,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1337446.6666666667, ans=0.95
2023-11-21 03:27:13,280 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.54 vs. limit=15.0
2023-11-21 03:27:15,035 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8250, loss[loss=0.09416, simple_loss=0.1174, pruned_loss=0.02874, audio_tagging_loss=0.006713, over 15609.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09954, pruned_loss=0.01818, audio_tagging_loss=0.009434, over 3055844.74 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:27:25,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1337513.3333333333, ans=0.125
2023-11-21 03:27:28,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1337580.0, ans=0.125
2023-11-21 03:27:36,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.15 vs. limit=15.0
2023-11-21 03:27:38,635 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.67 vs. limit=15.0
2023-11-21 03:27:39,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1337646.6666666667, ans=0.125
2023-11-21 03:27:41,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200650
2023-11-21 03:27:46,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1337646.6666666667, ans=0.125
2023-11-21 03:27:55,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1337713.3333333333, ans=0.5
2023-11-21 03:28:17,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1337780.0, ans=0.125
2023-11-21 03:28:17,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1337780.0, ans=0.2
2023-11-21 03:28:17,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1337780.0, ans=0.025
2023-11-21 03:28:19,777 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8300, loss[loss=0.0872, simple_loss=0.1152, pruned_loss=0.01944, audio_tagging_loss=0.01019, over 15874.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09885, pruned_loss=0.01792, audio_tagging_loss=0.009498, over 3060045.91 frames. ], batch size: 60, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:28:25,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1337846.6666666667, ans=0.125
2023-11-21 03:28:46,690 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200700
2023-11-21 03:28:48,753 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.58 vs. limit=22.5
2023-11-21 03:28:50,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.519e+01 8.192e+01 8.783e+01 9.537e+01 1.149e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 03:29:17,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.42 vs. limit=15.0
2023-11-21 03:29:23,792 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8350, loss[loss=0.07621, simple_loss=0.103, pruned_loss=0.01557, audio_tagging_loss=0.009119, over 15474.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09843, pruned_loss=0.01762, audio_tagging_loss=0.00936, over 3057577.24 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:29:27,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1338180.0, ans=0.09899494936611666
2023-11-21 03:29:50,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200750
2023-11-21 03:29:51,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1338313.3333333333, ans=0.125
2023-11-21 03:29:56,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1338313.3333333333, ans=0.125
2023-11-21 03:30:05,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.14 vs. limit=22.5
2023-11-21 03:30:27,688 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8400, loss[loss=0.06762, simple_loss=0.08034, pruned_loss=0.01454, audio_tagging_loss=0.01291, over 16012.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09845, pruned_loss=0.0177, audio_tagging_loss=0.009486, over 3060804.65 frames. ], batch size: 62, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:30:32,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff2.min_abs, batch_count=1338513.3333333333, ans=0.1
2023-11-21 03:30:36,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1338513.3333333333, ans=0.0
2023-11-21 03:30:49,914 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.99 vs. limit=6.0
2023-11-21 03:30:54,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200800
2023-11-21 03:30:57,990 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.656e+01 8.229e+01 8.784e+01 9.362e+01 1.197e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 03:31:20,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1338780.0, ans=0.0
2023-11-21 03:31:30,724 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.01 vs. limit=15.0
2023-11-21 03:31:31,830 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8450, loss[loss=0.0736, simple_loss=0.09175, pruned_loss=0.01479, audio_tagging_loss=0.01293, over 14768.00 frames. ], tot_loss[loss=0.07682, simple_loss=0.09895, pruned_loss=0.01787, audio_tagging_loss=0.009469, over 3053158.48 frames. ], batch size: 57, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:31:42,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1338913.3333333333, ans=10.0
2023-11-21 03:31:58,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200850
2023-11-21 03:31:58,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1338980.0, ans=0.0
2023-11-21 03:32:14,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1339046.6666666667, ans=0.125
2023-11-21 03:32:32,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1339113.3333333333, ans=0.0
2023-11-21 03:32:35,646 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8500, loss[loss=0.06866, simple_loss=0.07943, pruned_loss=0.01525, audio_tagging_loss=0.0137, over 14232.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.0988, pruned_loss=0.01768, audio_tagging_loss=0.009684, over 3054194.94 frames. ], batch size: 56, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:32:43,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1339180.0, ans=0.0
2023-11-21 03:33:02,934 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200900
2023-11-21 03:33:06,648 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 7.935e+01 8.661e+01 9.459e+01 1.270e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 03:33:29,181 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.16 vs. limit=12.0
2023-11-21 03:33:38,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1339513.3333333333, ans=0.0
2023-11-21 03:33:39,580 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8550, loss[loss=0.07208, simple_loss=0.09571, pruned_loss=0.01673, audio_tagging_loss=0.007494, over 15497.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.09782, pruned_loss=0.0176, audio_tagging_loss=0.009706, over 3051265.00 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:34:06,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 200950
2023-11-21 03:34:37,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1339780.0, ans=0.2
2023-11-21 03:34:40,415 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.88 vs. limit=15.0
2023-11-21 03:34:43,517 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8600, loss[loss=0.0801, simple_loss=0.09505, pruned_loss=0.0199, audio_tagging_loss=0.01268, over 14940.00 frames. ], tot_loss[loss=0.07645, simple_loss=0.09822, pruned_loss=0.01758, audio_tagging_loss=0.009763, over 3053156.03 frames. ], batch size: 56, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:34:48,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1339846.6666666667, ans=0.0
2023-11-21 03:35:10,357 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201000
2023-11-21 03:35:14,245 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.482e+01 7.981e+01 8.707e+01 9.381e+01 1.149e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-21 03:35:16,930 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:35:17,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1339980.0, ans=0.1
2023-11-21 03:35:25,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.78 vs. limit=22.5
2023-11-21 03:35:47,420 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8650, loss[loss=0.07666, simple_loss=0.1065, pruned_loss=0.01625, audio_tagging_loss=0.007171, over 15630.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09915, pruned_loss=0.01767, audio_tagging_loss=0.009779, over 3058523.54 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:35:48,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1340180.0, ans=0.0
2023-11-21 03:36:12,883 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.66 vs. limit=22.5
2023-11-21 03:36:14,565 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201050
2023-11-21 03:36:20,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1340313.3333333333, ans=0.0
2023-11-21 03:36:41,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1340446.6666666667, ans=0.125
2023-11-21 03:36:45,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1340446.6666666667, ans=0.125
2023-11-21 03:36:49,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1340513.3333333333, ans=0.125
2023-11-21 03:36:50,935 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8700, loss[loss=0.09971, simple_loss=0.1424, pruned_loss=0.02149, audio_tagging_loss=0.007013, over 15847.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09873, pruned_loss=0.01749, audio_tagging_loss=0.009799, over 3053161.03 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:36:51,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1340513.3333333333, ans=0.2
2023-11-21 03:36:56,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1340513.3333333333, ans=0.125
2023-11-21 03:36:58,843 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.04 vs. limit=22.5
2023-11-21 03:37:10,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1340580.0, ans=0.125
2023-11-21 03:37:17,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201100
2023-11-21 03:37:20,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1340646.6666666667, ans=0.0
2023-11-21 03:37:22,699 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.240e+01 8.992e+01 1.008e+02 1.229e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-21 03:37:29,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1340713.3333333333, ans=0.125
2023-11-21 03:37:32,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1340713.3333333333, ans=0.0
2023-11-21 03:37:38,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1340713.3333333333, ans=0.125
2023-11-21 03:37:47,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1340780.0, ans=0.0
2023-11-21 03:37:54,263 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8750, loss[loss=0.08767, simple_loss=0.1134, pruned_loss=0.01972, audio_tagging_loss=0.01126, over 15725.00 frames. ], tot_loss[loss=0.0767, simple_loss=0.09843, pruned_loss=0.01764, audio_tagging_loss=0.00985, over 3057253.70 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:37:54,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1340846.6666666667, ans=0.125
2023-11-21 03:37:59,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1340846.6666666667, ans=0.2
2023-11-21 03:38:04,096 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.86 vs. limit=15.0
2023-11-21 03:38:04,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1340846.6666666667, ans=0.125
2023-11-21 03:38:21,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201150
2023-11-21 03:38:28,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1340980.0, ans=0.1
2023-11-21 03:38:35,462 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.31 vs. limit=22.5
2023-11-21 03:38:46,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1341113.3333333333, ans=0.0
2023-11-21 03:38:53,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1341113.3333333333, ans=0.2
2023-11-21 03:38:58,535 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8800, loss[loss=0.07473, simple_loss=0.08875, pruned_loss=0.01924, audio_tagging_loss=0.01112, over 14282.00 frames. ], tot_loss[loss=0.07735, simple_loss=0.09931, pruned_loss=0.01784, audio_tagging_loss=0.009865, over 3052808.14 frames. ], batch size: 54, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:38:58,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1341180.0, ans=0.025
2023-11-21 03:39:12,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1341246.6666666667, ans=0.125
2023-11-21 03:39:15,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=13.70 vs. limit=15.0
2023-11-21 03:39:21,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1341246.6666666667, ans=0.125
2023-11-21 03:39:24,702 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201200
2023-11-21 03:39:30,381 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.246e+01 8.266e+01 8.872e+01 9.572e+01 1.229e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 03:39:31,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1341313.3333333333, ans=0.035
2023-11-21 03:39:37,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1341380.0, ans=0.0
2023-11-21 03:39:40,647 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.53 vs. limit=12.0
2023-11-21 03:39:43,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1341380.0, ans=0.125
2023-11-21 03:39:54,545 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.33 vs. limit=15.0
2023-11-21 03:40:00,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1341446.6666666667, ans=0.125
2023-11-21 03:40:02,356 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8850, loss[loss=0.08839, simple_loss=0.1125, pruned_loss=0.02318, audio_tagging_loss=0.008934, over 15496.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09868, pruned_loss=0.01777, audio_tagging_loss=0.009894, over 3046035.35 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:40:13,259 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:40:29,547 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201250
2023-11-21 03:40:34,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1341646.6666666667, ans=0.125
2023-11-21 03:40:36,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1341646.6666666667, ans=0.125
2023-11-21 03:41:05,626 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8900, loss[loss=0.05364, simple_loss=0.06873, pruned_loss=0.009018, audio_tagging_loss=0.01026, over 15050.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09872, pruned_loss=0.0177, audio_tagging_loss=0.009789, over 3050089.16 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:41:33,474 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201300
2023-11-21 03:41:38,118 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.120e+01 8.967e+01 9.567e+01 1.214e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 03:41:42,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1341980.0, ans=0.125
2023-11-21 03:41:58,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.81 vs. limit=6.0
2023-11-21 03:42:03,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1342113.3333333333, ans=0.1
2023-11-21 03:42:11,243 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 8950, loss[loss=0.08302, simple_loss=0.09673, pruned_loss=0.02582, audio_tagging_loss=0.008844, over 14927.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.09828, pruned_loss=0.01772, audio_tagging_loss=0.009667, over 3042243.51 frames. ], batch size: 54, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:42:37,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201350
2023-11-21 03:43:14,483 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9000, loss[loss=0.07821, simple_loss=0.1097, pruned_loss=0.01541, audio_tagging_loss=0.007962, over 15363.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09864, pruned_loss=0.01775, audio_tagging_loss=0.00959, over 3049379.01 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:43:14,484 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 03:43:55,729 INFO [train_asr.py:1253] (2/4) Epoch 17, validation: loss=0.06143, simple_loss=0.05268, pruned_loss=0.005433, audio_tagging_loss=0.02966, over 4681554.00 frames. 
2023-11-21 03:43:55,730 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 03:44:23,548 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201400
2023-11-21 03:44:28,641 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.286e+01 9.090e+01 9.750e+01 1.340e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-21 03:44:37,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1342713.3333333333, ans=0.0
2023-11-21 03:44:51,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1342780.0, ans=0.1
2023-11-21 03:45:00,654 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9050, loss[loss=0.06095, simple_loss=0.07213, pruned_loss=0.01361, audio_tagging_loss=0.01128, over 14324.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09936, pruned_loss=0.01784, audio_tagging_loss=0.009521, over 3054082.07 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 16.0
2023-11-21 03:45:10,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1342846.6666666667, ans=0.125
2023-11-21 03:45:26,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201450
2023-11-21 03:45:27,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1342980.0, ans=0.125
2023-11-21 03:45:38,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1343046.6666666667, ans=0.5
2023-11-21 03:45:39,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1343046.6666666667, ans=0.5
2023-11-21 03:45:47,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1343046.6666666667, ans=0.1
2023-11-21 03:45:48,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.66 vs. limit=15.0
2023-11-21 03:46:04,615 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9100, loss[loss=0.07953, simple_loss=0.109, pruned_loss=0.01931, audio_tagging_loss=0.005745, over 15803.00 frames. ], tot_loss[loss=0.07647, simple_loss=0.09896, pruned_loss=0.01759, audio_tagging_loss=0.009396, over 3055094.47 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 16.0
2023-11-21 03:46:06,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1343180.0, ans=0.125
2023-11-21 03:46:18,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1343246.6666666667, ans=0.0
2023-11-21 03:46:28,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1343246.6666666667, ans=0.04949747468305833
2023-11-21 03:46:32,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201500
2023-11-21 03:46:38,215 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.193e+01 8.891e+01 9.703e+01 1.287e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 03:46:39,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1343313.3333333333, ans=0.0
2023-11-21 03:46:40,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1343313.3333333333, ans=0.125
2023-11-21 03:46:42,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1343380.0, ans=0.0
2023-11-21 03:47:08,810 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9150, loss[loss=0.07726, simple_loss=0.09721, pruned_loss=0.01947, audio_tagging_loss=0.00919, over 15068.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.09816, pruned_loss=0.01737, audio_tagging_loss=0.009453, over 3052171.46 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 16.0
2023-11-21 03:47:14,968 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.78 vs. limit=10.0
2023-11-21 03:47:19,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.62 vs. limit=22.5
2023-11-21 03:47:24,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1343580.0, ans=0.0
2023-11-21 03:47:28,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1343580.0, ans=0.1
2023-11-21 03:47:29,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=1343580.0, ans=0.025
2023-11-21 03:47:36,967 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201550
2023-11-21 03:47:50,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1343713.3333333333, ans=0.2
2023-11-21 03:47:51,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1343713.3333333333, ans=0.1
2023-11-21 03:47:58,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1343780.0, ans=0.125
2023-11-21 03:48:13,936 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9200, loss[loss=0.08527, simple_loss=0.1111, pruned_loss=0.02328, audio_tagging_loss=0.006444, over 15701.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.09762, pruned_loss=0.01732, audio_tagging_loss=0.009466, over 3052925.30 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:48:29,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1343913.3333333333, ans=0.2
2023-11-21 03:48:29,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1343913.3333333333, ans=0.125
2023-11-21 03:48:37,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1343913.3333333333, ans=0.2
2023-11-21 03:48:38,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1343980.0, ans=0.1
2023-11-21 03:48:40,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201600
2023-11-21 03:48:44,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1343980.0, ans=0.09899494936611666
2023-11-21 03:48:46,815 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.039e+01 8.659e+01 9.327e+01 1.552e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 03:49:10,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1344113.3333333333, ans=0.1
2023-11-21 03:49:18,530 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9250, loss[loss=0.07155, simple_loss=0.09104, pruned_loss=0.01363, audio_tagging_loss=0.0124, over 15689.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.09766, pruned_loss=0.01752, audio_tagging_loss=0.009545, over 3048762.56 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:49:26,426 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.96 vs. limit=12.0
2023-11-21 03:49:28,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1344180.0, ans=0.125
2023-11-21 03:49:45,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201650
2023-11-21 03:49:45,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1344313.3333333333, ans=0.0
2023-11-21 03:49:48,638 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.20 vs. limit=15.0
2023-11-21 03:50:04,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1344380.0, ans=0.5
2023-11-21 03:50:16,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1344446.6666666667, ans=0.125
2023-11-21 03:50:19,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1344446.6666666667, ans=0.125
2023-11-21 03:50:20,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1344446.6666666667, ans=0.05
2023-11-21 03:50:22,232 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9300, loss[loss=0.08232, simple_loss=0.1073, pruned_loss=0.01963, audio_tagging_loss=0.009021, over 15927.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09744, pruned_loss=0.0174, audio_tagging_loss=0.009562, over 3054582.80 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:50:23,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1344513.3333333333, ans=0.0
2023-11-21 03:50:50,184 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201700
2023-11-21 03:50:56,654 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 7.965e+01 8.483e+01 9.250e+01 1.325e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-21 03:51:27,503 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9350, loss[loss=0.09214, simple_loss=0.1173, pruned_loss=0.02545, audio_tagging_loss=0.008058, over 15639.00 frames. ], tot_loss[loss=0.07545, simple_loss=0.09664, pruned_loss=0.01741, audio_tagging_loss=0.009713, over 3042095.64 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:51:42,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1344913.3333333333, ans=0.125
2023-11-21 03:51:47,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1344913.3333333333, ans=0.0
2023-11-21 03:51:54,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201750
2023-11-21 03:52:00,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1344980.0, ans=0.0
2023-11-21 03:52:10,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1345046.6666666667, ans=0.125
2023-11-21 03:52:32,193 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9400, loss[loss=0.07843, simple_loss=0.1046, pruned_loss=0.01817, audio_tagging_loss=0.007985, over 15892.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09759, pruned_loss=0.0175, audio_tagging_loss=0.009803, over 3059538.23 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:52:58,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201800
2023-11-21 03:53:02,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1345313.3333333333, ans=0.035
2023-11-21 03:53:04,951 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.174e+01 8.028e+01 8.710e+01 9.500e+01 1.230e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 03:53:07,977 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.55 vs. limit=15.0
2023-11-21 03:53:11,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-21 03:53:13,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.73 vs. limit=15.0
2023-11-21 03:53:33,229 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:53:35,693 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9450, loss[loss=0.05263, simple_loss=0.07158, pruned_loss=0.009267, audio_tagging_loss=0.007576, over 14587.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09712, pruned_loss=0.0172, audio_tagging_loss=0.009913, over 3050380.52 frames. ], batch size: 55, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:54:03,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201850
2023-11-21 03:54:06,992 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.92 vs. limit=15.0
2023-11-21 03:54:10,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1345646.6666666667, ans=0.1
2023-11-21 03:54:40,218 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9500, loss[loss=0.106, simple_loss=0.1426, pruned_loss=0.02741, audio_tagging_loss=0.007259, over 15272.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09784, pruned_loss=0.01739, audio_tagging_loss=0.009958, over 3049305.09 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:55:06,958 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201900
2023-11-21 03:55:12,969 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.020e+01 8.201e+01 8.880e+01 9.737e+01 1.239e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 03:55:30,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1346113.3333333333, ans=0.125
2023-11-21 03:55:32,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1346113.3333333333, ans=0.125
2023-11-21 03:55:37,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1346113.3333333333, ans=0.0
2023-11-21 03:55:44,097 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9550, loss[loss=0.0695, simple_loss=0.08704, pruned_loss=0.01514, audio_tagging_loss=0.01085, over 15320.00 frames. ], tot_loss[loss=0.07686, simple_loss=0.09861, pruned_loss=0.01759, audio_tagging_loss=0.009966, over 3048959.27 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:55:59,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1346246.6666666667, ans=0.125
2023-11-21 03:56:00,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1346246.6666666667, ans=0.1
2023-11-21 03:56:10,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 201950
2023-11-21 03:56:48,225 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9600, loss[loss=0.08979, simple_loss=0.1209, pruned_loss=0.02155, audio_tagging_loss=0.007808, over 15248.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09805, pruned_loss=0.01763, audio_tagging_loss=0.01002, over 3051227.93 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:57:03,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1346580.0, ans=0.95
2023-11-21 03:57:15,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202000
2023-11-21 03:57:18,864 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.48 vs. limit=15.0
2023-11-21 03:57:21,643 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.303e+01 7.767e+01 8.568e+01 9.180e+01 1.180e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 03:57:30,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1346713.3333333333, ans=0.0
2023-11-21 03:57:46,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1346780.0, ans=0.125
2023-11-21 03:57:53,420 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9650, loss[loss=0.08834, simple_loss=0.1171, pruned_loss=0.02054, audio_tagging_loss=0.009262, over 16024.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.09863, pruned_loss=0.01773, audio_tagging_loss=0.009939, over 3047286.67 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:57:54,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1346846.6666666667, ans=0.2
2023-11-21 03:57:57,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1346846.6666666667, ans=0.125
2023-11-21 03:58:09,732 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.58 vs. limit=15.0
2023-11-21 03:58:19,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1346980.0, ans=0.125
2023-11-21 03:58:20,082 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202050
2023-11-21 03:58:32,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1347046.6666666667, ans=0.0
2023-11-21 03:58:42,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1347046.6666666667, ans=0.125
2023-11-21 03:58:44,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1347113.3333333333, ans=0.015
2023-11-21 03:58:50,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.35 vs. limit=15.0
2023-11-21 03:58:52,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.84 vs. limit=15.0
2023-11-21 03:58:53,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1347113.3333333333, ans=0.125
2023-11-21 03:58:57,057 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9700, loss[loss=0.09221, simple_loss=0.122, pruned_loss=0.02148, audio_tagging_loss=0.009728, over 16302.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09766, pruned_loss=0.01752, audio_tagging_loss=0.009796, over 3050247.81 frames. ], batch size: 59, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:59:12,769 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.27 vs. limit=15.0
2023-11-21 03:59:19,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1347246.6666666667, ans=0.125
2023-11-21 03:59:24,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202100
2023-11-21 03:59:31,476 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.071e+01 8.852e+01 9.589e+01 1.175e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 03:59:37,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1347380.0, ans=0.125
2023-11-21 03:59:51,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1347446.6666666667, ans=0.1
2023-11-21 04:00:01,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1347513.3333333333, ans=0.0
2023-11-21 04:00:01,936 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9750, loss[loss=0.06368, simple_loss=0.08099, pruned_loss=0.01278, audio_tagging_loss=0.01041, over 14986.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09746, pruned_loss=0.01747, audio_tagging_loss=0.009649, over 3048878.43 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 04:00:04,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1347513.3333333333, ans=0.125
2023-11-21 04:00:11,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1347513.3333333333, ans=0.125
2023-11-21 04:00:28,899 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202150
2023-11-21 04:01:05,912 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9800, loss[loss=0.1048, simple_loss=0.146, pruned_loss=0.02444, audio_tagging_loss=0.007365, over 16239.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.098, pruned_loss=0.01755, audio_tagging_loss=0.00959, over 3052595.20 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 04:01:14,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1347846.6666666667, ans=0.0
2023-11-21 04:01:19,444 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.46 vs. limit=15.0
2023-11-21 04:01:21,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1347913.3333333333, ans=0.1
2023-11-21 04:01:33,206 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202200
2023-11-21 04:01:40,218 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.143e+01 8.218e+01 8.911e+01 9.582e+01 1.351e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 04:02:01,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1348113.3333333333, ans=0.125
2023-11-21 04:02:02,313 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:02:10,987 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9850, loss[loss=0.08224, simple_loss=0.1065, pruned_loss=0.01996, audio_tagging_loss=0.009006, over 16074.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.0979, pruned_loss=0.01754, audio_tagging_loss=0.009624, over 3052103.00 frames. ], batch size: 59, lr: 4.00e-03, grad_scale: 32.0
2023-11-21 04:02:29,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.30 vs. limit=15.0
2023-11-21 04:02:38,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202250
2023-11-21 04:02:38,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1348313.3333333333, ans=0.0
2023-11-21 04:02:38,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1348313.3333333333, ans=0.035
2023-11-21 04:02:42,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.51 vs. limit=15.0
2023-11-21 04:02:52,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1348380.0, ans=0.1
2023-11-21 04:03:00,036 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.89 vs. limit=15.0
2023-11-21 04:03:05,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1348446.6666666667, ans=0.1
2023-11-21 04:03:16,231 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9900, loss[loss=0.06819, simple_loss=0.08692, pruned_loss=0.01624, audio_tagging_loss=0.008488, over 14477.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.09856, pruned_loss=0.01753, audio_tagging_loss=0.009629, over 3056701.23 frames. ], batch size: 55, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:03:23,834 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.65 vs. limit=22.5
2023-11-21 04:03:35,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.57 vs. limit=5.0
2023-11-21 04:03:44,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202300
2023-11-21 04:03:46,033 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.34 vs. limit=15.0
2023-11-21 04:03:51,362 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.722e+01 8.036e+01 8.846e+01 9.540e+01 1.221e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 04:04:03,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1348713.3333333333, ans=0.125
2023-11-21 04:04:17,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1348780.0, ans=0.09899494936611666
2023-11-21 04:04:21,072 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 9950, loss[loss=0.0878, simple_loss=0.1083, pruned_loss=0.02348, audio_tagging_loss=0.01016, over 15847.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09771, pruned_loss=0.0175, audio_tagging_loss=0.00966, over 3050649.42 frames. ], batch size: 59, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:04:27,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1348846.6666666667, ans=0.125
2023-11-21 04:04:48,400 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202350
2023-11-21 04:05:06,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1349046.6666666667, ans=0.05
2023-11-21 04:05:17,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.74 vs. limit=12.0
2023-11-21 04:05:19,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1349113.3333333333, ans=0.125
2023-11-21 04:05:25,835 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10000, loss[loss=0.07193, simple_loss=0.08639, pruned_loss=0.01892, audio_tagging_loss=0.009815, over 16554.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09675, pruned_loss=0.01739, audio_tagging_loss=0.009649, over 3045842.37 frames. ], batch size: 62, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:05:29,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1349180.0, ans=0.0
2023-11-21 04:05:32,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1349180.0, ans=0.125
2023-11-21 04:05:40,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1349246.6666666667, ans=0.125
2023-11-21 04:05:52,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202400
2023-11-21 04:05:52,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1349313.3333333333, ans=0.125
2023-11-21 04:06:02,541 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.043e+01 8.738e+01 9.707e+01 1.240e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 04:06:16,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1349446.6666666667, ans=0.2
2023-11-21 04:06:30,495 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10050, loss[loss=0.07678, simple_loss=0.09558, pruned_loss=0.01931, audio_tagging_loss=0.009684, over 14684.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.09611, pruned_loss=0.01727, audio_tagging_loss=0.009725, over 3048271.55 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:06:36,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1349513.3333333333, ans=0.125
2023-11-21 04:06:44,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1349580.0, ans=0.125
2023-11-21 04:06:58,158 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202450
2023-11-21 04:07:00,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1349646.6666666667, ans=0.2
2023-11-21 04:07:22,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1349780.0, ans=0.1
2023-11-21 04:07:34,521 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10100, loss[loss=0.06769, simple_loss=0.08246, pruned_loss=0.01432, audio_tagging_loss=0.01214, over 14861.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09673, pruned_loss=0.0173, audio_tagging_loss=0.009676, over 3054252.51 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:07:36,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1349846.6666666667, ans=0.0
2023-11-21 04:07:57,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=15.0
2023-11-21 04:08:02,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202500
2023-11-21 04:08:05,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1349980.0, ans=0.125
2023-11-21 04:08:11,193 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.354e+01 8.956e+01 9.533e+01 1.205e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 04:08:11,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1349980.0, ans=0.0
2023-11-21 04:08:24,719 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:08:39,846 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10150, loss[loss=0.06106, simple_loss=0.07621, pruned_loss=0.01209, audio_tagging_loss=0.01086, over 14428.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09643, pruned_loss=0.01723, audio_tagging_loss=0.009753, over 3051523.64 frames. ], batch size: 54, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:08:55,810 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=22.5
2023-11-21 04:09:02,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1350246.6666666667, ans=0.125
2023-11-21 04:09:06,478 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202550
2023-11-21 04:09:07,630 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:09:25,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1350380.0, ans=0.125
2023-11-21 04:09:44,010 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10200, loss[loss=0.05557, simple_loss=0.07101, pruned_loss=0.007771, audio_tagging_loss=0.0123, over 16216.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09643, pruned_loss=0.01717, audio_tagging_loss=0.009811, over 3053103.63 frames. ], batch size: 61, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:09:46,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1350513.3333333333, ans=0.2
2023-11-21 04:10:04,733 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:10:10,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202600
2023-11-21 04:10:15,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1350646.6666666667, ans=0.0
2023-11-21 04:10:19,591 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.093e+01 8.508e+01 9.534e+01 1.524e+02, threshold=1.702e+02, percent-clipped=0.0
2023-11-21 04:10:20,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1350646.6666666667, ans=0.0
2023-11-21 04:10:24,217 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.66 vs. limit=15.0
2023-11-21 04:10:47,479 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10250, loss[loss=0.06634, simple_loss=0.08111, pruned_loss=0.01629, audio_tagging_loss=0.009495, over 14000.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09688, pruned_loss=0.01741, audio_tagging_loss=0.009871, over 3050447.30 frames. ], batch size: 54, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:10:54,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1350846.6666666667, ans=0.125
2023-11-21 04:10:58,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1350846.6666666667, ans=0.09899494936611666
2023-11-21 04:10:58,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1350846.6666666667, ans=0.1
2023-11-21 04:11:08,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1350913.3333333333, ans=0.1
2023-11-21 04:11:14,735 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202650
2023-11-21 04:11:18,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1350980.0, ans=0.07
2023-11-21 04:11:22,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1350980.0, ans=0.0
2023-11-21 04:11:31,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1351046.6666666667, ans=0.05
2023-11-21 04:11:52,697 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10300, loss[loss=0.07742, simple_loss=0.103, pruned_loss=0.01518, audio_tagging_loss=0.01074, over 16422.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09756, pruned_loss=0.0176, audio_tagging_loss=0.009824, over 3052683.65 frames. ], batch size: 59, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:12:12,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1351246.6666666667, ans=0.1
2023-11-21 04:12:12,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1351246.6666666667, ans=0.09899494936611666
2023-11-21 04:12:17,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1351313.3333333333, ans=0.0
2023-11-21 04:12:19,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202700
2023-11-21 04:12:22,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1351313.3333333333, ans=0.125
2023-11-21 04:12:24,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1351313.3333333333, ans=0.125
2023-11-21 04:12:27,554 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.37 vs. limit=15.0
2023-11-21 04:12:27,754 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.129e+01 8.690e+01 9.404e+01 1.408e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 04:12:57,297 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10350, loss[loss=0.0817, simple_loss=0.09058, pruned_loss=0.02062, audio_tagging_loss=0.0158, over 16139.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09803, pruned_loss=0.0177, audio_tagging_loss=0.009977, over 3053487.84 frames. ], batch size: 61, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:13:11,288 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.79 vs. limit=22.5
2023-11-21 04:13:19,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1351580.0, ans=0.0
2023-11-21 04:13:19,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1351580.0, ans=0.125
2023-11-21 04:13:20,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1351646.6666666667, ans=0.125
2023-11-21 04:13:24,398 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202750
2023-11-21 04:13:25,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1351646.6666666667, ans=0.0
2023-11-21 04:13:40,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1351713.3333333333, ans=0.025
2023-11-21 04:14:01,494 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10400, loss[loss=0.07844, simple_loss=0.09781, pruned_loss=0.01731, audio_tagging_loss=0.01223, over 15093.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.0971, pruned_loss=0.01756, audio_tagging_loss=0.01001, over 3044088.66 frames. ], batch size: 57, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:14:03,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1351846.6666666667, ans=0.125
2023-11-21 04:14:05,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1351846.6666666667, ans=0.2
2023-11-21 04:14:18,114 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.36 vs. limit=6.0
2023-11-21 04:14:27,183 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.54 vs. limit=15.0
2023-11-21 04:14:29,121 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202800
2023-11-21 04:14:39,605 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.428e+01 7.927e+01 8.563e+01 9.335e+01 1.364e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 04:14:47,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1352046.6666666667, ans=0.125
2023-11-21 04:15:04,766 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.48 vs. limit=15.0
2023-11-21 04:15:06,478 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10450, loss[loss=0.06035, simple_loss=0.07662, pruned_loss=0.01158, audio_tagging_loss=0.01046, over 14914.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09731, pruned_loss=0.01757, audio_tagging_loss=0.009925, over 3053627.65 frames. ], batch size: 57, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:15:07,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.43 vs. limit=15.0
2023-11-21 04:15:29,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1352246.6666666667, ans=0.125
2023-11-21 04:15:33,449 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202850
2023-11-21 04:15:43,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1352380.0, ans=0.0
2023-11-21 04:16:10,739 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10500, loss[loss=0.09855, simple_loss=0.126, pruned_loss=0.02721, audio_tagging_loss=0.008354, over 15936.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09782, pruned_loss=0.01765, audio_tagging_loss=0.009808, over 3053829.36 frames. ], batch size: 59, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:16:20,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0
2023-11-21 04:16:22,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1352580.0, ans=0.0
2023-11-21 04:16:25,887 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.34 vs. limit=22.5
2023-11-21 04:16:37,773 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202900
2023-11-21 04:16:50,045 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.399e+01 8.004e+01 8.567e+01 9.229e+01 1.198e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 04:16:56,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1352713.3333333333, ans=0.0
2023-11-21 04:17:00,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=1352713.3333333333, ans=10.0
2023-11-21 04:17:04,365 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.43 vs. limit=10.0
2023-11-21 04:17:08,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1352780.0, ans=0.125
2023-11-21 04:17:16,033 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10550, loss[loss=0.07912, simple_loss=0.09943, pruned_loss=0.02122, audio_tagging_loss=0.008189, over 15864.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09758, pruned_loss=0.01752, audio_tagging_loss=0.009681, over 3058797.43 frames. ], batch size: 61, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:17:43,211 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 202950
2023-11-21 04:18:04,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1353046.6666666667, ans=0.0
2023-11-21 04:18:14,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1353113.3333333333, ans=0.1
2023-11-21 04:18:14,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1353113.3333333333, ans=0.0
2023-11-21 04:18:21,417 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10600, loss[loss=0.08695, simple_loss=0.1028, pruned_loss=0.02412, audio_tagging_loss=0.01143, over 17558.00 frames. ], tot_loss[loss=0.07622, simple_loss=0.09804, pruned_loss=0.01757, audio_tagging_loss=0.009638, over 3057326.97 frames. ], batch size: 67, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:18:22,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1353180.0, ans=0.125
2023-11-21 04:18:26,491 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:18:48,696 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203000
2023-11-21 04:18:54,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1353313.3333333333, ans=0.125
2023-11-21 04:18:58,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1353380.0, ans=0.125
2023-11-21 04:18:59,881 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.372e+01 8.133e+01 8.810e+01 9.778e+01 1.215e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 04:19:15,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1353446.6666666667, ans=0.125
2023-11-21 04:19:18,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1353446.6666666667, ans=0.125
2023-11-21 04:19:26,437 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10650, loss[loss=0.07103, simple_loss=0.09523, pruned_loss=0.01389, audio_tagging_loss=0.009532, over 16236.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.09793, pruned_loss=0.01763, audio_tagging_loss=0.009699, over 3057247.34 frames. ], batch size: 61, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:19:26,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1353513.3333333333, ans=0.125
2023-11-21 04:19:39,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1353580.0, ans=0.05
2023-11-21 04:19:40,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1353580.0, ans=0.125
2023-11-21 04:19:43,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1353580.0, ans=0.0
2023-11-21 04:19:44,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1353580.0, ans=0.125
2023-11-21 04:19:53,000 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203050
2023-11-21 04:20:19,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1353780.0, ans=0.2
2023-11-21 04:20:31,167 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10700, loss[loss=0.08011, simple_loss=0.09494, pruned_loss=0.02007, audio_tagging_loss=0.01257, over 16003.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09771, pruned_loss=0.0176, audio_tagging_loss=0.009684, over 3054543.87 frames. ], batch size: 60, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:20:46,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1353913.3333333333, ans=0.1
2023-11-21 04:20:58,603 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203100
2023-11-21 04:21:09,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.69 vs. limit=10.0
2023-11-21 04:21:10,479 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.882e+01 7.991e+01 8.733e+01 9.483e+01 1.206e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 04:21:20,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.22 vs. limit=10.0
2023-11-21 04:21:23,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1354113.3333333333, ans=0.0
2023-11-21 04:21:35,723 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10750, loss[loss=0.09914, simple_loss=0.1279, pruned_loss=0.02592, audio_tagging_loss=0.00925, over 15097.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.09659, pruned_loss=0.01727, audio_tagging_loss=0.009645, over 3047815.69 frames. ], batch size: 54, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:21:40,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1354180.0, ans=0.125
2023-11-21 04:21:55,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1354246.6666666667, ans=0.1
2023-11-21 04:22:03,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203150
2023-11-21 04:22:06,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1354313.3333333333, ans=0.1
2023-11-21 04:22:18,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1354380.0, ans=0.2
2023-11-21 04:22:36,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1354446.6666666667, ans=0.5
2023-11-21 04:22:40,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1354513.3333333333, ans=0.125
2023-11-21 04:22:41,288 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10800, loss[loss=0.07947, simple_loss=0.1017, pruned_loss=0.01674, audio_tagging_loss=0.01188, over 14813.00 frames. ], tot_loss[loss=0.07526, simple_loss=0.09705, pruned_loss=0.01716, audio_tagging_loss=0.009572, over 3054245.93 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:22:50,889 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:22:53,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1354513.3333333333, ans=0.0
2023-11-21 04:23:08,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1354646.6666666667, ans=0.125
2023-11-21 04:23:09,418 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203200
2023-11-21 04:23:12,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1354646.6666666667, ans=0.07
2023-11-21 04:23:21,928 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.183e+01 7.867e+01 8.488e+01 9.244e+01 1.126e+02, threshold=1.698e+02, percent-clipped=0.0
2023-11-21 04:23:24,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1354713.3333333333, ans=0.125
2023-11-21 04:23:24,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1354713.3333333333, ans=0.0
2023-11-21 04:23:47,602 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10850, loss[loss=0.06413, simple_loss=0.07643, pruned_loss=0.01243, audio_tagging_loss=0.01348, over 15145.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09674, pruned_loss=0.0172, audio_tagging_loss=0.009668, over 3054178.66 frames. ], batch size: 59, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:23:50,702 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.17 vs. limit=15.0
2023-11-21 04:24:02,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1354913.3333333333, ans=0.125
2023-11-21 04:24:13,615 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:24:13,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1354980.0, ans=0.0
2023-11-21 04:24:14,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203250
2023-11-21 04:24:33,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1355046.6666666667, ans=0.0
2023-11-21 04:24:45,940 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:24:52,229 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10900, loss[loss=0.06847, simple_loss=0.0812, pruned_loss=0.01741, audio_tagging_loss=0.01046, over 14913.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09672, pruned_loss=0.01728, audio_tagging_loss=0.009721, over 3051681.11 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:24:59,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1355180.0, ans=0.0
2023-11-21 04:25:04,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1355246.6666666667, ans=0.125
2023-11-21 04:25:13,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1355246.6666666667, ans=0.0
2023-11-21 04:25:20,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203300
2023-11-21 04:25:23,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.12 vs. limit=12.0
2023-11-21 04:25:31,849 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.236e+01 9.164e+01 1.009e+02 1.942e+02, threshold=1.833e+02, percent-clipped=1.0
2023-11-21 04:25:57,467 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 10950, loss[loss=0.07012, simple_loss=0.08639, pruned_loss=0.01782, audio_tagging_loss=0.009104, over 16814.00 frames. ], tot_loss[loss=0.07503, simple_loss=0.09615, pruned_loss=0.01719, audio_tagging_loss=0.009763, over 3050279.89 frames. ], batch size: 65, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:26:00,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1355513.3333333333, ans=0.125
2023-11-21 04:26:00,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1355513.3333333333, ans=0.125
2023-11-21 04:26:05,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1355513.3333333333, ans=0.125
2023-11-21 04:26:10,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1355580.0, ans=0.5
2023-11-21 04:26:14,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1355580.0, ans=0.0
2023-11-21 04:26:24,767 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203350
2023-11-21 04:26:43,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1355713.3333333333, ans=0.0
2023-11-21 04:26:47,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1355713.3333333333, ans=0.2
2023-11-21 04:26:47,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1355713.3333333333, ans=0.1
2023-11-21 04:27:02,261 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11000, loss[loss=0.07647, simple_loss=0.09667, pruned_loss=0.019, audio_tagging_loss=0.009126, over 16034.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.0956, pruned_loss=0.01715, audio_tagging_loss=0.009866, over 3051911.17 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:27:10,225 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:27:26,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.19 vs. limit=12.0
2023-11-21 04:27:29,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203400
2023-11-21 04:27:38,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1355980.0, ans=0.1
2023-11-21 04:27:40,974 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.079e+01 8.776e+01 9.579e+01 1.135e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 04:27:47,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1356046.6666666667, ans=0.125
2023-11-21 04:27:50,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1356046.6666666667, ans=0.5
2023-11-21 04:27:57,696 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=12.0
2023-11-21 04:28:01,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.37 vs. limit=22.5
2023-11-21 04:28:02,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.81 vs. limit=15.0
2023-11-21 04:28:06,820 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11050, loss[loss=0.06514, simple_loss=0.07824, pruned_loss=0.01616, audio_tagging_loss=0.009855, over 16999.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09585, pruned_loss=0.01735, audio_tagging_loss=0.009842, over 3048909.05 frames. ], batch size: 67, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:28:23,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1356246.6666666667, ans=0.0
2023-11-21 04:28:34,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203450
2023-11-21 04:29:11,779 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11100, loss[loss=0.05491, simple_loss=0.06526, pruned_loss=0.01174, audio_tagging_loss=0.01054, over 14375.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09628, pruned_loss=0.01749, audio_tagging_loss=0.009921, over 3049166.06 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:29:13,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1356513.3333333333, ans=0.0
2023-11-21 04:29:21,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1356513.3333333333, ans=0.2
2023-11-21 04:29:32,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1356580.0, ans=0.125
2023-11-21 04:29:32,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1356580.0, ans=0.125
2023-11-21 04:29:37,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.32 vs. limit=15.0
2023-11-21 04:29:39,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203500
2023-11-21 04:29:51,157 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.645e+01 8.437e+01 9.142e+01 9.871e+01 1.258e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-21 04:29:58,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1356713.3333333333, ans=0.125
2023-11-21 04:30:07,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1356780.0, ans=0.5
2023-11-21 04:30:10,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1356780.0, ans=0.125
2023-11-21 04:30:16,967 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11150, loss[loss=0.06334, simple_loss=0.07795, pruned_loss=0.01024, audio_tagging_loss=0.01412, over 15811.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09715, pruned_loss=0.01744, audio_tagging_loss=0.009892, over 3047527.76 frames. ], batch size: 60, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:30:39,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1356913.3333333333, ans=0.0
2023-11-21 04:30:39,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1356913.3333333333, ans=0.1
2023-11-21 04:30:45,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203550
2023-11-21 04:31:01,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1357046.6666666667, ans=0.0
2023-11-21 04:31:21,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1357180.0, ans=0.2
2023-11-21 04:31:23,071 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11200, loss[loss=0.09654, simple_loss=0.1264, pruned_loss=0.02277, audio_tagging_loss=0.01056, over 14903.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.09761, pruned_loss=0.01744, audio_tagging_loss=0.009939, over 3054571.37 frames. ], batch size: 54, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:31:35,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1357246.6666666667, ans=0.2
2023-11-21 04:31:38,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1357246.6666666667, ans=0.125
2023-11-21 04:31:48,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1357313.3333333333, ans=0.2
2023-11-21 04:31:50,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.04 vs. limit=15.0
2023-11-21 04:31:51,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203600
2023-11-21 04:32:02,364 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.264e+01 8.069e+01 8.575e+01 9.412e+01 1.593e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 04:32:03,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1357380.0, ans=0.0
2023-11-21 04:32:09,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1357380.0, ans=0.125
2023-11-21 04:32:29,573 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11250, loss[loss=0.07894, simple_loss=0.1061, pruned_loss=0.01883, audio_tagging_loss=0.007079, over 16401.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09706, pruned_loss=0.01749, audio_tagging_loss=0.009978, over 3050353.00 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:32:32,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1357513.3333333333, ans=0.125
2023-11-21 04:32:32,758 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.23 vs. limit=15.0
2023-11-21 04:32:40,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1357513.3333333333, ans=0.125
2023-11-21 04:32:42,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1357580.0, ans=0.125
2023-11-21 04:32:44,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1357580.0, ans=0.07
2023-11-21 04:32:56,325 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203650
2023-11-21 04:33:23,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1357780.0, ans=0.0
2023-11-21 04:33:35,044 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11300, loss[loss=0.06552, simple_loss=0.08356, pruned_loss=0.01399, audio_tagging_loss=0.009758, over 14961.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09718, pruned_loss=0.01749, audio_tagging_loss=0.009752, over 3047749.61 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:33:36,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1357846.6666666667, ans=0.0
2023-11-21 04:33:51,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1357913.3333333333, ans=0.125
2023-11-21 04:33:56,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1357913.3333333333, ans=0.125
2023-11-21 04:34:02,638 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203700
2023-11-21 04:34:03,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.79 vs. limit=15.0
2023-11-21 04:34:08,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1357980.0, ans=0.0
2023-11-21 04:34:13,647 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0
2023-11-21 04:34:14,171 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.294e+01 8.174e+01 8.923e+01 9.788e+01 1.326e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 04:34:27,524 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.13 vs. limit=15.0
2023-11-21 04:34:39,472 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11350, loss[loss=0.06195, simple_loss=0.08479, pruned_loss=0.01247, audio_tagging_loss=0.007088, over 15827.00 frames. ], tot_loss[loss=0.07589, simple_loss=0.09748, pruned_loss=0.01753, audio_tagging_loss=0.009617, over 3048493.30 frames. ], batch size: 64, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:34:43,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1358180.0, ans=0.125
2023-11-21 04:34:55,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1358246.6666666667, ans=0.05
2023-11-21 04:34:56,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1358246.6666666667, ans=0.125
2023-11-21 04:34:57,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1358246.6666666667, ans=0.2
2023-11-21 04:34:59,234 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.88 vs. limit=22.5
2023-11-21 04:35:01,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten.whitening_limit, batch_count=1358246.6666666667, ans=15.0
2023-11-21 04:35:07,511 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203750
2023-11-21 04:35:34,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1358446.6666666667, ans=0.125
2023-11-21 04:35:46,324 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11400, loss[loss=0.05778, simple_loss=0.06685, pruned_loss=0.01295, audio_tagging_loss=0.0114, over 14051.00 frames. ], tot_loss[loss=0.076, simple_loss=0.09747, pruned_loss=0.01761, audio_tagging_loss=0.009658, over 3045042.59 frames. ], batch size: 54, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:35:49,518 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.03 vs. limit=12.0
2023-11-21 04:36:13,245 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203800
2023-11-21 04:36:14,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1358646.6666666667, ans=0.125
2023-11-21 04:36:24,721 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.066e+01 8.848e+01 9.491e+01 1.619e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 04:36:38,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1358780.0, ans=0.0
2023-11-21 04:36:52,320 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11450, loss[loss=0.06552, simple_loss=0.08323, pruned_loss=0.01631, audio_tagging_loss=0.007592, over 15067.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09733, pruned_loss=0.0176, audio_tagging_loss=0.009562, over 3032717.50 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:36:54,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1358846.6666666667, ans=0.125
2023-11-21 04:37:18,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203850
2023-11-21 04:37:25,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1358980.0, ans=0.0
2023-11-21 04:37:54,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1359113.3333333333, ans=0.125
2023-11-21 04:37:56,309 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11500, loss[loss=0.07206, simple_loss=0.08582, pruned_loss=0.01974, audio_tagging_loss=0.009412, over 14886.00 frames. ], tot_loss[loss=0.07624, simple_loss=0.0978, pruned_loss=0.01768, audio_tagging_loss=0.00966, over 3040238.48 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:38:04,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1359180.0, ans=0.125
2023-11-21 04:38:06,395 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:38:23,967 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203900
2023-11-21 04:38:35,524 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.753e+01 8.131e+01 8.864e+01 9.970e+01 1.435e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 04:38:37,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1359380.0, ans=0.125
2023-11-21 04:38:43,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1359380.0, ans=0.2
2023-11-21 04:39:01,261 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11550, loss[loss=0.06324, simple_loss=0.07898, pruned_loss=0.01289, audio_tagging_loss=0.01085, over 15679.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09749, pruned_loss=0.01764, audio_tagging_loss=0.009647, over 3045558.84 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:39:01,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1359513.3333333333, ans=0.1
2023-11-21 04:39:28,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 203950
2023-11-21 04:39:37,818 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:39:40,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1359713.3333333333, ans=0.2
2023-11-21 04:40:01,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1359780.0, ans=0.09899494936611666
2023-11-21 04:40:05,807 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11600, loss[loss=0.07104, simple_loss=0.0871, pruned_loss=0.01457, audio_tagging_loss=0.01292, over 16059.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.09651, pruned_loss=0.01741, audio_tagging_loss=0.009728, over 3047441.17 frames. ], batch size: 61, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:40:09,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1359846.6666666667, ans=0.0
2023-11-21 04:40:30,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1359980.0, ans=0.0
2023-11-21 04:40:32,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204000
2023-11-21 04:40:51,364 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.095e+01 8.795e+01 9.597e+01 1.214e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 04:40:51,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1360046.6666666667, ans=0.125
2023-11-21 04:41:08,872 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.58 vs. limit=15.0
2023-11-21 04:41:10,235 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.94 vs. limit=15.0
2023-11-21 04:41:14,560 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11650, loss[loss=0.06895, simple_loss=0.08001, pruned_loss=0.01553, audio_tagging_loss=0.01341, over 15066.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.09676, pruned_loss=0.01737, audio_tagging_loss=0.009788, over 3045039.82 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:41:39,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.75 vs. limit=15.0
2023-11-21 04:41:42,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204050
2023-11-21 04:42:07,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1360446.6666666667, ans=0.1
2023-11-21 04:42:09,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1360446.6666666667, ans=0.125
2023-11-21 04:42:19,567 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11700, loss[loss=0.08624, simple_loss=0.1088, pruned_loss=0.02073, audio_tagging_loss=0.0111, over 14786.00 frames. ], tot_loss[loss=0.07547, simple_loss=0.09689, pruned_loss=0.01728, audio_tagging_loss=0.009746, over 3040657.57 frames. ], batch size: 55, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:42:28,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1360513.3333333333, ans=0.125
2023-11-21 04:42:33,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1360580.0, ans=0.04949747468305833
2023-11-21 04:42:47,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204100
2023-11-21 04:42:50,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.92 vs. limit=15.0
2023-11-21 04:43:00,707 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.797e+01 8.480e+01 9.195e+01 1.315e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-21 04:43:06,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1360713.3333333333, ans=0.125
2023-11-21 04:43:24,812 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11750, loss[loss=0.07422, simple_loss=0.09341, pruned_loss=0.0173, audio_tagging_loss=0.01021, over 15142.00 frames. ], tot_loss[loss=0.07598, simple_loss=0.09741, pruned_loss=0.01752, audio_tagging_loss=0.009753, over 3038840.86 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:43:40,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1360913.3333333333, ans=0.0
2023-11-21 04:43:45,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1360913.3333333333, ans=0.2
2023-11-21 04:43:51,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204150
2023-11-21 04:44:10,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1361046.6666666667, ans=0.5
2023-11-21 04:44:14,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1361046.6666666667, ans=0.125
2023-11-21 04:44:29,360 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11800, loss[loss=0.05244, simple_loss=0.0712, pruned_loss=0.008646, audio_tagging_loss=0.008194, over 14352.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09762, pruned_loss=0.01761, audio_tagging_loss=0.009744, over 3038892.43 frames. ], batch size: 55, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:44:56,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204200
2023-11-21 04:45:07,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1361313.3333333333, ans=0.125
2023-11-21 04:45:11,502 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.252e+01 9.002e+01 9.752e+01 1.410e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 04:45:16,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1361380.0, ans=0.0
2023-11-21 04:45:33,896 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11850, loss[loss=0.05617, simple_loss=0.07067, pruned_loss=0.00862, audio_tagging_loss=0.01222, over 15758.00 frames. ], tot_loss[loss=0.07584, simple_loss=0.09715, pruned_loss=0.01734, audio_tagging_loss=0.009922, over 3039094.54 frames. ], batch size: 59, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:45:35,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1361513.3333333333, ans=0.2
2023-11-21 04:45:38,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1361513.3333333333, ans=0.0
2023-11-21 04:45:44,659 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.06 vs. limit=15.0
2023-11-21 04:45:45,804 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.78 vs. limit=15.0
2023-11-21 04:45:49,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1361580.0, ans=0.125
2023-11-21 04:46:02,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204250
2023-11-21 04:46:06,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1361646.6666666667, ans=0.09899494936611666
2023-11-21 04:46:12,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1361713.3333333333, ans=0.1
2023-11-21 04:46:32,559 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.06 vs. limit=15.0
2023-11-21 04:46:38,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1361846.6666666667, ans=0.1
2023-11-21 04:46:39,220 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11900, loss[loss=0.0745, simple_loss=0.09694, pruned_loss=0.01705, audio_tagging_loss=0.008983, over 14663.00 frames. ], tot_loss[loss=0.07636, simple_loss=0.09781, pruned_loss=0.01751, audio_tagging_loss=0.009941, over 3044808.86 frames. ], batch size: 56, lr: 3.98e-03, grad_scale: 16.0
2023-11-21 04:46:45,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1361846.6666666667, ans=0.125
2023-11-21 04:46:49,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1361846.6666666667, ans=0.125
2023-11-21 04:46:53,251 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.08 vs. limit=15.0
2023-11-21 04:47:06,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204300
2023-11-21 04:47:07,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1361980.0, ans=0.0
2023-11-21 04:47:11,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1361980.0, ans=0.1
2023-11-21 04:47:11,981 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.45 vs. limit=10.0
2023-11-21 04:47:20,908 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.041e+01 8.628e+01 9.158e+01 1.228e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 04:47:22,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1362046.6666666667, ans=0.125
2023-11-21 04:47:35,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1362113.3333333333, ans=0.125
2023-11-21 04:47:44,169 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 11950, loss[loss=0.06888, simple_loss=0.08361, pruned_loss=0.01691, audio_tagging_loss=0.01017, over 15104.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09802, pruned_loss=0.01765, audio_tagging_loss=0.009998, over 3040437.33 frames. ], batch size: 62, lr: 3.98e-03, grad_scale: 16.0
2023-11-21 04:47:46,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1362180.0, ans=0.0
2023-11-21 04:48:10,679 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204350
2023-11-21 04:48:13,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.22 vs. limit=15.0
2023-11-21 04:48:23,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1362380.0, ans=0.2
2023-11-21 04:48:24,282 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.69 vs. limit=15.0
2023-11-21 04:48:44,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1362513.3333333333, ans=0.2
2023-11-21 04:48:45,538 INFO [train_asr.py:1221] (2/4) Epoch 17, batch 12000, loss[loss=0.06888, simple_loss=0.09822, pruned_loss=0.01238, audio_tagging_loss=0.007394, over 15314.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.09809, pruned_loss=0.01782, audio_tagging_loss=0.01007, over 3041267.61 frames. ], batch size: 58, lr: 3.98e-03, grad_scale: 32.0
2023-11-21 04:48:45,538 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 04:49:09,136 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7596, 5.3393, 4.9860, 5.4636], device='cuda:2')
2023-11-21 04:49:16,241 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.1646, 2.3727, 5.0555, 2.5692], device='cuda:2')
2023-11-21 04:49:23,008 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.1981, 3.1970, 2.8711, 2.7660, 3.4378, 3.5397, 3.1710, 3.6538],
       device='cuda:2')
2023-11-21 04:49:26,101 INFO [train_asr.py:1253] (2/4) Epoch 17, validation: loss=0.06069, simple_loss=0.05267, pruned_loss=0.005387, audio_tagging_loss=0.02896, over 4681554.00 frames. 
2023-11-21 04:49:26,102 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 04:49:30,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1362513.3333333333, ans=0.2
2023-11-21 04:49:41,656 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:49:51,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204400
2023-11-21 04:50:33,265 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 0, loss[loss=0.09216, simple_loss=0.09716, pruned_loss=0.01985, audio_tagging_loss=0.02374, over 15226.00 frames. ], tot_loss[loss=0.09216, simple_loss=0.09716, pruned_loss=0.01985, audio_tagging_loss=0.02374, over 15226.00 frames. ], batch size: 58, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:50:33,266 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 04:50:53,805 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7645, 5.7581, 5.8497, 5.8597], device='cuda:2')
2023-11-21 04:51:08,571 INFO [train_asr.py:1253] (2/4) Epoch 18, validation: loss=0.05959, simple_loss=0.05266, pruned_loss=0.005405, audio_tagging_loss=0.02786, over 4681554.00 frames. 
2023-11-21 04:51:08,572 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 04:51:19,302 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.498e+01 8.070e+01 8.803e+01 9.675e+01 1.246e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 04:51:34,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.67 vs. limit=15.0
2023-11-21 04:51:38,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1362800.0, ans=0.2
2023-11-21 04:51:51,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1362866.6666666667, ans=0.1
2023-11-21 04:52:09,443 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204450
2023-11-21 04:52:11,801 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 50, loss[loss=0.07813, simple_loss=0.09769, pruned_loss=0.01168, audio_tagging_loss=0.0176, over 15241.00 frames. ], tot_loss[loss=0.085, simple_loss=0.09858, pruned_loss=0.0175, audio_tagging_loss=0.01821, over 689531.43 frames. ], batch size: 56, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:52:29,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1363066.6666666667, ans=0.0
2023-11-21 04:52:54,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1363200.0, ans=0.125
2023-11-21 04:52:56,457 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:53:06,036 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:53:13,393 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204500
2023-11-21 04:53:15,773 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 100, loss[loss=0.08917, simple_loss=0.108, pruned_loss=0.0187, audio_tagging_loss=0.01648, over 15265.00 frames. ], tot_loss[loss=0.08477, simple_loss=0.09898, pruned_loss=0.01734, audio_tagging_loss=0.01794, over 1214155.55 frames. ], batch size: 57, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:53:28,005 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.544e+01 9.245e+01 1.008e+02 1.490e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-21 04:53:35,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1363400.0, ans=0.125
2023-11-21 04:53:39,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1363400.0, ans=0.125
2023-11-21 04:54:18,650 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204550
2023-11-21 04:54:21,078 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 150, loss[loss=0.1028, simple_loss=0.129, pruned_loss=0.0253, audio_tagging_loss=0.01298, over 15130.00 frames. ], tot_loss[loss=0.08193, simple_loss=0.09745, pruned_loss=0.01709, audio_tagging_loss=0.01612, over 1617158.17 frames. ], batch size: 54, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:54:27,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.31 vs. limit=15.0
2023-11-21 04:54:44,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1363733.3333333333, ans=0.0
2023-11-21 04:54:44,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1363733.3333333333, ans=0.125
2023-11-21 04:54:53,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1363800.0, ans=0.125
2023-11-21 04:54:59,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1363866.6666666667, ans=0.2
2023-11-21 04:55:07,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1363866.6666666667, ans=0.0
2023-11-21 04:55:07,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.34 vs. limit=6.0
2023-11-21 04:55:23,536 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204600
2023-11-21 04:55:26,220 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 200, loss[loss=0.07177, simple_loss=0.08408, pruned_loss=0.01666, audio_tagging_loss=0.01307, over 14979.00 frames. ], tot_loss[loss=0.08043, simple_loss=0.09784, pruned_loss=0.01727, audio_tagging_loss=0.01424, over 1935460.23 frames. ], batch size: 60, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:55:29,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1364000.0, ans=0.125
2023-11-21 04:55:37,063 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.028e+01 8.648e+01 9.287e+01 1.143e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 04:56:26,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204650
2023-11-21 04:56:29,188 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 250, loss[loss=0.06408, simple_loss=0.07765, pruned_loss=0.01493, audio_tagging_loss=0.01033, over 15322.00 frames. ], tot_loss[loss=0.07957, simple_loss=0.09854, pruned_loss=0.01747, audio_tagging_loss=0.01284, over 2179394.68 frames. ], batch size: 56, lr: 3.87e-03, grad_scale: 16.0
2023-11-21 04:56:36,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1364333.3333333333, ans=0.0
2023-11-21 04:56:51,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1364400.0, ans=0.05
2023-11-21 04:56:59,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1364466.6666666667, ans=0.05
2023-11-21 04:57:09,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1364533.3333333333, ans=0.1
2023-11-21 04:57:10,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1364533.3333333333, ans=0.0
2023-11-21 04:57:12,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1364533.3333333333, ans=15.0
2023-11-21 04:57:17,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.55 vs. limit=15.0
2023-11-21 04:57:23,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1364600.0, ans=0.125
2023-11-21 04:57:31,696 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204700
2023-11-21 04:57:34,635 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 300, loss[loss=0.0588, simple_loss=0.0763, pruned_loss=0.01244, audio_tagging_loss=0.008205, over 14268.00 frames. ], tot_loss[loss=0.07868, simple_loss=0.09858, pruned_loss=0.01749, audio_tagging_loss=0.0119, over 2371389.59 frames. ], batch size: 54, lr: 3.87e-03, grad_scale: 16.0
2023-11-21 04:57:45,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1364733.3333333333, ans=0.125
2023-11-21 04:57:46,743 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.596e+01 8.107e+01 8.931e+01 9.351e+01 1.204e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 04:57:47,296 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.40 vs. limit=15.0
2023-11-21 04:57:47,406 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.69 vs. limit=22.5
2023-11-21 04:58:06,174 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.26 vs. limit=15.0
2023-11-21 04:58:28,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1364933.3333333333, ans=0.125
2023-11-21 04:58:35,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204750
2023-11-21 04:58:37,522 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 350, loss[loss=0.06553, simple_loss=0.09361, pruned_loss=0.01257, audio_tagging_loss=0.006155, over 14522.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09846, pruned_loss=0.01739, audio_tagging_loss=0.01121, over 2525000.32 frames. ], batch size: 53, lr: 3.87e-03, grad_scale: 16.0
2023-11-21 04:59:11,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1365133.3333333333, ans=0.125
2023-11-21 04:59:13,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1365133.3333333333, ans=0.0
2023-11-21 04:59:19,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1365200.0, ans=0.2
2023-11-21 04:59:22,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1365200.0, ans=0.2
2023-11-21 04:59:32,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1365266.6666666667, ans=0.125
2023-11-21 04:59:39,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204800
2023-11-21 04:59:42,261 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 400, loss[loss=0.08091, simple_loss=0.1007, pruned_loss=0.02025, audio_tagging_loss=0.0103, over 15175.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09851, pruned_loss=0.01733, audio_tagging_loss=0.01071, over 2644383.05 frames. ], batch size: 56, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:59:51,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1365333.3333333333, ans=0.2
2023-11-21 04:59:55,643 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.688e+01 8.042e+01 8.695e+01 9.547e+01 1.167e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 05:00:44,998 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204850
2023-11-21 05:00:46,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1365666.6666666667, ans=0.0
2023-11-21 05:00:47,474 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 450, loss[loss=0.07768, simple_loss=0.1013, pruned_loss=0.02138, audio_tagging_loss=0.005669, over 14462.00 frames. ], tot_loss[loss=0.07659, simple_loss=0.09796, pruned_loss=0.0172, audio_tagging_loss=0.01041, over 2730500.22 frames. ], batch size: 55, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 05:00:48,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1365666.6666666667, ans=0.2
2023-11-21 05:00:58,868 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.98 vs. limit=15.0
2023-11-21 05:01:02,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.39 vs. limit=22.5
2023-11-21 05:01:04,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.34 vs. limit=6.0
2023-11-21 05:01:27,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1365866.6666666667, ans=0.04949747468305833
2023-11-21 05:01:50,489 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204900
2023-11-21 05:01:52,777 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 500, loss[loss=0.0852, simple_loss=0.1138, pruned_loss=0.01898, audio_tagging_loss=0.009321, over 15693.00 frames. ], tot_loss[loss=0.07645, simple_loss=0.09799, pruned_loss=0.01722, audio_tagging_loss=0.01023, over 2795403.38 frames. ], batch size: 61, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:01:54,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1366000.0, ans=0.125
2023-11-21 05:02:01,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1366000.0, ans=0.125
2023-11-21 05:02:07,166 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.144e+01 8.900e+01 9.756e+01 1.828e+02, threshold=1.780e+02, percent-clipped=1.0
2023-11-21 05:02:08,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1366066.6666666667, ans=0.125
2023-11-21 05:02:11,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1366066.6666666667, ans=0.2
2023-11-21 05:02:17,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1366133.3333333333, ans=0.125
2023-11-21 05:02:31,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1366200.0, ans=0.125
2023-11-21 05:02:52,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1366266.6666666667, ans=0.0
2023-11-21 05:02:55,371 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 204950
2023-11-21 05:02:56,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1366333.3333333333, ans=0.125
2023-11-21 05:02:57,708 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 550, loss[loss=0.09288, simple_loss=0.1284, pruned_loss=0.02219, audio_tagging_loss=0.006462, over 15488.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.0984, pruned_loss=0.01744, audio_tagging_loss=0.01002, over 2846164.48 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:03:02,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.15 vs. limit=15.0
2023-11-21 05:03:24,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1366466.6666666667, ans=0.125
2023-11-21 05:03:31,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1366466.6666666667, ans=0.035
2023-11-21 05:03:59,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205000
2023-11-21 05:04:02,503 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 600, loss[loss=0.06003, simple_loss=0.08104, pruned_loss=0.008415, audio_tagging_loss=0.0111, over 14184.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09739, pruned_loss=0.01721, audio_tagging_loss=0.009968, over 2893602.36 frames. ], batch size: 53, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:04:02,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1366666.6666666667, ans=0.125
2023-11-21 05:04:08,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0
2023-11-21 05:04:16,851 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.071e+01 8.678e+01 9.459e+01 1.222e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 05:04:47,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1366866.6666666667, ans=0.125
2023-11-21 05:04:51,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1366866.6666666667, ans=0.125
2023-11-21 05:04:59,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1366933.3333333333, ans=0.2
2023-11-21 05:05:05,025 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205050
2023-11-21 05:05:07,391 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 650, loss[loss=0.07378, simple_loss=0.1084, pruned_loss=0.01381, audio_tagging_loss=0.005796, over 16672.00 frames. ], tot_loss[loss=0.07608, simple_loss=0.09736, pruned_loss=0.01737, audio_tagging_loss=0.01003, over 2924615.93 frames. ], batch size: 61, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:05:13,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1367000.0, ans=0.125
2023-11-21 05:05:16,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1367000.0, ans=0.125
2023-11-21 05:05:21,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1367066.6666666667, ans=0.125
2023-11-21 05:05:23,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1367066.6666666667, ans=0.0
2023-11-21 05:05:34,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1367133.3333333333, ans=0.125
2023-11-21 05:05:51,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1367200.0, ans=0.0
2023-11-21 05:05:56,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1367200.0, ans=0.1
2023-11-21 05:06:07,001 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=12.0
2023-11-21 05:06:10,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205100
2023-11-21 05:06:11,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1367333.3333333333, ans=0.125
2023-11-21 05:06:12,501 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 700, loss[loss=0.1087, simple_loss=0.1391, pruned_loss=0.03129, audio_tagging_loss=0.007885, over 14321.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09743, pruned_loss=0.01741, audio_tagging_loss=0.009972, over 2954610.83 frames. ], batch size: 52, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:06:26,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1367400.0, ans=0.125
2023-11-21 05:06:27,560 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.155e+01 8.827e+01 9.700e+01 1.144e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 05:06:30,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.16 vs. limit=22.5
2023-11-21 05:06:53,144 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.26 vs. limit=15.0
2023-11-21 05:06:55,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1367533.3333333333, ans=0.125
2023-11-21 05:07:01,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1367533.3333333333, ans=0.0
2023-11-21 05:07:16,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205150
2023-11-21 05:07:19,639 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 750, loss[loss=0.07407, simple_loss=0.09351, pruned_loss=0.01403, audio_tagging_loss=0.01329, over 14451.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09759, pruned_loss=0.01742, audio_tagging_loss=0.009958, over 2975398.87 frames. ], batch size: 53, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:07:57,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1367866.6666666667, ans=0.1
2023-11-21 05:08:09,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1367866.6666666667, ans=0.125
2023-11-21 05:08:16,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1367933.3333333333, ans=0.1
2023-11-21 05:08:22,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205200
2023-11-21 05:08:25,681 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 800, loss[loss=0.07763, simple_loss=0.1025, pruned_loss=0.01529, audio_tagging_loss=0.01111, over 14852.00 frames. ], tot_loss[loss=0.07688, simple_loss=0.09847, pruned_loss=0.01766, audio_tagging_loss=0.009987, over 2994197.12 frames. ], batch size: 55, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:08:31,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1368000.0, ans=0.2
2023-11-21 05:08:39,575 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.688e+01 8.168e+01 8.753e+01 9.823e+01 1.363e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 05:09:03,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.31 vs. limit=15.0
2023-11-21 05:09:27,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1368266.6666666667, ans=0.125
2023-11-21 05:09:28,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205250
2023-11-21 05:09:30,515 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 850, loss[loss=0.07996, simple_loss=0.1027, pruned_loss=0.01649, audio_tagging_loss=0.0121, over 15416.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09875, pruned_loss=0.01779, audio_tagging_loss=0.01004, over 2997645.98 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:09:37,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1368333.3333333333, ans=0.1
2023-11-21 05:09:55,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1368466.6666666667, ans=0.2
2023-11-21 05:09:57,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.78 vs. limit=22.5
2023-11-21 05:09:59,780 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:10:12,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1368533.3333333333, ans=0.1
2023-11-21 05:10:20,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1368600.0, ans=0.125
2023-11-21 05:10:21,077 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.75 vs. limit=15.0
2023-11-21 05:10:22,094 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:10:22,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1368600.0, ans=0.2
2023-11-21 05:10:32,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205300
2023-11-21 05:10:35,151 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 900, loss[loss=0.09495, simple_loss=0.1337, pruned_loss=0.01908, audio_tagging_loss=0.009049, over 15299.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09752, pruned_loss=0.0175, audio_tagging_loss=0.01014, over 3012091.49 frames. ], batch size: 53, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:10:50,731 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.117e+01 8.694e+01 9.614e+01 1.237e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 05:10:56,372 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.43 vs. limit=6.0
2023-11-21 05:11:22,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1368866.6666666667, ans=0.05
2023-11-21 05:11:36,333 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.16 vs. limit=15.0
2023-11-21 05:11:37,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1368933.3333333333, ans=0.04949747468305833
2023-11-21 05:11:39,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205350
2023-11-21 05:11:41,823 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 950, loss[loss=0.05211, simple_loss=0.05939, pruned_loss=0.01214, audio_tagging_loss=0.01028, over 15411.00 frames. ], tot_loss[loss=0.07646, simple_loss=0.09769, pruned_loss=0.01747, audio_tagging_loss=0.01014, over 3013221.50 frames. ], batch size: 59, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:11:42,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1369000.0, ans=0.0
2023-11-21 05:11:45,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1369000.0, ans=0.07
2023-11-21 05:11:48,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1369000.0, ans=0.125
2023-11-21 05:12:35,661 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.92 vs. limit=15.0
2023-11-21 05:12:43,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205400
2023-11-21 05:12:46,464 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1000, loss[loss=0.07752, simple_loss=0.1023, pruned_loss=0.01593, audio_tagging_loss=0.01043, over 16583.00 frames. ], tot_loss[loss=0.07575, simple_loss=0.09683, pruned_loss=0.01743, audio_tagging_loss=0.009905, over 3014967.69 frames. ], batch size: 60, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:12:52,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=1369333.3333333333, ans=10.0
2023-11-21 05:12:56,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1369333.3333333333, ans=0.0
2023-11-21 05:12:59,226 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:13:01,197 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.289e+01 9.122e+01 9.845e+01 1.226e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-21 05:13:07,972 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.36 vs. limit=15.0
2023-11-21 05:13:14,387 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:13:26,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1369533.3333333333, ans=0.125
2023-11-21 05:13:27,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1369533.3333333333, ans=0.0
2023-11-21 05:13:42,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.50 vs. limit=15.0
2023-11-21 05:13:48,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205450
2023-11-21 05:13:50,687 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1050, loss[loss=0.08458, simple_loss=0.1183, pruned_loss=0.01742, audio_tagging_loss=0.008022, over 14913.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09632, pruned_loss=0.01717, audio_tagging_loss=0.009734, over 3024936.94 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:13:55,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1369666.6666666667, ans=0.1
2023-11-21 05:14:18,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1369800.0, ans=0.1
2023-11-21 05:14:31,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.65 vs. limit=15.0
2023-11-21 05:14:43,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1369933.3333333333, ans=0.125
2023-11-21 05:14:55,365 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205500
2023-11-21 05:14:57,784 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1100, loss[loss=0.07764, simple_loss=0.1037, pruned_loss=0.01586, audio_tagging_loss=0.009952, over 15125.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09696, pruned_loss=0.01728, audio_tagging_loss=0.009656, over 3031824.58 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:15:01,453 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:15:12,507 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 8.447e+01 9.123e+01 9.824e+01 1.304e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-21 05:15:12,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1370066.6666666667, ans=0.0
2023-11-21 05:15:14,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1370066.6666666667, ans=0.125
2023-11-21 05:15:20,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1370066.6666666667, ans=0.125
2023-11-21 05:15:20,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.31 vs. limit=22.5
2023-11-21 05:15:52,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1370266.6666666667, ans=0.125
2023-11-21 05:15:58,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1370266.6666666667, ans=0.0
2023-11-21 05:15:59,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205550
2023-11-21 05:16:02,097 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1150, loss[loss=0.06555, simple_loss=0.08456, pruned_loss=0.01057, audio_tagging_loss=0.0127, over 15930.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09667, pruned_loss=0.01718, audio_tagging_loss=0.009604, over 3032744.25 frames. ], batch size: 60, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:16:05,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.26 vs. limit=6.0
2023-11-21 05:16:54,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1370600.0, ans=0.0
2023-11-21 05:17:04,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205600
2023-11-21 05:17:07,016 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1200, loss[loss=0.07267, simple_loss=0.08874, pruned_loss=0.01858, audio_tagging_loss=0.009715, over 14980.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09686, pruned_loss=0.01743, audio_tagging_loss=0.009592, over 3031903.70 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:17:23,715 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.058e+01 8.835e+01 9.727e+01 1.276e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 05:17:45,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1370866.6666666667, ans=0.0
2023-11-21 05:17:50,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1370866.6666666667, ans=0.1
2023-11-21 05:18:02,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1370933.3333333333, ans=0.2
2023-11-21 05:18:10,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205650
2023-11-21 05:18:12,817 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1250, loss[loss=0.07929, simple_loss=0.1008, pruned_loss=0.01726, audio_tagging_loss=0.01165, over 14370.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.09669, pruned_loss=0.01733, audio_tagging_loss=0.00953, over 3030363.36 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:18:21,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.49 vs. limit=15.0
2023-11-21 05:18:21,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1371000.0, ans=0.1
2023-11-21 05:18:21,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1371000.0, ans=0.0
2023-11-21 05:18:22,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1371000.0, ans=22.5
2023-11-21 05:18:35,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1371066.6666666667, ans=0.1
2023-11-21 05:18:59,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1371200.0, ans=0.0
2023-11-21 05:19:05,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.00 vs. limit=15.0
2023-11-21 05:19:16,747 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205700
2023-11-21 05:19:19,154 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1300, loss[loss=0.06185, simple_loss=0.08503, pruned_loss=0.01005, audio_tagging_loss=0.00928, over 13650.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.09762, pruned_loss=0.01751, audio_tagging_loss=0.009481, over 3027758.14 frames. ], batch size: 55, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:19:19,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1371333.3333333333, ans=0.0
2023-11-21 05:19:23,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1371333.3333333333, ans=0.125
2023-11-21 05:19:27,318 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.32 vs. limit=15.0
2023-11-21 05:19:33,940 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.156e+01 8.683e+01 9.710e+01 1.341e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 05:19:35,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1371400.0, ans=0.125
2023-11-21 05:19:42,166 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:20:09,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1371533.3333333333, ans=0.125
2023-11-21 05:20:21,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205750
2023-11-21 05:20:24,020 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1350, loss[loss=0.07585, simple_loss=0.09376, pruned_loss=0.01858, audio_tagging_loss=0.01038, over 15027.00 frames. ], tot_loss[loss=0.07556, simple_loss=0.09745, pruned_loss=0.01729, audio_tagging_loss=0.00954, over 3031584.24 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:20:33,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1371666.6666666667, ans=0.0
2023-11-21 05:20:43,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1371733.3333333333, ans=0.125
2023-11-21 05:20:47,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1371733.3333333333, ans=0.04949747468305833
2023-11-21 05:21:05,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1371866.6666666667, ans=0.2
2023-11-21 05:21:11,030 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:21:26,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205800
2023-11-21 05:21:29,786 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1400, loss[loss=0.08169, simple_loss=0.1104, pruned_loss=0.0164, audio_tagging_loss=0.01009, over 15216.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.09785, pruned_loss=0.01755, audio_tagging_loss=0.009615, over 3034064.69 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:21:45,818 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.942e+01 8.095e+01 8.796e+01 9.675e+01 1.234e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 05:22:32,392 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205850
2023-11-21 05:22:35,541 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1450, loss[loss=0.08748, simple_loss=0.1168, pruned_loss=0.02134, audio_tagging_loss=0.00772, over 14847.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.09813, pruned_loss=0.01749, audio_tagging_loss=0.009702, over 3041757.69 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:22:55,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1372400.0, ans=0.1
2023-11-21 05:23:09,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1372466.6666666667, ans=0.1
2023-11-21 05:23:15,288 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:23:19,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1372533.3333333333, ans=0.125
2023-11-21 05:23:36,891 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.10 vs. limit=15.0
2023-11-21 05:23:37,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205900
2023-11-21 05:23:40,179 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1500, loss[loss=0.0737, simple_loss=0.09002, pruned_loss=0.01533, audio_tagging_loss=0.01336, over 15251.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09858, pruned_loss=0.01759, audio_tagging_loss=0.009794, over 3039965.03 frames. ], batch size: 58, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:23:53,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1372733.3333333333, ans=0.125
2023-11-21 05:23:55,897 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 8.193e+01 8.878e+01 9.448e+01 1.243e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 05:23:58,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1372733.3333333333, ans=0.09899494936611666
2023-11-21 05:24:00,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1372733.3333333333, ans=0.125
2023-11-21 05:24:08,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1372800.0, ans=0.025
2023-11-21 05:24:14,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1372800.0, ans=0.0
2023-11-21 05:24:18,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1372866.6666666667, ans=0.1
2023-11-21 05:24:20,023 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:24:26,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1372866.6666666667, ans=0.125
2023-11-21 05:24:28,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1372866.6666666667, ans=0.0
2023-11-21 05:24:43,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 205950
2023-11-21 05:24:45,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1373000.0, ans=0.125
2023-11-21 05:24:45,769 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1550, loss[loss=0.08664, simple_loss=0.109, pruned_loss=0.02461, audio_tagging_loss=0.007518, over 15486.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09952, pruned_loss=0.01775, audio_tagging_loss=0.009805, over 3044095.37 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:24:59,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1373066.6666666667, ans=0.0
2023-11-21 05:25:00,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1373066.6666666667, ans=0.125
2023-11-21 05:25:15,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1373133.3333333333, ans=0.0
2023-11-21 05:25:31,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1373200.0, ans=0.0
2023-11-21 05:25:35,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1373200.0, ans=0.1
2023-11-21 05:25:35,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1373200.0, ans=0.0
2023-11-21 05:25:36,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1373266.6666666667, ans=0.0
2023-11-21 05:25:37,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1373266.6666666667, ans=0.125
2023-11-21 05:25:46,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1373266.6666666667, ans=0.125
2023-11-21 05:25:48,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206000
2023-11-21 05:25:51,192 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1600, loss[loss=0.0705, simple_loss=0.09383, pruned_loss=0.01348, audio_tagging_loss=0.01011, over 14317.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09859, pruned_loss=0.0176, audio_tagging_loss=0.009912, over 3038002.79 frames. ], batch size: 53, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:25:54,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1373333.3333333333, ans=0.125
2023-11-21 05:26:01,731 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.52 vs. limit=22.5
2023-11-21 05:26:07,352 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.284e+01 8.965e+01 9.783e+01 1.317e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 05:26:29,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1373466.6666666667, ans=0.125
2023-11-21 05:26:32,528 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.96 vs. limit=15.0
2023-11-21 05:26:35,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1373533.3333333333, ans=0.125
2023-11-21 05:26:42,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1373533.3333333333, ans=0.05
2023-11-21 05:26:52,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1373600.0, ans=0.015
2023-11-21 05:26:54,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1373600.0, ans=0.1
2023-11-21 05:26:55,247 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206050
2023-11-21 05:26:57,584 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1650, loss[loss=0.09463, simple_loss=0.1232, pruned_loss=0.02303, audio_tagging_loss=0.01, over 15930.00 frames. ], tot_loss[loss=0.07638, simple_loss=0.09835, pruned_loss=0.01735, audio_tagging_loss=0.009858, over 3044555.36 frames. ], batch size: 58, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:26:57,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1373666.6666666667, ans=0.0
2023-11-21 05:27:22,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1373733.3333333333, ans=22.5
2023-11-21 05:27:22,116 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.80 vs. limit=22.5
2023-11-21 05:27:38,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1373866.6666666667, ans=0.2
2023-11-21 05:28:00,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206100
2023-11-21 05:28:03,616 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1700, loss[loss=0.09368, simple_loss=0.1228, pruned_loss=0.02145, audio_tagging_loss=0.01081, over 16006.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.0982, pruned_loss=0.01745, audio_tagging_loss=0.009771, over 3046368.45 frames. ], batch size: 60, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:28:03,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1374000.0, ans=0.0
2023-11-21 05:28:18,889 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.074e+01 8.560e+01 9.664e+01 1.224e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 05:28:22,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1374066.6666666667, ans=0.125
2023-11-21 05:28:30,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1374133.3333333333, ans=0.125
2023-11-21 05:28:31,746 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.67 vs. limit=22.5
2023-11-21 05:28:40,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1374200.0, ans=0.035
2023-11-21 05:28:59,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1374266.6666666667, ans=0.0
2023-11-21 05:29:02,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1374266.6666666667, ans=0.125
2023-11-21 05:29:06,278 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206150
2023-11-21 05:29:08,603 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1750, loss[loss=0.06822, simple_loss=0.08751, pruned_loss=0.01402, audio_tagging_loss=0.01045, over 15856.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09785, pruned_loss=0.01748, audio_tagging_loss=0.009751, over 3045313.43 frames. ], batch size: 62, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:29:17,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1374333.3333333333, ans=0.0
2023-11-21 05:29:24,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.87 vs. limit=22.5
2023-11-21 05:29:50,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1374533.3333333333, ans=0.125
2023-11-21 05:29:51,463 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.89 vs. limit=15.0
2023-11-21 05:30:03,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1374600.0, ans=0.0
2023-11-21 05:30:05,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1374600.0, ans=0.125
2023-11-21 05:30:09,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1374600.0, ans=0.5
2023-11-21 05:30:10,792 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206200
2023-11-21 05:30:13,607 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1800, loss[loss=0.07651, simple_loss=0.109, pruned_loss=0.01495, audio_tagging_loss=0.007045, over 16363.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.097, pruned_loss=0.01722, audio_tagging_loss=0.009824, over 3045632.87 frames. ], batch size: 61, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:30:29,694 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.350e+01 8.891e+01 9.779e+01 1.328e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 05:30:44,191 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.96 vs. limit=10.0
2023-11-21 05:30:46,380 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.85 vs. limit=15.0
2023-11-21 05:30:50,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1374800.0, ans=0.125
2023-11-21 05:31:00,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1374866.6666666667, ans=0.125
2023-11-21 05:31:01,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1374866.6666666667, ans=0.125
2023-11-21 05:31:04,850 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.83 vs. limit=15.0
2023-11-21 05:31:11,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1374933.3333333333, ans=0.0
2023-11-21 05:31:16,131 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206250
2023-11-21 05:31:16,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1374933.3333333333, ans=0.0
2023-11-21 05:31:18,491 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1850, loss[loss=0.07371, simple_loss=0.09465, pruned_loss=0.01693, audio_tagging_loss=0.009455, over 14939.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09672, pruned_loss=0.0172, audio_tagging_loss=0.009733, over 3037359.29 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:31:32,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1375066.6666666667, ans=0.04949747468305833
2023-11-21 05:31:33,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1375066.6666666667, ans=0.125
2023-11-21 05:31:34,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1375066.6666666667, ans=0.125
2023-11-21 05:31:40,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1375066.6666666667, ans=0.0
2023-11-21 05:32:02,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.95 vs. limit=10.0
2023-11-21 05:32:19,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1375266.6666666667, ans=0.125
2023-11-21 05:32:22,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206300
2023-11-21 05:32:24,918 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1900, loss[loss=0.07743, simple_loss=0.1021, pruned_loss=0.01588, audio_tagging_loss=0.01052, over 16790.00 frames. ], tot_loss[loss=0.075, simple_loss=0.0962, pruned_loss=0.01712, audio_tagging_loss=0.009778, over 3040305.51 frames. ], batch size: 62, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:32:37,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1375400.0, ans=0.125
2023-11-21 05:32:40,736 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.312e+01 7.949e+01 8.733e+01 9.558e+01 1.328e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 05:32:40,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1375400.0, ans=0.0
2023-11-21 05:32:43,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1375400.0, ans=0.1
2023-11-21 05:33:03,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1375533.3333333333, ans=0.125
2023-11-21 05:33:09,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1375533.3333333333, ans=0.125
2023-11-21 05:33:14,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.06 vs. limit=15.0
2023-11-21 05:33:14,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.16 vs. limit=12.0
2023-11-21 05:33:27,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206350
2023-11-21 05:33:29,454 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 1950, loss[loss=0.07969, simple_loss=0.1144, pruned_loss=0.01715, audio_tagging_loss=0.005349, over 15849.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.096, pruned_loss=0.01696, audio_tagging_loss=0.009702, over 3037297.96 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:33:29,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1375666.6666666667, ans=0.125
2023-11-21 05:33:33,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1375666.6666666667, ans=0.1
2023-11-21 05:33:55,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.35 vs. limit=15.0
2023-11-21 05:33:59,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1375800.0, ans=0.125
2023-11-21 05:34:06,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1375800.0, ans=0.0
2023-11-21 05:34:11,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1375866.6666666667, ans=0.125
2023-11-21 05:34:27,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1375933.3333333333, ans=0.125
2023-11-21 05:34:28,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1375933.3333333333, ans=0.0
2023-11-21 05:34:30,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206400
2023-11-21 05:34:34,499 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2000, loss[loss=0.0721, simple_loss=0.09575, pruned_loss=0.01434, audio_tagging_loss=0.009883, over 14957.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09486, pruned_loss=0.01675, audio_tagging_loss=0.009706, over 3033033.06 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:34:53,478 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.010e+01 8.880e+01 9.739e+01 1.562e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 05:35:05,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1376133.3333333333, ans=0.0
2023-11-21 05:35:37,810 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206450
2023-11-21 05:35:40,838 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2050, loss[loss=0.07161, simple_loss=0.09844, pruned_loss=0.01408, audio_tagging_loss=0.00831, over 14761.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09638, pruned_loss=0.01718, audio_tagging_loss=0.009676, over 3036163.77 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:35:42,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1376333.3333333333, ans=0.125
2023-11-21 05:35:45,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1376333.3333333333, ans=0.025
2023-11-21 05:35:58,284 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:35:58,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1376400.0, ans=0.025
2023-11-21 05:36:00,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1376400.0, ans=0.0
2023-11-21 05:36:13,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1376466.6666666667, ans=0.125
2023-11-21 05:36:22,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=22.5
2023-11-21 05:36:42,066 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206500
2023-11-21 05:36:44,380 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2100, loss[loss=0.0874, simple_loss=0.1204, pruned_loss=0.01933, audio_tagging_loss=0.007875, over 14859.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09601, pruned_loss=0.01717, audio_tagging_loss=0.009619, over 3039188.86 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:36:44,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1376666.6666666667, ans=0.0
2023-11-21 05:37:01,369 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 8.008e+01 8.892e+01 9.733e+01 1.759e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 05:37:21,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1376866.6666666667, ans=0.125
2023-11-21 05:37:21,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1376866.6666666667, ans=0.1
2023-11-21 05:37:27,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1376866.6666666667, ans=0.2
2023-11-21 05:37:30,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1376866.6666666667, ans=0.125
2023-11-21 05:37:35,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1376933.3333333333, ans=0.125
2023-11-21 05:37:44,467 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206550
2023-11-21 05:37:46,815 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2150, loss[loss=0.08552, simple_loss=0.09999, pruned_loss=0.02565, audio_tagging_loss=0.009882, over 14650.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09686, pruned_loss=0.01741, audio_tagging_loss=0.009568, over 3044463.52 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:37:48,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1377000.0, ans=0.125
2023-11-21 05:38:13,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1377133.3333333333, ans=0.125
2023-11-21 05:38:13,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1377133.3333333333, ans=0.07
2023-11-21 05:38:18,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1377133.3333333333, ans=0.125
2023-11-21 05:38:24,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1377200.0, ans=0.125
2023-11-21 05:38:26,404 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:38:48,819 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206600
2023-11-21 05:38:51,629 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2200, loss[loss=0.08822, simple_loss=0.1052, pruned_loss=0.02281, audio_tagging_loss=0.01282, over 15296.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.0971, pruned_loss=0.01748, audio_tagging_loss=0.009554, over 3038828.48 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:39:09,738 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.508e+01 7.959e+01 8.463e+01 9.337e+01 1.181e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-21 05:39:23,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1377466.6666666667, ans=0.125
2023-11-21 05:39:24,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1377466.6666666667, ans=0.125
2023-11-21 05:39:34,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1377533.3333333333, ans=0.125
2023-11-21 05:39:50,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1377600.0, ans=0.125
2023-11-21 05:39:50,134 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.40 vs. limit=22.5
2023-11-21 05:39:50,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.55 vs. limit=15.0
2023-11-21 05:39:54,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206650
2023-11-21 05:39:57,074 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2250, loss[loss=0.05998, simple_loss=0.0724, pruned_loss=0.01326, audio_tagging_loss=0.01052, over 13768.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.09714, pruned_loss=0.01742, audio_tagging_loss=0.009612, over 3036811.59 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:40:11,271 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.67 vs. limit=15.0
2023-11-21 05:40:13,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1377733.3333333333, ans=0.2
2023-11-21 05:40:15,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1377733.3333333333, ans=0.0
2023-11-21 05:40:23,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1377800.0, ans=0.125
2023-11-21 05:40:43,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1377866.6666666667, ans=0.0
2023-11-21 05:40:49,842 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.26 vs. limit=15.0
2023-11-21 05:40:59,043 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206700
2023-11-21 05:41:01,592 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2300, loss[loss=0.05017, simple_loss=0.06255, pruned_loss=0.008153, audio_tagging_loss=0.01074, over 14202.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09732, pruned_loss=0.01761, audio_tagging_loss=0.009668, over 3033892.04 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:41:09,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1378000.0, ans=0.1
2023-11-21 05:41:20,807 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.169e+01 8.944e+01 9.700e+01 1.275e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 05:41:24,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.75 vs. limit=15.0
2023-11-21 05:41:34,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1378133.3333333333, ans=0.0
2023-11-21 05:41:55,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1378266.6666666667, ans=0.0
2023-11-21 05:42:00,220 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:42:04,564 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206750
2023-11-21 05:42:07,017 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2350, loss[loss=0.08405, simple_loss=0.1192, pruned_loss=0.01701, audio_tagging_loss=0.007435, over 15264.00 frames. ], tot_loss[loss=0.07595, simple_loss=0.09761, pruned_loss=0.01745, audio_tagging_loss=0.009696, over 3035808.20 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:42:10,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.41 vs. limit=12.0
2023-11-21 05:42:42,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1378466.6666666667, ans=0.0
2023-11-21 05:42:57,798 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.44 vs. limit=15.0
2023-11-21 05:43:09,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206800
2023-11-21 05:43:09,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1378600.0, ans=0.125
2023-11-21 05:43:12,378 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2400, loss[loss=0.06128, simple_loss=0.07657, pruned_loss=0.01141, audio_tagging_loss=0.01158, over 15742.00 frames. ], tot_loss[loss=0.07596, simple_loss=0.09747, pruned_loss=0.01747, audio_tagging_loss=0.009758, over 3042866.37 frames. ], batch size: 60, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:43:13,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1378666.6666666667, ans=0.1
2023-11-21 05:43:26,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1378733.3333333333, ans=0.125
2023-11-21 05:43:28,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1378733.3333333333, ans=0.0
2023-11-21 05:43:29,495 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.194e+01 8.214e+01 8.717e+01 9.423e+01 1.179e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 05:43:48,288 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.72 vs. limit=15.0
2023-11-21 05:43:55,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1378866.6666666667, ans=0.125
2023-11-21 05:43:56,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1378866.6666666667, ans=0.0
2023-11-21 05:44:02,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1378933.3333333333, ans=0.07
2023-11-21 05:44:06,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1378933.3333333333, ans=0.0
2023-11-21 05:44:13,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206850
2023-11-21 05:44:15,710 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2450, loss[loss=0.08739, simple_loss=0.1162, pruned_loss=0.02078, audio_tagging_loss=0.008496, over 16470.00 frames. ], tot_loss[loss=0.07661, simple_loss=0.09816, pruned_loss=0.01776, audio_tagging_loss=0.009772, over 3040751.54 frames. ], batch size: 59, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:44:41,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1379133.3333333333, ans=0.025
2023-11-21 05:44:51,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1379133.3333333333, ans=0.125
2023-11-21 05:45:16,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206900
2023-11-21 05:45:19,796 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2500, loss[loss=0.05867, simple_loss=0.07513, pruned_loss=0.0121, audio_tagging_loss=0.009013, over 15109.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09807, pruned_loss=0.01763, audio_tagging_loss=0.009874, over 3053641.89 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:45:33,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1379400.0, ans=0.125
2023-11-21 05:45:38,422 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.201e+01 8.897e+01 9.730e+01 1.405e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 05:45:39,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1379400.0, ans=0.04949747468305833
2023-11-21 05:45:43,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1379400.0, ans=0.2
2023-11-21 05:45:45,016 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.02 vs. limit=15.0
2023-11-21 05:45:49,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1379466.6666666667, ans=0.0
2023-11-21 05:45:49,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1379466.6666666667, ans=0.1
2023-11-21 05:45:53,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1379466.6666666667, ans=0.0
2023-11-21 05:46:22,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 206950
2023-11-21 05:46:25,212 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2550, loss[loss=0.08872, simple_loss=0.1174, pruned_loss=0.02323, audio_tagging_loss=0.006772, over 14482.00 frames. ], tot_loss[loss=0.07618, simple_loss=0.09778, pruned_loss=0.01755, audio_tagging_loss=0.009744, over 3047838.92 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:46:29,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1379666.6666666667, ans=0.125
2023-11-21 05:46:30,669 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.24 vs. limit=12.0
2023-11-21 05:46:32,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1379666.6666666667, ans=0.1
2023-11-21 05:46:33,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1379666.6666666667, ans=0.125
2023-11-21 05:46:33,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1379666.6666666667, ans=0.2
2023-11-21 05:46:55,176 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:46:55,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1379800.0, ans=0.1
2023-11-21 05:47:09,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1379866.6666666667, ans=0.1
2023-11-21 05:47:17,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1379933.3333333333, ans=0.2
2023-11-21 05:47:24,018 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.15 vs. limit=15.0
2023-11-21 05:47:25,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207000
2023-11-21 05:47:28,466 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2600, loss[loss=0.06727, simple_loss=0.0925, pruned_loss=0.01359, audio_tagging_loss=0.007427, over 14386.00 frames. ], tot_loss[loss=0.07584, simple_loss=0.09744, pruned_loss=0.01746, audio_tagging_loss=0.009658, over 3041788.60 frames. ], batch size: 53, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:47:41,172 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.54 vs. limit=15.0
2023-11-21 05:47:47,825 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.166e+01 8.159e+01 8.772e+01 9.350e+01 1.184e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 05:47:50,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1380066.6666666667, ans=0.125
2023-11-21 05:48:03,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1380133.3333333333, ans=0.125
2023-11-21 05:48:26,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1380266.6666666667, ans=0.0
2023-11-21 05:48:27,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1380266.6666666667, ans=0.125
2023-11-21 05:48:29,805 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207050
2023-11-21 05:48:32,779 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2650, loss[loss=0.07236, simple_loss=0.09385, pruned_loss=0.01604, audio_tagging_loss=0.009399, over 14957.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09704, pruned_loss=0.01734, audio_tagging_loss=0.009573, over 3046333.17 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:48:42,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1380333.3333333333, ans=0.0
2023-11-21 05:48:47,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1380400.0, ans=0.1
2023-11-21 05:49:05,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1380466.6666666667, ans=0.04949747468305833
2023-11-21 05:49:17,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1380533.3333333333, ans=0.125
2023-11-21 05:49:34,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1380600.0, ans=0.125
2023-11-21 05:49:34,963 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207100
2023-11-21 05:49:37,446 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2700, loss[loss=0.07289, simple_loss=0.1044, pruned_loss=0.01006, audio_tagging_loss=0.01063, over 16676.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09747, pruned_loss=0.0174, audio_tagging_loss=0.009515, over 3045707.80 frames. ], batch size: 59, lr: 3.84e-03, grad_scale: 8.0
2023-11-21 05:49:49,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1380733.3333333333, ans=0.125
2023-11-21 05:49:51,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1380733.3333333333, ans=0.1
2023-11-21 05:49:57,498 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 7.943e+01 8.665e+01 9.544e+01 1.191e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 05:49:58,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.56 vs. limit=15.0
2023-11-21 05:50:24,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1380866.6666666667, ans=0.2
2023-11-21 05:50:40,246 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207150
2023-11-21 05:50:42,627 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2750, loss[loss=0.06514, simple_loss=0.0769, pruned_loss=0.0167, audio_tagging_loss=0.009991, over 14295.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.09638, pruned_loss=0.01741, audio_tagging_loss=0.009601, over 3043640.52 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 8.0
2023-11-21 05:51:09,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1381133.3333333333, ans=0.125
2023-11-21 05:51:10,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1381133.3333333333, ans=0.125
2023-11-21 05:51:22,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.73 vs. limit=22.5
2023-11-21 05:51:40,431 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:51:45,586 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207200
2023-11-21 05:51:48,302 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2800, loss[loss=0.1001, simple_loss=0.1309, pruned_loss=0.02776, audio_tagging_loss=0.006851, over 14347.00 frames. ], tot_loss[loss=0.07508, simple_loss=0.09606, pruned_loss=0.01744, audio_tagging_loss=0.009608, over 3042780.31 frames. ], batch size: 53, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:51:51,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1381333.3333333333, ans=0.125
2023-11-21 05:51:57,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1381333.3333333333, ans=0.0
2023-11-21 05:52:02,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1381400.0, ans=0.0
2023-11-21 05:52:08,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1381400.0, ans=0.2
2023-11-21 05:52:09,759 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 7.920e+01 8.497e+01 9.162e+01 1.236e+02, threshold=1.699e+02, percent-clipped=0.0
2023-11-21 05:52:17,007 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.02 vs. limit=15.0
2023-11-21 05:52:25,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1381466.6666666667, ans=0.125
2023-11-21 05:52:33,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1381533.3333333333, ans=0.0
2023-11-21 05:52:52,378 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207250
2023-11-21 05:52:54,689 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2850, loss[loss=0.09583, simple_loss=0.1367, pruned_loss=0.02092, audio_tagging_loss=0.006557, over 16602.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09601, pruned_loss=0.01721, audio_tagging_loss=0.009622, over 3048002.60 frames. ], batch size: 60, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:53:10,229 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.81 vs. limit=15.0
2023-11-21 05:53:12,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1381733.3333333333, ans=0.125
2023-11-21 05:53:19,745 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.00 vs. limit=10.0
2023-11-21 05:53:23,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.43 vs. limit=12.0
2023-11-21 05:53:34,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1381866.6666666667, ans=0.125
2023-11-21 05:53:37,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.51 vs. limit=15.0
2023-11-21 05:53:57,035 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207300
2023-11-21 05:53:59,448 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2900, loss[loss=0.08226, simple_loss=0.1081, pruned_loss=0.02034, audio_tagging_loss=0.007867, over 15350.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09719, pruned_loss=0.01742, audio_tagging_loss=0.009555, over 3045537.09 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:54:00,216 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=12.0
2023-11-21 05:54:06,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.43 vs. limit=22.5
2023-11-21 05:54:20,423 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.83 vs. limit=10.0
2023-11-21 05:54:20,742 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 7.887e+01 8.991e+01 9.698e+01 1.464e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-21 05:54:25,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1382133.3333333333, ans=0.0
2023-11-21 05:54:30,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1382133.3333333333, ans=0.125
2023-11-21 05:54:35,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1382133.3333333333, ans=0.125
2023-11-21 05:55:02,153 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207350
2023-11-21 05:55:04,500 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 2950, loss[loss=0.07817, simple_loss=0.09866, pruned_loss=0.02014, audio_tagging_loss=0.008702, over 14634.00 frames. ], tot_loss[loss=0.07598, simple_loss=0.09761, pruned_loss=0.01751, audio_tagging_loss=0.00966, over 3047243.12 frames. ], batch size: 54, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:55:12,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1382333.3333333333, ans=0.125
2023-11-21 05:55:27,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1382400.0, ans=0.1
2023-11-21 05:55:36,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1382466.6666666667, ans=0.0
2023-11-21 05:55:54,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1382533.3333333333, ans=0.1
2023-11-21 05:56:05,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1382600.0, ans=0.1
2023-11-21 05:56:07,569 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207400
2023-11-21 05:56:10,294 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3000, loss[loss=0.07295, simple_loss=0.09271, pruned_loss=0.01853, audio_tagging_loss=0.008071, over 16220.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.09804, pruned_loss=0.0176, audio_tagging_loss=0.009701, over 3054622.99 frames. ], batch size: 60, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:56:10,295 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 05:56:26,983 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.9314, 5.4180, 5.7985, 5.1933], device='cuda:2')
2023-11-21 05:56:38,587 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.1956, 3.1594, 2.8726, 2.8567, 3.4914, 3.5232, 3.0474, 3.6355],
       device='cuda:2')
2023-11-21 05:56:42,196 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.9928, 5.8535, 5.5976, 5.5791], device='cuda:2')
2023-11-21 05:56:50,263 INFO [train_asr.py:1253] (2/4) Epoch 18, validation: loss=0.06024, simple_loss=0.05252, pruned_loss=0.00529, audio_tagging_loss=0.02869, over 4681554.00 frames. 
2023-11-21 05:56:50,264 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 05:56:58,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1382666.6666666667, ans=0.0
2023-11-21 05:57:11,729 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.230e+01 8.309e+01 8.951e+01 9.724e+01 1.389e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-21 05:57:26,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1382800.0, ans=0.125
2023-11-21 05:57:52,814 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207450
2023-11-21 05:57:55,799 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3050, loss[loss=0.08157, simple_loss=0.1111, pruned_loss=0.01744, audio_tagging_loss=0.008594, over 15411.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09893, pruned_loss=0.01795, audio_tagging_loss=0.009663, over 3047444.57 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:58:12,817 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.03 vs. limit=15.0
2023-11-21 05:58:16,702 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.52 vs. limit=22.5
2023-11-21 05:58:19,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1383066.6666666667, ans=0.125
2023-11-21 05:58:21,397 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.49 vs. limit=15.0
2023-11-21 05:58:34,427 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:58:58,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207500
2023-11-21 05:59:01,620 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3100, loss[loss=0.07026, simple_loss=0.09019, pruned_loss=0.01557, audio_tagging_loss=0.009592, over 16358.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.09888, pruned_loss=0.01796, audio_tagging_loss=0.009668, over 3043829.44 frames. ], batch size: 62, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:59:07,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1383333.3333333333, ans=0.125
2023-11-21 05:59:21,231 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.138e+01 8.768e+01 9.517e+01 1.436e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 05:59:33,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1383466.6666666667, ans=0.125
2023-11-21 05:59:33,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1383466.6666666667, ans=0.1
2023-11-21 05:59:51,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1383533.3333333333, ans=0.125
2023-11-21 06:00:01,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1383600.0, ans=0.2
2023-11-21 06:00:03,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207550
2023-11-21 06:00:06,336 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3150, loss[loss=0.07732, simple_loss=0.09689, pruned_loss=0.01437, audio_tagging_loss=0.0145, over 16064.00 frames. ], tot_loss[loss=0.07703, simple_loss=0.09866, pruned_loss=0.01793, audio_tagging_loss=0.009763, over 3039784.46 frames. ], batch size: 60, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 06:00:36,810 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.73 vs. limit=15.0
2023-11-21 06:00:42,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1383800.0, ans=0.125
2023-11-21 06:00:54,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1383866.6666666667, ans=0.125
2023-11-21 06:00:56,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1383933.3333333333, ans=0.125
2023-11-21 06:01:00,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1383933.3333333333, ans=0.0
2023-11-21 06:01:07,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207600
2023-11-21 06:01:10,784 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3200, loss[loss=0.0738, simple_loss=0.08409, pruned_loss=0.01519, audio_tagging_loss=0.01657, over 14643.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09912, pruned_loss=0.01817, audio_tagging_loss=0.009779, over 3040226.43 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:01:32,255 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.077e+01 8.904e+01 9.561e+01 1.400e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 06:01:47,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.11 vs. limit=15.0
2023-11-21 06:01:56,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1384200.0, ans=0.0
2023-11-21 06:02:07,864 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.73 vs. limit=15.0
2023-11-21 06:02:13,607 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207650
2023-11-21 06:02:13,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1384266.6666666667, ans=0.125
2023-11-21 06:02:15,882 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3250, loss[loss=0.1005, simple_loss=0.1224, pruned_loss=0.03051, audio_tagging_loss=0.008839, over 15192.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09961, pruned_loss=0.01824, audio_tagging_loss=0.009791, over 3046789.31 frames. ], batch size: 54, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:02:42,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1384466.6666666667, ans=0.1
2023-11-21 06:02:44,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.52 vs. limit=15.0
2023-11-21 06:02:49,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1384466.6666666667, ans=0.125
2023-11-21 06:03:15,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1384600.0, ans=0.125
2023-11-21 06:03:17,989 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207700
2023-11-21 06:03:20,339 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3300, loss[loss=0.06979, simple_loss=0.08938, pruned_loss=0.01548, audio_tagging_loss=0.009621, over 15814.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09973, pruned_loss=0.01819, audio_tagging_loss=0.009851, over 3047446.79 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:03:31,890 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:03:36,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1384733.3333333333, ans=0.1
2023-11-21 06:03:39,910 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:03:40,725 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.701e+01 8.200e+01 8.829e+01 9.665e+01 1.279e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 06:04:18,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1384933.3333333333, ans=0.125
2023-11-21 06:04:20,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1384933.3333333333, ans=0.125
2023-11-21 06:04:21,597 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207750
2023-11-21 06:04:24,063 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3350, loss[loss=0.05055, simple_loss=0.06587, pruned_loss=0.007266, audio_tagging_loss=0.01036, over 14897.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09988, pruned_loss=0.0181, audio_tagging_loss=0.009755, over 3046262.47 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:04:31,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1385000.0, ans=0.0
2023-11-21 06:04:37,517 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.67 vs. limit=10.0
2023-11-21 06:04:51,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1385133.3333333333, ans=0.0
2023-11-21 06:05:26,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.92 vs. limit=15.0
2023-11-21 06:05:27,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207800
2023-11-21 06:05:30,658 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3400, loss[loss=0.06739, simple_loss=0.08932, pruned_loss=0.01258, audio_tagging_loss=0.01014, over 15231.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09999, pruned_loss=0.01807, audio_tagging_loss=0.009653, over 3048929.00 frames. ], batch size: 59, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:05:48,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1385400.0, ans=0.125
2023-11-21 06:05:50,869 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 8.344e+01 9.058e+01 9.925e+01 1.172e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-21 06:05:51,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1385400.0, ans=0.125
2023-11-21 06:06:02,878 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.45 vs. limit=6.0
2023-11-21 06:06:06,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1385466.6666666667, ans=0.1
2023-11-21 06:06:13,862 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.40 vs. limit=15.0
2023-11-21 06:06:23,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.50 vs. limit=15.0
2023-11-21 06:06:28,324 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:06:33,003 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207850
2023-11-21 06:06:35,419 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3450, loss[loss=0.06699, simple_loss=0.08441, pruned_loss=0.01407, audio_tagging_loss=0.01071, over 14983.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09932, pruned_loss=0.0179, audio_tagging_loss=0.009548, over 3052233.37 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:06:39,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1385666.6666666667, ans=0.125
2023-11-21 06:06:43,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1385666.6666666667, ans=0.2
2023-11-21 06:06:53,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1385733.3333333333, ans=0.1
2023-11-21 06:06:57,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1385733.3333333333, ans=0.0
2023-11-21 06:07:00,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1385800.0, ans=0.125
2023-11-21 06:07:12,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1385800.0, ans=0.035
2023-11-21 06:07:37,329 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207900
2023-11-21 06:07:38,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1386000.0, ans=0.1
2023-11-21 06:07:39,802 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3500, loss[loss=0.07976, simple_loss=0.1098, pruned_loss=0.01665, audio_tagging_loss=0.008189, over 14853.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09858, pruned_loss=0.01774, audio_tagging_loss=0.009496, over 3047911.13 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:07:53,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1386066.6666666667, ans=0.0
2023-11-21 06:08:02,172 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.378e+01 8.049e+01 8.836e+01 9.881e+01 1.284e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 06:08:06,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1386133.3333333333, ans=0.1
2023-11-21 06:08:14,508 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:08:27,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1386200.0, ans=0.2
2023-11-21 06:08:28,779 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.67 vs. limit=15.0
2023-11-21 06:08:43,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 207950
2023-11-21 06:08:43,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1386266.6666666667, ans=0.1
2023-11-21 06:08:45,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1386333.3333333333, ans=0.1
2023-11-21 06:08:45,810 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3550, loss[loss=0.07365, simple_loss=0.1027, pruned_loss=0.01662, audio_tagging_loss=0.005691, over 16133.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09768, pruned_loss=0.01755, audio_tagging_loss=0.009463, over 3050303.21 frames. ], batch size: 60, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:08:48,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1386333.3333333333, ans=0.2
2023-11-21 06:08:51,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1386333.3333333333, ans=0.09899494936611666
2023-11-21 06:08:53,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1386333.3333333333, ans=0.0
2023-11-21 06:09:11,403 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.65 vs. limit=6.0
2023-11-21 06:09:27,136 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.24 vs. limit=15.0
2023-11-21 06:09:33,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1386533.3333333333, ans=0.125
2023-11-21 06:09:37,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1386600.0, ans=0.0
2023-11-21 06:09:49,115 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208000
2023-11-21 06:09:49,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1386600.0, ans=0.125
2023-11-21 06:09:54,969 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3600, loss[loss=0.05719, simple_loss=0.0718, pruned_loss=0.01233, audio_tagging_loss=0.008966, over 14318.00 frames. ], tot_loss[loss=0.07589, simple_loss=0.0979, pruned_loss=0.01754, audio_tagging_loss=0.0094, over 3055614.04 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:09:56,863 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.42 vs. limit=10.0
2023-11-21 06:10:06,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1386733.3333333333, ans=0.07
2023-11-21 06:10:14,781 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.348e+01 7.918e+01 8.568e+01 9.474e+01 1.185e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 06:10:15,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1386733.3333333333, ans=0.0
2023-11-21 06:10:36,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1386866.6666666667, ans=0.04949747468305833
2023-11-21 06:10:40,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1386866.6666666667, ans=0.5
2023-11-21 06:10:56,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208050
2023-11-21 06:10:58,440 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3650, loss[loss=0.08096, simple_loss=0.1136, pruned_loss=0.01899, audio_tagging_loss=0.005151, over 15230.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09783, pruned_loss=0.0177, audio_tagging_loss=0.009452, over 3050019.02 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:11:12,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.84 vs. limit=22.5
2023-11-21 06:12:01,542 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208100
2023-11-21 06:12:03,988 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3700, loss[loss=0.0618, simple_loss=0.07709, pruned_loss=0.01168, audio_tagging_loss=0.01157, over 14501.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09667, pruned_loss=0.01749, audio_tagging_loss=0.009504, over 3051832.73 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:12:09,825 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.47 vs. limit=10.0
2023-11-21 06:12:25,069 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.253e+01 8.797e+01 9.684e+01 1.136e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 06:12:26,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1387400.0, ans=0.07
2023-11-21 06:12:42,156 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.45 vs. limit=15.0
2023-11-21 06:12:42,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1387533.3333333333, ans=0.0
2023-11-21 06:12:55,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1387600.0, ans=0.2
2023-11-21 06:13:07,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208150
2023-11-21 06:13:08,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.05 vs. limit=15.0
2023-11-21 06:13:09,734 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3750, loss[loss=0.0782, simple_loss=0.1028, pruned_loss=0.01827, audio_tagging_loss=0.008507, over 14941.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.0971, pruned_loss=0.01734, audio_tagging_loss=0.009523, over 3053804.43 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:13:13,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1387666.6666666667, ans=0.125
2023-11-21 06:13:17,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1387666.6666666667, ans=0.1
2023-11-21 06:13:20,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1387666.6666666667, ans=0.125
2023-11-21 06:13:36,849 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:13:37,269 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.77 vs. limit=6.0
2023-11-21 06:13:55,229 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:14:11,935 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208200
2023-11-21 06:14:14,586 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3800, loss[loss=0.0876, simple_loss=0.1105, pruned_loss=0.02209, audio_tagging_loss=0.01026, over 14743.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.0971, pruned_loss=0.01732, audio_tagging_loss=0.009783, over 3055010.92 frames. ], batch size: 54, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:14:17,916 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.75 vs. limit=15.0
2023-11-21 06:14:23,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1388000.0, ans=0.025
2023-11-21 06:14:26,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1388066.6666666667, ans=0.95
2023-11-21 06:14:32,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1388066.6666666667, ans=0.0
2023-11-21 06:14:35,463 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.099e+01 8.755e+01 9.581e+01 1.226e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 06:14:35,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1388066.6666666667, ans=0.125
2023-11-21 06:14:38,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1388066.6666666667, ans=0.1
2023-11-21 06:14:56,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1388200.0, ans=0.0
2023-11-21 06:15:10,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1388266.6666666667, ans=0.125
2023-11-21 06:15:12,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1388266.6666666667, ans=0.1
2023-11-21 06:15:16,777 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208250
2023-11-21 06:15:19,736 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3850, loss[loss=0.06072, simple_loss=0.07736, pruned_loss=0.01201, audio_tagging_loss=0.01003, over 14018.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.0966, pruned_loss=0.01708, audio_tagging_loss=0.009866, over 3048361.84 frames. ], batch size: 53, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:15:27,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1388333.3333333333, ans=0.125
2023-11-21 06:15:51,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1388466.6666666667, ans=0.125
2023-11-21 06:16:15,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=1388600.0, ans=0.2
2023-11-21 06:16:22,595 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208300
2023-11-21 06:16:24,947 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3900, loss[loss=0.08991, simple_loss=0.1187, pruned_loss=0.02151, audio_tagging_loss=0.009033, over 15568.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.0954, pruned_loss=0.01678, audio_tagging_loss=0.009911, over 3045498.89 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:16:30,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1388666.6666666667, ans=0.0
2023-11-21 06:16:46,313 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.288e+01 8.904e+01 9.716e+01 1.197e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 06:17:04,312 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.72 vs. limit=12.0
2023-11-21 06:17:27,507 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208350
2023-11-21 06:17:29,820 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 3950, loss[loss=0.0714, simple_loss=0.09039, pruned_loss=0.01317, audio_tagging_loss=0.01303, over 15401.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09611, pruned_loss=0.01701, audio_tagging_loss=0.009862, over 3049523.23 frames. ], batch size: 60, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:17:41,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1389066.6666666667, ans=0.04949747468305833
2023-11-21 06:17:50,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1389066.6666666667, ans=0.125
2023-11-21 06:17:52,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1389066.6666666667, ans=0.0
2023-11-21 06:18:13,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1389200.0, ans=0.1
2023-11-21 06:18:31,494 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208400
2023-11-21 06:18:34,776 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4000, loss[loss=0.09103, simple_loss=0.1153, pruned_loss=0.02559, audio_tagging_loss=0.007772, over 15467.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09589, pruned_loss=0.01698, audio_tagging_loss=0.009946, over 3046054.66 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:18:56,801 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.265e+01 8.108e+01 8.819e+01 9.792e+01 1.460e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 06:19:02,390 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.89 vs. limit=15.0
2023-11-21 06:19:13,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1389533.3333333333, ans=0.1
2023-11-21 06:19:18,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1389533.3333333333, ans=0.2
2023-11-21 06:19:21,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.51 vs. limit=15.0
2023-11-21 06:19:37,696 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208450
2023-11-21 06:19:40,013 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4050, loss[loss=0.08584, simple_loss=0.1143, pruned_loss=0.01794, audio_tagging_loss=0.01074, over 14527.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.09537, pruned_loss=0.01688, audio_tagging_loss=0.009968, over 3042610.25 frames. ], batch size: 53, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:19:43,774 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:19:47,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1389666.6666666667, ans=0.125
2023-11-21 06:20:15,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1389800.0, ans=0.125
2023-11-21 06:20:25,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1389866.6666666667, ans=0.2
2023-11-21 06:20:36,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1389933.3333333333, ans=0.2
2023-11-21 06:20:37,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1389933.3333333333, ans=0.125
2023-11-21 06:20:41,766 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208500
2023-11-21 06:20:44,118 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4100, loss[loss=0.07836, simple_loss=0.09949, pruned_loss=0.02239, audio_tagging_loss=0.006222, over 16200.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09652, pruned_loss=0.01712, audio_tagging_loss=0.009912, over 3047348.07 frames. ], batch size: 63, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:20:44,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1390000.0, ans=0.0
2023-11-21 06:21:06,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1390066.6666666667, ans=0.0
2023-11-21 06:21:08,097 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.154e+01 8.779e+01 9.471e+01 2.037e+02, threshold=1.756e+02, percent-clipped=1.0
2023-11-21 06:21:20,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1390133.3333333333, ans=0.2
2023-11-21 06:21:27,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1390200.0, ans=0.0
2023-11-21 06:21:29,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1390200.0, ans=0.2
2023-11-21 06:21:36,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.54 vs. limit=22.5
2023-11-21 06:21:46,758 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208550
2023-11-21 06:21:49,126 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4150, loss[loss=0.07037, simple_loss=0.0937, pruned_loss=0.01363, audio_tagging_loss=0.009889, over 14943.00 frames. ], tot_loss[loss=0.07526, simple_loss=0.09667, pruned_loss=0.01714, audio_tagging_loss=0.009785, over 3043821.37 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:21:51,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.89 vs. limit=22.5
2023-11-21 06:21:59,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-21 06:22:01,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1390400.0, ans=0.1
2023-11-21 06:22:10,259 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.69 vs. limit=6.0
2023-11-21 06:22:37,694 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:22:49,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1390600.0, ans=0.1
2023-11-21 06:22:52,613 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208600
2023-11-21 06:22:52,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1390600.0, ans=0.125
2023-11-21 06:22:55,643 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4200, loss[loss=0.08817, simple_loss=0.1129, pruned_loss=0.02048, audio_tagging_loss=0.01124, over 15156.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09675, pruned_loss=0.01732, audio_tagging_loss=0.009709, over 3040174.71 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:23:02,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1390666.6666666667, ans=0.2
2023-11-21 06:23:03,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1390666.6666666667, ans=0.125
2023-11-21 06:23:08,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1390733.3333333333, ans=0.125
2023-11-21 06:23:08,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1390733.3333333333, ans=0.125
2023-11-21 06:23:17,927 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 7.993e+01 8.905e+01 1.014e+02 1.723e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 06:23:22,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1390800.0, ans=0.125
2023-11-21 06:23:23,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1390800.0, ans=0.09899494936611666
2023-11-21 06:23:40,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1390866.6666666667, ans=0.0
2023-11-21 06:23:47,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1390933.3333333333, ans=0.125
2023-11-21 06:23:57,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208650
2023-11-21 06:23:59,937 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4250, loss[loss=0.08976, simple_loss=0.1129, pruned_loss=0.02154, audio_tagging_loss=0.01177, over 14733.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09674, pruned_loss=0.01728, audio_tagging_loss=0.009641, over 3040990.13 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:24:14,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1391066.6666666667, ans=0.0
2023-11-21 06:24:51,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1391266.6666666667, ans=0.0
2023-11-21 06:24:51,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1391266.6666666667, ans=0.125
2023-11-21 06:24:52,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff2.min_abs, batch_count=1391266.6666666667, ans=0.1
2023-11-21 06:25:02,611 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208700
2023-11-21 06:25:05,642 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4300, loss[loss=0.1023, simple_loss=0.1322, pruned_loss=0.02944, audio_tagging_loss=0.006768, over 15367.00 frames. ], tot_loss[loss=0.07561, simple_loss=0.09727, pruned_loss=0.01743, audio_tagging_loss=0.009545, over 3042724.34 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:25:10,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1391333.3333333333, ans=0.125
2023-11-21 06:25:30,098 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.185e+01 9.001e+01 9.859e+01 1.198e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 06:25:39,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1391466.6666666667, ans=0.1
2023-11-21 06:26:09,725 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208750
2023-11-21 06:26:12,107 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4350, loss[loss=0.06883, simple_loss=0.08697, pruned_loss=0.01285, audio_tagging_loss=0.0125, over 16121.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.09764, pruned_loss=0.01745, audio_tagging_loss=0.009404, over 3044993.09 frames. ], batch size: 59, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:26:20,553 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:26:20,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1391666.6666666667, ans=0.0
2023-11-21 06:26:38,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.45 vs. limit=15.0
2023-11-21 06:26:44,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1391800.0, ans=0.125
2023-11-21 06:26:58,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1391866.6666666667, ans=0.2
2023-11-21 06:27:00,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1391866.6666666667, ans=0.1
2023-11-21 06:27:03,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1391933.3333333333, ans=0.0
2023-11-21 06:27:14,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208800
2023-11-21 06:27:16,886 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4400, loss[loss=0.1085, simple_loss=0.1387, pruned_loss=0.02989, audio_tagging_loss=0.009284, over 16886.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09819, pruned_loss=0.01763, audio_tagging_loss=0.009422, over 3046873.83 frames. ], batch size: 61, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:27:21,637 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.70 vs. limit=22.5
2023-11-21 06:27:40,624 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.458e+01 8.121e+01 8.675e+01 9.482e+01 1.551e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 06:28:15,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1392266.6666666667, ans=0.0
2023-11-21 06:28:18,996 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208850
2023-11-21 06:28:21,398 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4450, loss[loss=0.08616, simple_loss=0.1042, pruned_loss=0.02231, audio_tagging_loss=0.01176, over 15043.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.09738, pruned_loss=0.01736, audio_tagging_loss=0.009493, over 3053847.75 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:29:17,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1392600.0, ans=0.1
2023-11-21 06:29:23,822 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208900
2023-11-21 06:29:26,792 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4500, loss[loss=0.06926, simple_loss=0.0881, pruned_loss=0.01594, audio_tagging_loss=0.009274, over 15017.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09755, pruned_loss=0.01733, audio_tagging_loss=0.009516, over 3048144.40 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:29:27,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1392666.6666666667, ans=0.0
2023-11-21 06:29:28,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.11 vs. limit=22.5
2023-11-21 06:29:30,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1392666.6666666667, ans=0.125
2023-11-21 06:29:34,639 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.90 vs. limit=15.0
2023-11-21 06:29:49,973 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.149e+01 8.678e+01 9.442e+01 1.327e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 06:29:58,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1392800.0, ans=0.125
2023-11-21 06:30:01,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1392800.0, ans=0.125
2023-11-21 06:30:16,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1392866.6666666667, ans=0.0
2023-11-21 06:30:19,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1392933.3333333333, ans=0.0
2023-11-21 06:30:27,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1392933.3333333333, ans=0.125
2023-11-21 06:30:29,895 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 208950
2023-11-21 06:30:31,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1393000.0, ans=0.0
2023-11-21 06:30:32,289 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4550, loss[loss=0.07318, simple_loss=0.08989, pruned_loss=0.01634, audio_tagging_loss=0.01189, over 15655.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09682, pruned_loss=0.01731, audio_tagging_loss=0.009632, over 3048294.76 frames. ], batch size: 59, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:30:45,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1393066.6666666667, ans=0.125
2023-11-21 06:30:57,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.85 vs. limit=10.0
2023-11-21 06:31:23,181 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:31:25,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1393266.6666666667, ans=0.1
2023-11-21 06:31:34,230 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209000
2023-11-21 06:31:36,996 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4600, loss[loss=0.08318, simple_loss=0.1083, pruned_loss=0.01981, audio_tagging_loss=0.009209, over 15871.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09627, pruned_loss=0.01714, audio_tagging_loss=0.009742, over 3055529.76 frames. ], batch size: 59, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:32:01,351 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.053e+01 8.667e+01 9.436e+01 2.145e+02, threshold=1.733e+02, percent-clipped=1.0
2023-11-21 06:32:09,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1393466.6666666667, ans=0.1
2023-11-21 06:32:10,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1393466.6666666667, ans=0.125
2023-11-21 06:32:12,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1393466.6666666667, ans=0.1
2023-11-21 06:32:28,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.08 vs. limit=15.0
2023-11-21 06:32:39,149 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209050
2023-11-21 06:32:41,477 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4650, loss[loss=0.0768, simple_loss=0.09707, pruned_loss=0.02019, audio_tagging_loss=0.008076, over 15356.00 frames. ], tot_loss[loss=0.07477, simple_loss=0.09586, pruned_loss=0.01704, audio_tagging_loss=0.009793, over 3061509.74 frames. ], batch size: 58, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:32:56,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1393733.3333333333, ans=0.0
2023-11-21 06:33:12,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.35 vs. limit=15.0
2023-11-21 06:33:31,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1393866.6666666667, ans=0.1
2023-11-21 06:33:32,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1393933.3333333333, ans=0.0
2023-11-21 06:33:43,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1393933.3333333333, ans=0.2
2023-11-21 06:33:44,881 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209100
2023-11-21 06:33:47,224 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4700, loss[loss=0.08997, simple_loss=0.113, pruned_loss=0.02392, audio_tagging_loss=0.00957, over 15148.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09698, pruned_loss=0.0173, audio_tagging_loss=0.00986, over 3061466.58 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:33:49,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1394000.0, ans=0.125
2023-11-21 06:33:56,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1394000.0, ans=0.0
2023-11-21 06:34:07,537 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:34:10,862 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.149e+01 8.884e+01 9.885e+01 2.074e+02, threshold=1.777e+02, percent-clipped=1.0
2023-11-21 06:34:35,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1394200.0, ans=0.0
2023-11-21 06:34:48,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209150
2023-11-21 06:34:51,285 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4750, loss[loss=0.07759, simple_loss=0.09114, pruned_loss=0.01995, audio_tagging_loss=0.01206, over 15464.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09761, pruned_loss=0.01748, audio_tagging_loss=0.00984, over 3060776.35 frames. ], batch size: 60, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:34:52,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1394333.3333333333, ans=0.125
2023-11-21 06:35:12,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1394400.0, ans=0.1
2023-11-21 06:35:25,570 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.10 vs. limit=15.0
2023-11-21 06:35:50,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1394600.0, ans=0.125
2023-11-21 06:35:51,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1394600.0, ans=0.125
2023-11-21 06:35:54,383 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209200
2023-11-21 06:35:57,088 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4800, loss[loss=0.09733, simple_loss=0.131, pruned_loss=0.02099, audio_tagging_loss=0.01084, over 16115.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.09788, pruned_loss=0.01765, audio_tagging_loss=0.00995, over 3056127.93 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:36:21,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1394733.3333333333, ans=0.125
2023-11-21 06:36:24,238 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.238e+01 8.873e+01 9.665e+01 1.246e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 06:36:28,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1394800.0, ans=0.125
2023-11-21 06:36:34,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1394800.0, ans=0.0
2023-11-21 06:36:43,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1394866.6666666667, ans=0.125
2023-11-21 06:36:46,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1394866.6666666667, ans=0.125
2023-11-21 06:36:55,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1394933.3333333333, ans=0.125
2023-11-21 06:37:01,201 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209250
2023-11-21 06:37:04,145 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4850, loss[loss=0.07515, simple_loss=0.0903, pruned_loss=0.01685, audio_tagging_loss=0.01315, over 15380.00 frames. ], tot_loss[loss=0.07655, simple_loss=0.09776, pruned_loss=0.01764, audio_tagging_loss=0.01003, over 3045246.62 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:37:12,427 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.89 vs. limit=22.5
2023-11-21 06:37:14,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1395000.0, ans=0.5
2023-11-21 06:37:39,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1395133.3333333333, ans=0.125
2023-11-21 06:37:56,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1395266.6666666667, ans=0.0
2023-11-21 06:38:00,917 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.09 vs. limit=10.0
2023-11-21 06:38:04,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1395266.6666666667, ans=0.1
2023-11-21 06:38:06,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209300
2023-11-21 06:38:06,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1395266.6666666667, ans=0.125
2023-11-21 06:38:08,815 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4900, loss[loss=0.06643, simple_loss=0.08729, pruned_loss=0.01403, audio_tagging_loss=0.00876, over 15083.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09776, pruned_loss=0.01756, audio_tagging_loss=0.009956, over 3042546.65 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:38:14,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.95 vs. limit=15.0
2023-11-21 06:38:34,575 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.171e+01 8.744e+01 9.651e+01 1.565e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 06:38:49,708 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:38:50,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=22.5
2023-11-21 06:38:58,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1395533.3333333333, ans=0.0
2023-11-21 06:39:05,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1395600.0, ans=0.0
2023-11-21 06:39:10,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209350
2023-11-21 06:39:13,589 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 4950, loss[loss=0.04919, simple_loss=0.04959, pruned_loss=0.01095, audio_tagging_loss=0.01344, over 15595.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09733, pruned_loss=0.01737, audio_tagging_loss=0.009875, over 3043295.37 frames. ], batch size: 63, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:39:23,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1395666.6666666667, ans=0.0
2023-11-21 06:39:24,505 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:39:39,767 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.58 vs. limit=15.0
2023-11-21 06:40:14,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1395933.3333333333, ans=0.0
2023-11-21 06:40:17,555 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209400
2023-11-21 06:40:20,257 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5000, loss[loss=0.05857, simple_loss=0.07342, pruned_loss=0.01134, audio_tagging_loss=0.01052, over 15543.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.09746, pruned_loss=0.0174, audio_tagging_loss=0.009656, over 3041980.97 frames. ], batch size: 60, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:40:36,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1396066.6666666667, ans=0.2
2023-11-21 06:40:47,089 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.108e+01 8.729e+01 9.428e+01 1.080e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 06:41:12,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1396266.6666666667, ans=0.125
2023-11-21 06:41:23,053 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209450
2023-11-21 06:41:25,441 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5050, loss[loss=0.05007, simple_loss=0.05687, pruned_loss=0.01265, audio_tagging_loss=0.008983, over 14507.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09689, pruned_loss=0.01735, audio_tagging_loss=0.009627, over 3042869.55 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:41:26,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.38 vs. limit=22.5
2023-11-21 06:41:28,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1396333.3333333333, ans=0.2
2023-11-21 06:41:44,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1396400.0, ans=0.0
2023-11-21 06:41:58,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1396466.6666666667, ans=0.125
2023-11-21 06:42:12,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1396533.3333333333, ans=0.0
2023-11-21 06:42:19,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1396600.0, ans=0.125
2023-11-21 06:42:27,036 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209500
2023-11-21 06:42:29,382 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5100, loss[loss=0.08502, simple_loss=0.1039, pruned_loss=0.02257, audio_tagging_loss=0.01052, over 14500.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.09643, pruned_loss=0.01709, audio_tagging_loss=0.009616, over 3039945.92 frames. ], batch size: 55, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:42:39,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1396666.6666666667, ans=0.2
2023-11-21 06:42:43,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1396733.3333333333, ans=0.125
2023-11-21 06:42:57,166 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.138e+01 8.643e+01 9.208e+01 2.160e+02, threshold=1.729e+02, percent-clipped=1.0
2023-11-21 06:42:57,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1396800.0, ans=0.125
2023-11-21 06:42:59,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1396800.0, ans=0.2
2023-11-21 06:43:27,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1396933.3333333333, ans=0.0
2023-11-21 06:43:32,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209550
2023-11-21 06:43:34,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1397000.0, ans=0.1
2023-11-21 06:43:35,668 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5150, loss[loss=0.05919, simple_loss=0.07469, pruned_loss=0.01238, audio_tagging_loss=0.009469, over 16253.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09564, pruned_loss=0.0169, audio_tagging_loss=0.00965, over 3040139.34 frames. ], batch size: 62, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:43:46,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.32 vs. limit=15.0
2023-11-21 06:43:55,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1397066.6666666667, ans=0.0
2023-11-21 06:44:00,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1397133.3333333333, ans=0.0
2023-11-21 06:44:06,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1397133.3333333333, ans=0.0
2023-11-21 06:44:10,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1397133.3333333333, ans=0.0
2023-11-21 06:44:17,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1397200.0, ans=0.1
2023-11-21 06:44:24,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1397200.0, ans=0.125
2023-11-21 06:44:38,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209600
2023-11-21 06:44:40,790 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5200, loss[loss=0.08112, simple_loss=0.1023, pruned_loss=0.02103, audio_tagging_loss=0.00894, over 15498.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09599, pruned_loss=0.0172, audio_tagging_loss=0.009597, over 3042840.55 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:44:48,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.57 vs. limit=22.5
2023-11-21 06:45:03,471 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.50 vs. limit=15.0
2023-11-21 06:45:07,624 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.125e+01 8.190e+01 8.762e+01 9.315e+01 1.275e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 06:45:30,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1397533.3333333333, ans=0.0
2023-11-21 06:45:42,973 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209650
2023-11-21 06:45:43,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1397600.0, ans=0.125
2023-11-21 06:45:45,307 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5250, loss[loss=0.06934, simple_loss=0.09015, pruned_loss=0.01461, audio_tagging_loss=0.009661, over 15452.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09657, pruned_loss=0.01713, audio_tagging_loss=0.009515, over 3042655.13 frames. ], batch size: 55, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:45:47,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1397666.6666666667, ans=0.1
2023-11-21 06:46:01,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.12 vs. limit=22.5
2023-11-21 06:46:05,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1397733.3333333333, ans=0.0
2023-11-21 06:46:12,181 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.52 vs. limit=15.0
2023-11-21 06:46:15,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=1397800.0, ans=8.0
2023-11-21 06:46:18,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1397800.0, ans=0.1
2023-11-21 06:46:24,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1397866.6666666667, ans=0.0
2023-11-21 06:46:26,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1397866.6666666667, ans=0.125
2023-11-21 06:46:47,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209700
2023-11-21 06:46:50,778 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5300, loss[loss=0.08555, simple_loss=0.1103, pruned_loss=0.02193, audio_tagging_loss=0.008443, over 16353.00 frames. ], tot_loss[loss=0.07584, simple_loss=0.09799, pruned_loss=0.01743, audio_tagging_loss=0.009416, over 3041877.22 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:47:12,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1398066.6666666667, ans=0.1
2023-11-21 06:47:17,456 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.345e+01 8.152e+01 8.842e+01 9.695e+01 1.214e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 06:47:21,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1398133.3333333333, ans=10.0
2023-11-21 06:47:36,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1398200.0, ans=0.09899494936611666
2023-11-21 06:47:52,589 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209750
2023-11-21 06:47:54,998 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5350, loss[loss=0.08156, simple_loss=0.1104, pruned_loss=0.01765, audio_tagging_loss=0.0087, over 14613.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09804, pruned_loss=0.01726, audio_tagging_loss=0.009407, over 3045859.41 frames. ], batch size: 55, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:48:00,921 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.09 vs. limit=15.0
2023-11-21 06:48:39,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1398533.3333333333, ans=0.0
2023-11-21 06:48:41,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1398533.3333333333, ans=0.125
2023-11-21 06:48:56,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1398600.0, ans=0.0
2023-11-21 06:48:57,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209800
2023-11-21 06:49:00,613 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5400, loss[loss=0.07117, simple_loss=0.08879, pruned_loss=0.01679, audio_tagging_loss=0.009984, over 14834.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.09837, pruned_loss=0.01728, audio_tagging_loss=0.009435, over 3048910.82 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:49:09,073 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:49:12,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1398733.3333333333, ans=0.125
2023-11-21 06:49:24,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1398733.3333333333, ans=0.125
2023-11-21 06:49:28,205 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.049e+01 8.809e+01 9.277e+01 1.117e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 06:49:33,833 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.13 vs. limit=15.0
2023-11-21 06:49:41,048 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.00 vs. limit=15.0
2023-11-21 06:50:02,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209850
2023-11-21 06:50:04,921 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5450, loss[loss=0.07245, simple_loss=0.0914, pruned_loss=0.01557, audio_tagging_loss=0.01117, over 14956.00 frames. ], tot_loss[loss=0.076, simple_loss=0.09815, pruned_loss=0.01732, audio_tagging_loss=0.009609, over 3046215.41 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:50:09,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1399000.0, ans=0.125
2023-11-21 06:50:31,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.16 vs. limit=22.5
2023-11-21 06:50:33,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1399133.3333333333, ans=0.0
2023-11-21 06:50:39,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.47 vs. limit=15.0
2023-11-21 06:50:49,895 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.50 vs. limit=15.0
2023-11-21 06:51:06,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.91 vs. limit=10.0
2023-11-21 06:51:08,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209900
2023-11-21 06:51:10,509 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5500, loss[loss=0.06189, simple_loss=0.08537, pruned_loss=0.01095, audio_tagging_loss=0.008257, over 16279.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09854, pruned_loss=0.01742, audio_tagging_loss=0.00956, over 3047733.60 frames. ], batch size: 60, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:51:13,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1399333.3333333333, ans=0.0
2023-11-21 06:51:31,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1399400.0, ans=0.125
2023-11-21 06:51:36,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0
2023-11-21 06:51:37,451 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 7.977e+01 8.798e+01 9.625e+01 1.160e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 06:51:41,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1399466.6666666667, ans=0.125
2023-11-21 06:51:45,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1399466.6666666667, ans=0.1
2023-11-21 06:52:02,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1399600.0, ans=0.0
2023-11-21 06:52:11,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 209950
2023-11-21 06:52:14,289 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5550, loss[loss=0.1019, simple_loss=0.1343, pruned_loss=0.02565, audio_tagging_loss=0.009155, over 16432.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09755, pruned_loss=0.01716, audio_tagging_loss=0.009719, over 3042675.17 frames. ], batch size: 61, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:52:28,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1399733.3333333333, ans=0.5
2023-11-21 06:52:28,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1399733.3333333333, ans=0.2
2023-11-21 06:52:28,452 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.99 vs. limit=15.0
2023-11-21 06:52:44,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.whiten.whitening_limit, batch_count=1399800.0, ans=12.0
2023-11-21 06:52:50,801 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.08 vs. limit=15.0
2023-11-21 06:52:54,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1399866.6666666667, ans=0.0
2023-11-21 06:53:17,496 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210000
2023-11-21 06:53:20,242 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5600, loss[loss=0.09349, simple_loss=0.1303, pruned_loss=0.01958, audio_tagging_loss=0.008772, over 16280.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09728, pruned_loss=0.01701, audio_tagging_loss=0.009827, over 3046609.46 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:53:36,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1400066.6666666667, ans=0.04949747468305833
2023-11-21 06:53:40,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1400066.6666666667, ans=0.125
2023-11-21 06:53:45,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1400133.3333333333, ans=0.125
2023-11-21 06:53:47,690 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.103e+01 7.991e+01 8.686e+01 9.530e+01 1.132e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 06:53:55,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1400133.3333333333, ans=0.2
2023-11-21 06:54:07,384 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:54:07,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1400200.0, ans=0.0
2023-11-21 06:54:08,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.29 vs. limit=15.0
2023-11-21 06:54:14,701 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=12.0
2023-11-21 06:54:22,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1400266.6666666667, ans=0.2
2023-11-21 06:54:23,469 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210050
2023-11-21 06:54:25,749 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5650, loss[loss=0.07901, simple_loss=0.1005, pruned_loss=0.01843, audio_tagging_loss=0.01034, over 16703.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09797, pruned_loss=0.01721, audio_tagging_loss=0.009818, over 3052759.31 frames. ], batch size: 63, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:54:39,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1400400.0, ans=0.0
2023-11-21 06:54:56,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1400466.6666666667, ans=0.125
2023-11-21 06:55:04,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1400533.3333333333, ans=0.125
2023-11-21 06:55:26,363 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210100
2023-11-21 06:55:28,790 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5700, loss[loss=0.06924, simple_loss=0.0842, pruned_loss=0.01843, audio_tagging_loss=0.008716, over 15679.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09737, pruned_loss=0.01718, audio_tagging_loss=0.009856, over 3056891.74 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:55:57,080 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.246e+01 8.120e+01 8.868e+01 9.513e+01 1.273e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 06:55:59,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1400800.0, ans=0.125
2023-11-21 06:56:06,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1400800.0, ans=0.2
2023-11-21 06:56:07,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1400866.6666666667, ans=0.0
2023-11-21 06:56:19,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1400933.3333333333, ans=0.125
2023-11-21 06:56:28,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1400933.3333333333, ans=0.0
2023-11-21 06:56:30,825 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210150
2023-11-21 06:56:33,104 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.21 vs. limit=15.0
2023-11-21 06:56:33,740 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5750, loss[loss=0.04282, simple_loss=0.05476, pruned_loss=0.007415, audio_tagging_loss=0.008026, over 15079.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.09725, pruned_loss=0.01725, audio_tagging_loss=0.009722, over 3055503.70 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:56:41,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1401000.0, ans=0.125
2023-11-21 06:56:43,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1401000.0, ans=0.0
2023-11-21 06:57:01,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1401133.3333333333, ans=0.2
2023-11-21 06:57:32,339 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.78 vs. limit=12.0
2023-11-21 06:57:36,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210200
2023-11-21 06:57:39,698 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5800, loss[loss=0.0747, simple_loss=0.08935, pruned_loss=0.01953, audio_tagging_loss=0.01049, over 15461.00 frames. ], tot_loss[loss=0.07517, simple_loss=0.09652, pruned_loss=0.01716, audio_tagging_loss=0.009741, over 3053699.57 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:57:42,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1401333.3333333333, ans=0.0
2023-11-21 06:57:50,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1401400.0, ans=0.125
2023-11-21 06:58:00,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.20 vs. limit=15.0
2023-11-21 06:58:05,606 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.333e+01 8.998e+01 9.692e+01 1.329e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 06:58:13,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1401466.6666666667, ans=0.0
2023-11-21 06:58:26,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1401533.3333333333, ans=0.125
2023-11-21 06:58:32,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1401600.0, ans=0.0
2023-11-21 06:58:41,992 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210250
2023-11-21 06:58:42,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1401600.0, ans=0.0
2023-11-21 06:58:44,334 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5850, loss[loss=0.089, simple_loss=0.1186, pruned_loss=0.02275, audio_tagging_loss=0.006937, over 15092.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09675, pruned_loss=0.01725, audio_tagging_loss=0.009576, over 3048152.93 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:58:55,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1401733.3333333333, ans=0.1
2023-11-21 06:58:59,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1401733.3333333333, ans=0.0
2023-11-21 06:59:18,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.21 vs. limit=6.0
2023-11-21 06:59:28,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1401866.6666666667, ans=0.125
2023-11-21 06:59:31,457 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.11 vs. limit=10.0
2023-11-21 06:59:38,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1401933.3333333333, ans=0.125
2023-11-21 06:59:42,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1401933.3333333333, ans=0.125
2023-11-21 06:59:43,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1401933.3333333333, ans=0.125
2023-11-21 06:59:45,588 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210300
2023-11-21 06:59:47,884 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5900, loss[loss=0.07846, simple_loss=0.1001, pruned_loss=0.01889, audio_tagging_loss=0.009536, over 14997.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09719, pruned_loss=0.01724, audio_tagging_loss=0.009544, over 3047747.69 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 06:59:59,490 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=22.5
2023-11-21 07:00:17,480 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.132e+01 8.306e+01 8.999e+01 9.965e+01 1.195e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 07:00:47,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1402266.6666666667, ans=0.125
2023-11-21 07:00:49,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.50 vs. limit=22.5
2023-11-21 07:00:51,595 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210350
2023-11-21 07:00:54,588 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 5950, loss[loss=0.08709, simple_loss=0.1076, pruned_loss=0.02352, audio_tagging_loss=0.00976, over 13777.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09758, pruned_loss=0.01732, audio_tagging_loss=0.009463, over 3048331.05 frames. ], batch size: 53, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:01:02,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.28 vs. limit=15.0
2023-11-21 07:01:11,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1402400.0, ans=0.1
2023-11-21 07:01:21,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1402466.6666666667, ans=0.0
2023-11-21 07:01:28,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1402466.6666666667, ans=0.2
2023-11-21 07:01:43,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1402533.3333333333, ans=0.125
2023-11-21 07:01:55,506 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210400
2023-11-21 07:01:58,243 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6000, loss[loss=0.09385, simple_loss=0.1306, pruned_loss=0.02245, audio_tagging_loss=0.006102, over 14319.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09746, pruned_loss=0.01716, audio_tagging_loss=0.009435, over 3046772.83 frames. ], batch size: 53, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:01:58,243 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 07:02:25,443 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9919, 3.3007, 2.8948, 3.2637, 3.4997, 2.8261, 3.3542, 2.6227],
       device='cuda:2')
2023-11-21 07:02:40,603 INFO [train_asr.py:1253] (2/4) Epoch 18, validation: loss=0.0604, simple_loss=0.05257, pruned_loss=0.00537, audio_tagging_loss=0.02874, over 4681554.00 frames. 
2023-11-21 07:02:40,604 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 07:03:09,963 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 7.991e+01 8.851e+01 9.572e+01 1.607e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 07:03:27,309 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:03:44,126 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210450
2023-11-21 07:03:47,039 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6050, loss[loss=0.08375, simple_loss=0.09847, pruned_loss=0.0234, audio_tagging_loss=0.01111, over 14812.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.097, pruned_loss=0.01715, audio_tagging_loss=0.009421, over 3047407.40 frames. ], batch size: 54, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:03:48,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1403000.0, ans=0.125
2023-11-21 07:03:48,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1403000.0, ans=0.125
2023-11-21 07:03:57,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1403000.0, ans=0.2
2023-11-21 07:03:58,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1403066.6666666667, ans=0.125
2023-11-21 07:04:18,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1403133.3333333333, ans=0.0
2023-11-21 07:04:38,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1403266.6666666667, ans=0.125
2023-11-21 07:04:41,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1403266.6666666667, ans=10.0
2023-11-21 07:04:41,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1403266.6666666667, ans=0.2
2023-11-21 07:04:46,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1403266.6666666667, ans=0.125
2023-11-21 07:04:48,716 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210500
2023-11-21 07:04:51,000 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6100, loss[loss=0.07555, simple_loss=0.1037, pruned_loss=0.01591, audio_tagging_loss=0.007783, over 14368.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09651, pruned_loss=0.01701, audio_tagging_loss=0.00937, over 3046106.93 frames. ], batch size: 53, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:04:51,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1403333.3333333333, ans=0.0
2023-11-21 07:04:55,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1403333.3333333333, ans=0.1
2023-11-21 07:05:09,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.35 vs. limit=15.0
2023-11-21 07:05:20,549 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.632e+01 8.223e+01 8.598e+01 9.128e+01 4.094e+02, threshold=1.720e+02, percent-clipped=1.0
2023-11-21 07:05:53,461 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210550
2023-11-21 07:05:53,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1403600.0, ans=0.0
2023-11-21 07:05:55,824 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6150, loss[loss=0.09007, simple_loss=0.1269, pruned_loss=0.01823, audio_tagging_loss=0.008375, over 16805.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.0969, pruned_loss=0.01714, audio_tagging_loss=0.009401, over 3043417.26 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:05:57,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1403666.6666666667, ans=0.5
2023-11-21 07:06:01,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1403666.6666666667, ans=0.125
2023-11-21 07:06:19,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1403733.3333333333, ans=0.2
2023-11-21 07:06:21,069 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:06:39,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1403866.6666666667, ans=0.025
2023-11-21 07:06:51,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1403933.3333333333, ans=0.125
2023-11-21 07:07:00,659 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210600
2023-11-21 07:07:00,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1403933.3333333333, ans=0.0
2023-11-21 07:07:03,403 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6200, loss[loss=0.06166, simple_loss=0.08191, pruned_loss=0.01066, audio_tagging_loss=0.01004, over 14979.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09667, pruned_loss=0.01719, audio_tagging_loss=0.009502, over 3045097.95 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:07:05,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1404000.0, ans=0.0
2023-11-21 07:07:05,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1404000.0, ans=0.0
2023-11-21 07:07:06,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1404000.0, ans=0.125
2023-11-21 07:07:11,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1404000.0, ans=0.125
2023-11-21 07:07:13,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.66 vs. limit=15.0
2023-11-21 07:07:21,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1404066.6666666667, ans=0.0
2023-11-21 07:07:24,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1404066.6666666667, ans=0.125
2023-11-21 07:07:24,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1404066.6666666667, ans=0.1
2023-11-21 07:07:31,503 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.071e+01 8.570e+01 9.490e+01 1.156e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 07:07:33,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1404133.3333333333, ans=0.0
2023-11-21 07:07:45,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1404200.0, ans=0.125
2023-11-21 07:07:49,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1404200.0, ans=0.125
2023-11-21 07:08:04,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1404266.6666666667, ans=0.125
2023-11-21 07:08:06,207 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210650
2023-11-21 07:08:08,593 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6250, loss[loss=0.05933, simple_loss=0.07689, pruned_loss=0.01305, audio_tagging_loss=0.007833, over 14222.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09738, pruned_loss=0.01731, audio_tagging_loss=0.009619, over 3040469.60 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:08:09,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1404333.3333333333, ans=0.04949747468305833
2023-11-21 07:08:33,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1404466.6666666667, ans=0.125
2023-11-21 07:08:35,312 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.53 vs. limit=15.0
2023-11-21 07:08:37,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1404466.6666666667, ans=0.0
2023-11-21 07:08:39,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1404466.6666666667, ans=0.1
2023-11-21 07:08:51,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1404533.3333333333, ans=0.125
2023-11-21 07:09:01,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1404600.0, ans=0.125
2023-11-21 07:09:10,063 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210700
2023-11-21 07:09:12,478 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6300, loss[loss=0.07506, simple_loss=0.09228, pruned_loss=0.01609, audio_tagging_loss=0.01283, over 15257.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09714, pruned_loss=0.01714, audio_tagging_loss=0.009776, over 3037539.07 frames. ], batch size: 59, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:09:18,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1404666.6666666667, ans=0.0
2023-11-21 07:09:31,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1404733.3333333333, ans=0.0
2023-11-21 07:09:40,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1404800.0, ans=0.09899494936611666
2023-11-21 07:09:41,470 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.208e+01 8.088e+01 8.800e+01 9.638e+01 1.138e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 07:09:43,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1404800.0, ans=0.125
2023-11-21 07:09:43,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1404800.0, ans=0.125
2023-11-21 07:09:47,691 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=12.0
2023-11-21 07:10:00,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1404866.6666666667, ans=0.125
2023-11-21 07:10:13,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.11 vs. limit=22.5
2023-11-21 07:10:16,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210750
2023-11-21 07:10:17,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1405000.0, ans=0.125
2023-11-21 07:10:18,566 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6350, loss[loss=0.0514, simple_loss=0.06545, pruned_loss=0.006711, audio_tagging_loss=0.01196, over 13533.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09643, pruned_loss=0.01697, audio_tagging_loss=0.00979, over 3033389.48 frames. ], batch size: 54, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:10:46,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0
2023-11-21 07:11:00,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1405200.0, ans=0.1
2023-11-21 07:11:12,695 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.82 vs. limit=22.5
2023-11-21 07:11:20,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210800
2023-11-21 07:11:23,335 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6400, loss[loss=0.0903, simple_loss=0.1144, pruned_loss=0.02301, audio_tagging_loss=0.01007, over 15143.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.09611, pruned_loss=0.01692, audio_tagging_loss=0.009848, over 3032326.12 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:11:55,153 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.546e+01 8.103e+01 8.679e+01 9.633e+01 1.173e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 07:12:02,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1405533.3333333333, ans=0.125
2023-11-21 07:12:02,577 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.74 vs. limit=15.0
2023-11-21 07:12:03,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1405533.3333333333, ans=0.125
2023-11-21 07:12:09,522 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:12:14,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1405600.0, ans=0.0
2023-11-21 07:12:25,972 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210850
2023-11-21 07:12:28,236 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6450, loss[loss=0.07806, simple_loss=0.09954, pruned_loss=0.01861, audio_tagging_loss=0.009685, over 15811.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.09655, pruned_loss=0.01691, audio_tagging_loss=0.009886, over 3030896.99 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:12:55,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1405800.0, ans=0.09899494936611666
2023-11-21 07:12:55,556 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-21 07:13:09,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1405866.6666666667, ans=0.1
2023-11-21 07:13:28,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1405933.3333333333, ans=0.0
2023-11-21 07:13:32,038 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210900
2023-11-21 07:13:34,492 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6500, loss[loss=0.07144, simple_loss=0.09859, pruned_loss=0.01468, audio_tagging_loss=0.00747, over 16161.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09609, pruned_loss=0.01692, audio_tagging_loss=0.009977, over 3042380.15 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:13:46,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1406066.6666666667, ans=0.125
2023-11-21 07:13:49,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1406066.6666666667, ans=0.1
2023-11-21 07:13:49,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1406066.6666666667, ans=0.0
2023-11-21 07:13:51,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1406066.6666666667, ans=0.125
2023-11-21 07:14:05,523 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.017e+01 8.785e+01 9.599e+01 1.242e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 07:14:37,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 210950
2023-11-21 07:14:39,597 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6550, loss[loss=0.06382, simple_loss=0.08282, pruned_loss=0.01392, audio_tagging_loss=0.008489, over 15400.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09564, pruned_loss=0.01682, audio_tagging_loss=0.009896, over 3040585.12 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:14:56,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1406400.0, ans=0.0
2023-11-21 07:15:01,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1406400.0, ans=0.125
2023-11-21 07:15:07,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1406466.6666666667, ans=0.125
2023-11-21 07:15:12,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1406466.6666666667, ans=0.0
2023-11-21 07:15:12,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1406466.6666666667, ans=0.05
2023-11-21 07:15:35,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1406600.0, ans=0.0
2023-11-21 07:15:41,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211000
2023-11-21 07:15:43,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1406666.6666666667, ans=0.125
2023-11-21 07:15:44,650 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6600, loss[loss=0.0698, simple_loss=0.09439, pruned_loss=0.01587, audio_tagging_loss=0.006737, over 16399.00 frames. ], tot_loss[loss=0.07474, simple_loss=0.09636, pruned_loss=0.0168, audio_tagging_loss=0.009765, over 3045966.74 frames. ], batch size: 63, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:16:02,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1406733.3333333333, ans=0.0
2023-11-21 07:16:06,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1406733.3333333333, ans=0.125
2023-11-21 07:16:07,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1406733.3333333333, ans=0.0
2023-11-21 07:16:16,221 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 8.176e+01 8.712e+01 9.397e+01 1.234e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 07:16:47,280 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211050
2023-11-21 07:16:50,688 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6650, loss[loss=0.05938, simple_loss=0.07955, pruned_loss=0.009382, audio_tagging_loss=0.01023, over 15470.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09597, pruned_loss=0.01698, audio_tagging_loss=0.00976, over 3048470.35 frames. ], batch size: 58, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:17:05,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1407066.6666666667, ans=0.0
2023-11-21 07:17:15,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1407133.3333333333, ans=0.2
2023-11-21 07:17:28,804 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.96 vs. limit=22.5
2023-11-21 07:17:41,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1407200.0, ans=0.1
2023-11-21 07:17:41,509 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.88 vs. limit=6.0
2023-11-21 07:17:53,711 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211100
2023-11-21 07:17:56,139 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6700, loss[loss=0.0811, simple_loss=0.1097, pruned_loss=0.01824, audio_tagging_loss=0.008035, over 15280.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09526, pruned_loss=0.01686, audio_tagging_loss=0.009766, over 3048094.58 frames. ], batch size: 55, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:18:04,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1407333.3333333333, ans=0.125
2023-11-21 07:18:06,574 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:18:27,205 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.113e+01 8.605e+01 9.333e+01 1.253e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 07:18:32,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1407466.6666666667, ans=10.0
2023-11-21 07:18:33,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1407466.6666666667, ans=0.0
2023-11-21 07:18:38,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1407533.3333333333, ans=0.125
2023-11-21 07:18:52,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1407600.0, ans=0.2
2023-11-21 07:18:54,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1407600.0, ans=0.0
2023-11-21 07:18:58,347 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211150
2023-11-21 07:19:00,702 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6750, loss[loss=0.05524, simple_loss=0.07048, pruned_loss=0.009258, audio_tagging_loss=0.01074, over 14779.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09543, pruned_loss=0.01672, audio_tagging_loss=0.009718, over 3039299.89 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:19:17,288 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.40 vs. limit=22.5
2023-11-21 07:19:27,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1407800.0, ans=0.0
2023-11-21 07:19:36,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1407800.0, ans=0.125
2023-11-21 07:19:40,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=1407866.6666666667, ans=15.0
2023-11-21 07:19:54,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1407933.3333333333, ans=0.2
2023-11-21 07:19:54,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.43 vs. limit=15.0
2023-11-21 07:20:03,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211200
2023-11-21 07:20:06,270 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6800, loss[loss=0.07916, simple_loss=0.1107, pruned_loss=0.01517, audio_tagging_loss=0.008666, over 15769.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09521, pruned_loss=0.01688, audio_tagging_loss=0.009684, over 3034974.85 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:20:09,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1408000.0, ans=0.0
2023-11-21 07:20:14,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1408000.0, ans=0.125
2023-11-21 07:20:26,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.97 vs. limit=10.0
2023-11-21 07:20:36,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1408133.3333333333, ans=0.125
2023-11-21 07:20:37,396 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.078e+01 8.657e+01 9.285e+01 1.100e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 07:20:56,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.50 vs. limit=15.0
2023-11-21 07:21:05,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1408266.6666666667, ans=0.0
2023-11-21 07:21:09,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211250
2023-11-21 07:21:11,850 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6850, loss[loss=0.06924, simple_loss=0.08822, pruned_loss=0.01497, audio_tagging_loss=0.01016, over 15579.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09502, pruned_loss=0.01692, audio_tagging_loss=0.00969, over 3037882.38 frames. ], batch size: 62, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:21:14,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1408333.3333333333, ans=0.125
2023-11-21 07:21:14,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1408333.3333333333, ans=0.0
2023-11-21 07:21:14,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1408333.3333333333, ans=0.2
2023-11-21 07:21:18,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1408333.3333333333, ans=0.125
2023-11-21 07:21:22,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1408333.3333333333, ans=0.0
2023-11-21 07:21:23,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.32 vs. limit=15.0
2023-11-21 07:21:38,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1408466.6666666667, ans=0.125
2023-11-21 07:21:44,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1408466.6666666667, ans=0.125
2023-11-21 07:22:02,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1408533.3333333333, ans=0.0
2023-11-21 07:22:14,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211300
2023-11-21 07:22:15,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1408666.6666666667, ans=0.125
2023-11-21 07:22:16,884 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6900, loss[loss=0.06942, simple_loss=0.08496, pruned_loss=0.01784, audio_tagging_loss=0.009096, over 14617.00 frames. ], tot_loss[loss=0.07461, simple_loss=0.09571, pruned_loss=0.01708, audio_tagging_loss=0.00968, over 3029530.29 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:22:24,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1408666.6666666667, ans=0.0
2023-11-21 07:22:25,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.36 vs. limit=15.0
2023-11-21 07:22:28,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1408733.3333333333, ans=0.125
2023-11-21 07:22:38,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1408733.3333333333, ans=0.125
2023-11-21 07:22:43,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1408800.0, ans=0.0
2023-11-21 07:22:44,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1408800.0, ans=0.2
2023-11-21 07:22:45,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1408800.0, ans=0.125
2023-11-21 07:22:50,579 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.236e+01 8.770e+01 9.429e+01 1.247e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 07:22:52,439 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=12.83 vs. limit=15.0
2023-11-21 07:22:58,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1408866.6666666667, ans=0.125
2023-11-21 07:23:03,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=1408866.6666666667, ans=0.02
2023-11-21 07:23:06,775 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:23:18,985 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211350
2023-11-21 07:23:21,804 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 6950, loss[loss=0.08218, simple_loss=0.1002, pruned_loss=0.02114, audio_tagging_loss=0.01093, over 13676.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09577, pruned_loss=0.01707, audio_tagging_loss=0.009665, over 3030252.48 frames. ], batch size: 54, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:23:24,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=1409000.0, ans=0.2
2023-11-21 07:23:25,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1409000.0, ans=0.0
2023-11-21 07:23:27,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.67 vs. limit=10.0
2023-11-21 07:23:41,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1409066.6666666667, ans=0.0
2023-11-21 07:24:05,308 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.94 vs. limit=15.0
2023-11-21 07:24:21,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1409266.6666666667, ans=0.125
2023-11-21 07:24:25,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211400
2023-11-21 07:24:27,829 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7000, loss[loss=0.1044, simple_loss=0.1216, pruned_loss=0.03182, audio_tagging_loss=0.01174, over 15203.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09548, pruned_loss=0.01694, audio_tagging_loss=0.009722, over 3028829.83 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:24:35,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1409333.3333333333, ans=0.125
2023-11-21 07:24:55,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1409466.6666666667, ans=0.125
2023-11-21 07:24:59,168 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 7.925e+01 8.553e+01 9.217e+01 1.576e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 07:24:59,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1409466.6666666667, ans=0.125
2023-11-21 07:24:59,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1409466.6666666667, ans=0.125
2023-11-21 07:25:14,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=1409533.3333333333, ans=0.1
2023-11-21 07:25:30,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211450
2023-11-21 07:25:32,876 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7050, loss[loss=0.08025, simple_loss=0.1048, pruned_loss=0.01881, audio_tagging_loss=0.009064, over 14092.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.0962, pruned_loss=0.01713, audio_tagging_loss=0.009724, over 3038857.54 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:25:34,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.00 vs. limit=22.5
2023-11-21 07:25:46,448 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-21 07:25:48,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1409733.3333333333, ans=0.125
2023-11-21 07:25:49,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1409733.3333333333, ans=0.1
2023-11-21 07:26:07,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1409800.0, ans=0.1
2023-11-21 07:26:21,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1409866.6666666667, ans=0.2
2023-11-21 07:26:34,571 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211500
2023-11-21 07:26:36,972 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7100, loss[loss=0.06451, simple_loss=0.07463, pruned_loss=0.01607, audio_tagging_loss=0.01112, over 13588.00 frames. ], tot_loss[loss=0.07551, simple_loss=0.09673, pruned_loss=0.01733, audio_tagging_loss=0.009812, over 3039402.74 frames. ], batch size: 53, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:26:37,755 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.07 vs. limit=6.0
2023-11-21 07:26:46,419 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:26:54,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1410066.6666666667, ans=0.2
2023-11-21 07:26:56,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1410066.6666666667, ans=0.07
2023-11-21 07:26:57,120 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.03 vs. limit=15.0
2023-11-21 07:27:10,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.252e+01 8.974e+01 1.013e+02 1.228e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-21 07:27:21,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1410200.0, ans=0.0
2023-11-21 07:27:41,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211550
2023-11-21 07:27:43,606 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7150, loss[loss=0.07101, simple_loss=0.08428, pruned_loss=0.01483, audio_tagging_loss=0.01404, over 14798.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09727, pruned_loss=0.01735, audio_tagging_loss=0.009745, over 3046178.79 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:27:53,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.67 vs. limit=15.0
2023-11-21 07:28:14,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1410466.6666666667, ans=0.125
2023-11-21 07:28:19,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.26 vs. limit=15.0
2023-11-21 07:28:40,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1410600.0, ans=0.125
2023-11-21 07:28:44,686 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=15.0
2023-11-21 07:28:45,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211600
2023-11-21 07:28:48,051 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7200, loss[loss=0.07254, simple_loss=0.08904, pruned_loss=0.01774, audio_tagging_loss=0.01029, over 14744.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09744, pruned_loss=0.01728, audio_tagging_loss=0.009819, over 3041281.64 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:29:07,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1410733.3333333333, ans=0.0
2023-11-21 07:29:14,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1410800.0, ans=0.125
2023-11-21 07:29:14,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1410800.0, ans=0.0
2023-11-21 07:29:15,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=16.85 vs. limit=22.5
2023-11-21 07:29:20,722 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 7.997e+01 8.820e+01 9.312e+01 1.213e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 07:29:39,704 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.77 vs. limit=10.0
2023-11-21 07:29:41,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1410933.3333333333, ans=0.0
2023-11-21 07:29:50,175 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211650
2023-11-21 07:29:52,664 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7250, loss[loss=0.08753, simple_loss=0.1087, pruned_loss=0.02481, audio_tagging_loss=0.008355, over 14687.00 frames. ], tot_loss[loss=0.07595, simple_loss=0.09763, pruned_loss=0.01734, audio_tagging_loss=0.0098, over 3041592.77 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:29:58,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1411000.0, ans=0.125
2023-11-21 07:30:12,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1411066.6666666667, ans=0.0
2023-11-21 07:30:13,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1411066.6666666667, ans=0.0
2023-11-21 07:30:55,398 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211700
2023-11-21 07:30:58,971 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7300, loss[loss=0.06302, simple_loss=0.07447, pruned_loss=0.01132, audio_tagging_loss=0.01446, over 15430.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.09803, pruned_loss=0.01742, audio_tagging_loss=0.009778, over 3049743.80 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:31:00,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.93 vs. limit=15.0
2023-11-21 07:31:08,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1411333.3333333333, ans=0.125
2023-11-21 07:31:18,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=1411400.0, ans=0.1
2023-11-21 07:31:21,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.13 vs. limit=15.0
2023-11-21 07:31:30,110 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.648e+01 8.326e+01 8.844e+01 9.526e+01 1.717e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 07:31:37,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1411533.3333333333, ans=0.0
2023-11-21 07:31:51,327 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0
2023-11-21 07:32:01,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211750
2023-11-21 07:32:03,692 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7350, loss[loss=0.05223, simple_loss=0.0649, pruned_loss=0.006969, audio_tagging_loss=0.01281, over 14616.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09731, pruned_loss=0.01733, audio_tagging_loss=0.0097, over 3047539.38 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:32:04,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1411666.6666666667, ans=0.125
2023-11-21 07:33:02,939 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.95 vs. limit=15.0
2023-11-21 07:33:04,759 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211800
2023-11-21 07:33:07,571 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7400, loss[loss=0.06245, simple_loss=0.0837, pruned_loss=0.01097, audio_tagging_loss=0.00963, over 15242.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09708, pruned_loss=0.01709, audio_tagging_loss=0.009701, over 3048700.72 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:33:40,830 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 7.896e+01 8.684e+01 9.314e+01 1.199e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 07:34:09,124 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.91 vs. limit=15.0
2023-11-21 07:34:09,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211850
2023-11-21 07:34:12,024 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7450, loss[loss=0.08175, simple_loss=0.1078, pruned_loss=0.01992, audio_tagging_loss=0.007953, over 14615.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09843, pruned_loss=0.01735, audio_tagging_loss=0.009631, over 3047783.46 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:34:27,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1412400.0, ans=0.125
2023-11-21 07:34:34,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.44 vs. limit=22.5
2023-11-21 07:34:41,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1412466.6666666667, ans=0.125
2023-11-21 07:34:51,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1412533.3333333333, ans=0.125
2023-11-21 07:34:53,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1412533.3333333333, ans=0.0
2023-11-21 07:35:13,534 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211900
2023-11-21 07:35:14,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.87 vs. limit=10.0
2023-11-21 07:35:15,908 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7500, loss[loss=0.0752, simple_loss=0.09197, pruned_loss=0.01833, audio_tagging_loss=0.01089, over 16007.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09837, pruned_loss=0.01746, audio_tagging_loss=0.009457, over 3051604.51 frames. ], batch size: 60, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:35:18,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1412666.6666666667, ans=0.125
2023-11-21 07:35:27,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1412733.3333333333, ans=0.125
2023-11-21 07:35:47,758 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.064e+01 8.532e+01 9.428e+01 1.202e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 07:35:52,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1412800.0, ans=0.125
2023-11-21 07:35:53,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.31 vs. limit=22.5
2023-11-21 07:36:15,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1412933.3333333333, ans=0.125
2023-11-21 07:36:17,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 211950
2023-11-21 07:36:20,301 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7550, loss[loss=0.08861, simple_loss=0.1138, pruned_loss=0.02448, audio_tagging_loss=0.00721, over 15467.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09731, pruned_loss=0.01729, audio_tagging_loss=0.009452, over 3050512.43 frames. ], batch size: 61, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:36:31,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1413000.0, ans=0.0
2023-11-21 07:37:22,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212000
2023-11-21 07:37:28,039 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7600, loss[loss=0.0697, simple_loss=0.08972, pruned_loss=0.0152, audio_tagging_loss=0.009645, over 17095.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.09707, pruned_loss=0.01728, audio_tagging_loss=0.009436, over 3056429.49 frames. ], batch size: 65, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:37:36,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1413333.3333333333, ans=0.1
2023-11-21 07:37:49,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1413400.0, ans=0.0
2023-11-21 07:38:00,162 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.299e+01 8.757e+01 9.275e+01 1.161e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 07:38:13,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1413533.3333333333, ans=0.125
2023-11-21 07:38:19,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1413600.0, ans=0.125
2023-11-21 07:38:24,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.80 vs. limit=15.0
2023-11-21 07:38:30,244 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212050
2023-11-21 07:38:32,597 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7650, loss[loss=0.05478, simple_loss=0.0597, pruned_loss=0.01257, audio_tagging_loss=0.01236, over 15833.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09627, pruned_loss=0.01715, audio_tagging_loss=0.009507, over 3056111.82 frames. ], batch size: 62, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:38:48,703 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.87 vs. limit=6.0
2023-11-21 07:38:52,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1413733.3333333333, ans=0.2
2023-11-21 07:38:53,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1413733.3333333333, ans=0.0
2023-11-21 07:38:59,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1413800.0, ans=0.125
2023-11-21 07:39:10,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1413800.0, ans=0.2
2023-11-21 07:39:25,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1413933.3333333333, ans=0.0
2023-11-21 07:39:33,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1413933.3333333333, ans=0.125
2023-11-21 07:39:34,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212100
2023-11-21 07:39:37,892 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7700, loss[loss=0.1021, simple_loss=0.1296, pruned_loss=0.02603, audio_tagging_loss=0.01121, over 14332.00 frames. ], tot_loss[loss=0.0749, simple_loss=0.09647, pruned_loss=0.01712, audio_tagging_loss=0.009544, over 3063992.70 frames. ], batch size: 53, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:40:10,453 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.127e+01 8.808e+01 9.594e+01 1.135e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 07:40:17,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.08 vs. limit=15.0
2023-11-21 07:40:40,147 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212150
2023-11-21 07:40:42,680 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7750, loss[loss=0.08287, simple_loss=0.1079, pruned_loss=0.02059, audio_tagging_loss=0.008325, over 15757.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09755, pruned_loss=0.01738, audio_tagging_loss=0.009572, over 3061426.53 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:40:53,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1414333.3333333333, ans=0.125
2023-11-21 07:41:33,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1414533.3333333333, ans=0.125
2023-11-21 07:41:46,869 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212200
2023-11-21 07:41:49,596 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7800, loss[loss=0.07726, simple_loss=0.09747, pruned_loss=0.0179, audio_tagging_loss=0.01062, over 15743.00 frames. ], tot_loss[loss=0.07582, simple_loss=0.0976, pruned_loss=0.01735, audio_tagging_loss=0.009674, over 3060420.40 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:41:53,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1414666.6666666667, ans=0.07
2023-11-21 07:42:16,606 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:42:21,725 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.253e+01 8.940e+01 9.769e+01 1.668e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 07:42:28,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1414866.6666666667, ans=0.0
2023-11-21 07:42:30,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1414866.6666666667, ans=0.125
2023-11-21 07:42:30,861 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.18 vs. limit=12.0
2023-11-21 07:42:39,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1414866.6666666667, ans=0.125
2023-11-21 07:42:49,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1414933.3333333333, ans=0.0
2023-11-21 07:42:51,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212250
2023-11-21 07:42:53,710 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7850, loss[loss=0.06774, simple_loss=0.09679, pruned_loss=0.01225, audio_tagging_loss=0.007092, over 15024.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09786, pruned_loss=0.01725, audio_tagging_loss=0.009747, over 3060904.12 frames. ], batch size: 53, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:42:55,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1415000.0, ans=0.125
2023-11-21 07:43:18,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1415066.6666666667, ans=0.0
2023-11-21 07:43:34,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1415200.0, ans=0.125
2023-11-21 07:43:56,307 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212300
2023-11-21 07:43:59,287 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7900, loss[loss=0.09343, simple_loss=0.1201, pruned_loss=0.02598, audio_tagging_loss=0.007384, over 15902.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09878, pruned_loss=0.01745, audio_tagging_loss=0.009713, over 3056704.70 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:44:02,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1415333.3333333333, ans=0.09899494936611666
2023-11-21 07:44:23,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1415466.6666666667, ans=0.0
2023-11-21 07:44:31,973 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.051e+01 8.734e+01 9.440e+01 1.312e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 07:44:34,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1415466.6666666667, ans=0.2
2023-11-21 07:44:35,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.49 vs. limit=15.0
2023-11-21 07:44:50,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1415600.0, ans=0.95
2023-11-21 07:44:53,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1415600.0, ans=0.0
2023-11-21 07:45:01,045 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212350
2023-11-21 07:45:03,401 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 7950, loss[loss=0.07287, simple_loss=0.09157, pruned_loss=0.01695, audio_tagging_loss=0.01013, over 14714.00 frames. ], tot_loss[loss=0.07674, simple_loss=0.09877, pruned_loss=0.0176, audio_tagging_loss=0.009757, over 3060490.22 frames. ], batch size: 54, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:45:17,044 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:45:33,894 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.17 vs. limit=22.5
2023-11-21 07:45:46,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1415866.6666666667, ans=0.0
2023-11-21 07:45:55,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1415933.3333333333, ans=0.125
2023-11-21 07:45:56,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1415933.3333333333, ans=0.125
2023-11-21 07:45:57,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.11 vs. limit=12.0
2023-11-21 07:46:04,062 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212400
2023-11-21 07:46:06,800 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8000, loss[loss=0.08026, simple_loss=0.1008, pruned_loss=0.01677, audio_tagging_loss=0.01309, over 14257.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09745, pruned_loss=0.01728, audio_tagging_loss=0.00992, over 3058741.43 frames. ], batch size: 54, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:46:40,031 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 7.990e+01 8.488e+01 9.256e+01 1.198e+02, threshold=1.698e+02, percent-clipped=0.0
2023-11-21 07:46:54,265 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.13 vs. limit=6.0
2023-11-21 07:47:07,391 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212450
2023-11-21 07:47:09,803 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8050, loss[loss=0.06908, simple_loss=0.08603, pruned_loss=0.01705, audio_tagging_loss=0.00902, over 14529.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09669, pruned_loss=0.01727, audio_tagging_loss=0.009956, over 3053760.38 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:47:25,795 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.03 vs. limit=15.0
2023-11-21 07:47:45,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1416466.6666666667, ans=10.0
2023-11-21 07:47:48,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1416533.3333333333, ans=0.2
2023-11-21 07:47:49,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.51 vs. limit=10.0
2023-11-21 07:47:56,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.99 vs. limit=15.0
2023-11-21 07:48:06,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1416600.0, ans=0.125
2023-11-21 07:48:13,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1416600.0, ans=0.125
2023-11-21 07:48:14,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212500
2023-11-21 07:48:16,511 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8100, loss[loss=0.06896, simple_loss=0.08709, pruned_loss=0.01634, audio_tagging_loss=0.009071, over 15326.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09714, pruned_loss=0.01738, audio_tagging_loss=0.009866, over 3051179.71 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:48:34,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1416733.3333333333, ans=0.125
2023-11-21 07:48:36,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1416733.3333333333, ans=0.1
2023-11-21 07:48:48,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.164e+01 8.831e+01 9.512e+01 1.309e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 07:49:00,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1416866.6666666667, ans=0.125
2023-11-21 07:49:11,898 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.71 vs. limit=22.5
2023-11-21 07:49:18,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212550
2023-11-21 07:49:18,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1416933.3333333333, ans=0.1
2023-11-21 07:49:20,835 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8150, loss[loss=0.07793, simple_loss=0.09786, pruned_loss=0.01794, audio_tagging_loss=0.01106, over 14991.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09727, pruned_loss=0.01737, audio_tagging_loss=0.009729, over 3051732.36 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:49:23,995 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.87 vs. limit=22.5
2023-11-21 07:49:40,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1417066.6666666667, ans=0.1
2023-11-21 07:49:45,407 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0
2023-11-21 07:49:45,425 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.20 vs. limit=22.5
2023-11-21 07:49:52,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.19 vs. limit=22.5
2023-11-21 07:49:57,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.79 vs. limit=10.0
2023-11-21 07:50:02,483 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0
2023-11-21 07:50:21,545 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212600
2023-11-21 07:50:24,251 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8200, loss[loss=0.07953, simple_loss=0.1095, pruned_loss=0.01553, audio_tagging_loss=0.009243, over 15500.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09804, pruned_loss=0.01759, audio_tagging_loss=0.009538, over 3056314.62 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:50:24,307 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:50:33,194 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.24 vs. limit=22.5
2023-11-21 07:50:35,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1417333.3333333333, ans=0.125
2023-11-21 07:50:46,222 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.30 vs. limit=10.0
2023-11-21 07:50:58,023 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.391e+01 9.092e+01 9.919e+01 1.241e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-21 07:51:15,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1417600.0, ans=0.025
2023-11-21 07:51:22,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1417600.0, ans=0.125
2023-11-21 07:51:26,129 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212650
2023-11-21 07:51:29,808 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8250, loss[loss=0.06461, simple_loss=0.0746, pruned_loss=0.01127, audio_tagging_loss=0.01604, over 15577.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.09794, pruned_loss=0.01772, audio_tagging_loss=0.009627, over 3047279.05 frames. ], batch size: 60, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:51:34,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1417666.6666666667, ans=0.95
2023-11-21 07:51:36,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1417666.6666666667, ans=0.09899494936611666
2023-11-21 07:51:49,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.91 vs. limit=22.5
2023-11-21 07:52:32,638 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212700
2023-11-21 07:52:35,017 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8300, loss[loss=0.0698, simple_loss=0.08032, pruned_loss=0.01767, audio_tagging_loss=0.01196, over 15776.00 frames. ], tot_loss[loss=0.07622, simple_loss=0.09775, pruned_loss=0.01768, audio_tagging_loss=0.009672, over 3049230.78 frames. ], batch size: 61, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:53:10,531 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.350e+01 9.034e+01 9.751e+01 1.328e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 07:53:17,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1418200.0, ans=0.0
2023-11-21 07:53:37,302 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212750
2023-11-21 07:53:39,700 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8350, loss[loss=0.06677, simple_loss=0.08529, pruned_loss=0.0139, audio_tagging_loss=0.01023, over 15757.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09727, pruned_loss=0.01757, audio_tagging_loss=0.009619, over 3043980.13 frames. ], batch size: 58, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 07:54:00,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1418400.0, ans=0.125
2023-11-21 07:54:05,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1418400.0, ans=0.125
2023-11-21 07:54:05,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0
2023-11-21 07:54:16,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1418466.6666666667, ans=0.5
2023-11-21 07:54:42,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212800
2023-11-21 07:54:42,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1418600.0, ans=0.0
2023-11-21 07:54:45,232 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8400, loss[loss=0.06853, simple_loss=0.09606, pruned_loss=0.01275, audio_tagging_loss=0.007746, over 15169.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.0963, pruned_loss=0.01714, audio_tagging_loss=0.009686, over 3038445.86 frames. ], batch size: 59, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:54:45,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1418666.6666666667, ans=0.2
2023-11-21 07:54:49,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1418666.6666666667, ans=0.0
2023-11-21 07:54:54,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1418666.6666666667, ans=0.1
2023-11-21 07:55:01,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1418733.3333333333, ans=0.125
2023-11-21 07:55:07,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1418733.3333333333, ans=0.125
2023-11-21 07:55:16,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1418800.0, ans=0.125
2023-11-21 07:55:21,070 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.837e+01 7.810e+01 8.454e+01 9.398e+01 1.177e+02, threshold=1.691e+02, percent-clipped=0.0
2023-11-21 07:55:22,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1418800.0, ans=0.125
2023-11-21 07:55:23,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1418866.6666666667, ans=0.125
2023-11-21 07:55:44,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1418933.3333333333, ans=0.125
2023-11-21 07:55:44,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1418933.3333333333, ans=0.125
2023-11-21 07:55:49,258 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212850
2023-11-21 07:55:51,655 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8450, loss[loss=0.07289, simple_loss=0.09098, pruned_loss=0.01745, audio_tagging_loss=0.009952, over 15471.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.0971, pruned_loss=0.01731, audio_tagging_loss=0.009679, over 3032142.81 frames. ], batch size: 63, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:56:22,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-21 07:56:45,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1419266.6666666667, ans=0.2
2023-11-21 07:56:53,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212900
2023-11-21 07:56:55,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.61 vs. limit=15.0
2023-11-21 07:56:55,734 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8500, loss[loss=0.08339, simple_loss=0.1095, pruned_loss=0.01935, audio_tagging_loss=0.009284, over 15220.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09769, pruned_loss=0.01748, audio_tagging_loss=0.009701, over 3032554.86 frames. ], batch size: 54, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:56:55,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1419333.3333333333, ans=0.1
2023-11-21 07:57:02,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1419333.3333333333, ans=0.0
2023-11-21 07:57:09,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1419400.0, ans=0.1
2023-11-21 07:57:18,423 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.45 vs. limit=15.0
2023-11-21 07:57:32,213 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.883e+01 8.044e+01 8.694e+01 9.368e+01 1.136e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 07:57:35,397 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.67 vs. limit=15.0
2023-11-21 07:57:38,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1419533.3333333333, ans=0.125
2023-11-21 07:57:43,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1419533.3333333333, ans=0.125
2023-11-21 07:57:45,403 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-21 07:57:58,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 212950
2023-11-21 07:58:01,285 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8550, loss[loss=0.1033, simple_loss=0.1421, pruned_loss=0.02545, audio_tagging_loss=0.006834, over 14717.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.0966, pruned_loss=0.01727, audio_tagging_loss=0.00984, over 3030646.11 frames. ], batch size: 53, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:58:21,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1419733.3333333333, ans=0.125
2023-11-21 07:58:39,230 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.20 vs. limit=15.0
2023-11-21 07:58:53,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.99 vs. limit=22.5
2023-11-21 07:59:03,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213000
2023-11-21 07:59:06,327 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8600, loss[loss=0.07571, simple_loss=0.09281, pruned_loss=0.02045, audio_tagging_loss=0.008853, over 14910.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.0967, pruned_loss=0.01716, audio_tagging_loss=0.009898, over 3029470.69 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:59:23,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1420066.6666666667, ans=0.015
2023-11-21 07:59:40,773 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.296e+01 8.926e+01 9.744e+01 1.392e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 07:59:58,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1420266.6666666667, ans=0.125
2023-11-21 08:00:08,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213050
2023-11-21 08:00:10,614 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8650, loss[loss=0.05985, simple_loss=0.07388, pruned_loss=0.01291, audio_tagging_loss=0.009999, over 16367.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09679, pruned_loss=0.01702, audio_tagging_loss=0.009822, over 3038090.87 frames. ], batch size: 63, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:00:20,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1420333.3333333333, ans=0.125
2023-11-21 08:00:36,842 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.77 vs. limit=15.0
2023-11-21 08:00:55,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1420533.3333333333, ans=0.0
2023-11-21 08:01:12,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213100
2023-11-21 08:01:12,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1420600.0, ans=0.0
2023-11-21 08:01:15,616 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8700, loss[loss=0.1, simple_loss=0.1279, pruned_loss=0.02647, audio_tagging_loss=0.009586, over 14847.00 frames. ], tot_loss[loss=0.076, simple_loss=0.09773, pruned_loss=0.01733, audio_tagging_loss=0.009799, over 3052522.61 frames. ], batch size: 53, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:01:24,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1420666.6666666667, ans=0.0
2023-11-21 08:01:52,035 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.710e+01 8.075e+01 9.054e+01 1.029e+02 1.299e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-21 08:01:58,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1420866.6666666667, ans=0.1
2023-11-21 08:02:01,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1420866.6666666667, ans=0.0
2023-11-21 08:02:05,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1420866.6666666667, ans=0.125
2023-11-21 08:02:13,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1420933.3333333333, ans=0.125
2023-11-21 08:02:18,935 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213150
2023-11-21 08:02:21,305 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8750, loss[loss=0.06052, simple_loss=0.06935, pruned_loss=0.01335, audio_tagging_loss=0.0125, over 15050.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09845, pruned_loss=0.01741, audio_tagging_loss=0.009895, over 3055746.79 frames. ], batch size: 57, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:02:35,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1421066.6666666667, ans=0.125
2023-11-21 08:03:13,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1421266.6666666667, ans=0.025
2023-11-21 08:03:23,198 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213200
2023-11-21 08:03:26,466 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8800, loss[loss=0.1016, simple_loss=0.1284, pruned_loss=0.02807, audio_tagging_loss=0.009378, over 16351.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.1, pruned_loss=0.01767, audio_tagging_loss=0.009788, over 3058154.48 frames. ], batch size: 57, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 08:03:27,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.51 vs. limit=22.5
2023-11-21 08:03:35,476 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0
2023-11-21 08:03:48,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1421400.0, ans=0.1
2023-11-21 08:04:03,666 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.113e+01 8.751e+01 9.814e+01 1.127e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 08:04:04,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1421533.3333333333, ans=0.2
2023-11-21 08:04:22,252 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.43 vs. limit=15.0
2023-11-21 08:04:27,667 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213250
2023-11-21 08:04:29,979 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8850, loss[loss=0.07872, simple_loss=0.1014, pruned_loss=0.01885, audio_tagging_loss=0.009151, over 15428.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.0997, pruned_loss=0.01748, audio_tagging_loss=0.009748, over 3060253.54 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:04:43,089 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:04:43,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1421733.3333333333, ans=0.0
2023-11-21 08:05:04,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1421800.0, ans=0.2
2023-11-21 08:05:19,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1421866.6666666667, ans=0.95
2023-11-21 08:05:33,396 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213300
2023-11-21 08:05:33,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1421933.3333333333, ans=0.125
2023-11-21 08:05:35,832 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8900, loss[loss=0.07326, simple_loss=0.09533, pruned_loss=0.01682, audio_tagging_loss=0.008769, over 14737.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09924, pruned_loss=0.01735, audio_tagging_loss=0.00965, over 3065819.52 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:05:47,641 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.67 vs. limit=15.0
2023-11-21 08:06:02,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1422133.3333333333, ans=0.2
2023-11-21 08:06:12,890 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.059e+01 8.546e+01 9.604e+01 1.318e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-21 08:06:18,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1422200.0, ans=0.5
2023-11-21 08:06:20,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1422200.0, ans=0.0
2023-11-21 08:06:37,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213350
2023-11-21 08:06:40,323 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 8950, loss[loss=0.09255, simple_loss=0.1233, pruned_loss=0.02446, audio_tagging_loss=0.006459, over 15378.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.09876, pruned_loss=0.01735, audio_tagging_loss=0.009486, over 3065090.44 frames. ], batch size: 58, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:06:43,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1422333.3333333333, ans=0.125
2023-11-21 08:06:49,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1422333.3333333333, ans=0.0
2023-11-21 08:07:28,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.94 vs. limit=15.0
2023-11-21 08:07:35,075 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.86 vs. limit=15.0
2023-11-21 08:07:42,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213400
2023-11-21 08:07:45,030 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9000, loss[loss=0.07976, simple_loss=0.1051, pruned_loss=0.01653, audio_tagging_loss=0.01069, over 14681.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.0978, pruned_loss=0.01728, audio_tagging_loss=0.009499, over 3061024.32 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:07:45,031 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 08:08:25,297 INFO [train_asr.py:1253] (2/4) Epoch 18, validation: loss=0.06098, simple_loss=0.05248, pruned_loss=0.005341, audio_tagging_loss=0.02939, over 4681554.00 frames. 
2023-11-21 08:08:25,298 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 08:08:25,930 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.15 vs. limit=15.0
2023-11-21 08:08:32,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1422666.6666666667, ans=0.035
2023-11-21 08:08:36,821 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=12.0
2023-11-21 08:08:37,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1422733.3333333333, ans=0.1
2023-11-21 08:09:01,759 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.253e+01 9.144e+01 1.038e+02 1.327e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-21 08:09:03,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1422866.6666666667, ans=0.125
2023-11-21 08:09:26,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213450
2023-11-21 08:09:29,092 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9050, loss[loss=0.09028, simple_loss=0.1124, pruned_loss=0.02511, audio_tagging_loss=0.008997, over 15441.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09815, pruned_loss=0.01737, audio_tagging_loss=0.00948, over 3058849.72 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:09:47,108 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.75 vs. limit=15.0
2023-11-21 08:09:52,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1423066.6666666667, ans=0.125
2023-11-21 08:10:24,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.10 vs. limit=15.0
2023-11-21 08:10:31,387 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213500
2023-11-21 08:10:33,717 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9100, loss[loss=0.07466, simple_loss=0.09044, pruned_loss=0.01992, audio_tagging_loss=0.00952, over 14216.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09685, pruned_loss=0.0171, audio_tagging_loss=0.009591, over 3056861.04 frames. ], batch size: 54, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:10:38,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1423333.3333333333, ans=0.125
2023-11-21 08:11:11,201 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.196e+01 8.647e+01 9.263e+01 1.186e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-21 08:11:11,884 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0
2023-11-21 08:11:20,033 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.66 vs. limit=6.0
2023-11-21 08:11:22,567 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.57 vs. limit=15.0
2023-11-21 08:11:37,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213550
2023-11-21 08:11:40,079 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9150, loss[loss=0.06705, simple_loss=0.07554, pruned_loss=0.01663, audio_tagging_loss=0.01266, over 15975.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09664, pruned_loss=0.01701, audio_tagging_loss=0.009611, over 3052811.22 frames. ], batch size: 64, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:12:00,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1423733.3333333333, ans=0.0
2023-11-21 08:12:31,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1423933.3333333333, ans=0.0
2023-11-21 08:12:37,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1423933.3333333333, ans=0.0
2023-11-21 08:12:42,554 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213600
2023-11-21 08:12:45,246 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9200, loss[loss=0.0763, simple_loss=0.1045, pruned_loss=0.01317, audio_tagging_loss=0.0109, over 15660.00 frames. ], tot_loss[loss=0.07501, simple_loss=0.0967, pruned_loss=0.01706, audio_tagging_loss=0.009593, over 3049911.12 frames. ], batch size: 59, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 08:12:53,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1424000.0, ans=0.1
2023-11-21 08:12:56,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1424066.6666666667, ans=0.125
2023-11-21 08:13:07,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1424066.6666666667, ans=0.0
2023-11-21 08:13:09,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1424066.6666666667, ans=0.125
2023-11-21 08:13:15,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=13.00 vs. limit=15.0
2023-11-21 08:13:23,944 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 7.966e+01 8.700e+01 9.351e+01 1.190e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 08:13:38,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1424266.6666666667, ans=0.2
2023-11-21 08:13:47,560 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213650
2023-11-21 08:13:49,895 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9250, loss[loss=0.09035, simple_loss=0.1072, pruned_loss=0.02486, audio_tagging_loss=0.01189, over 15705.00 frames. ], tot_loss[loss=0.0749, simple_loss=0.09651, pruned_loss=0.01708, audio_tagging_loss=0.009572, over 3055817.13 frames. ], batch size: 59, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:14:20,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1424466.6666666667, ans=0.125
2023-11-21 08:14:24,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1424466.6666666667, ans=0.2
2023-11-21 08:14:27,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.76 vs. limit=22.5
2023-11-21 08:14:40,321 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:14:43,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1424600.0, ans=0.125
2023-11-21 08:14:52,606 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213700
2023-11-21 08:14:54,994 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9300, loss[loss=0.06247, simple_loss=0.08244, pruned_loss=0.01339, audio_tagging_loss=0.00786, over 15187.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09722, pruned_loss=0.01719, audio_tagging_loss=0.009532, over 3059805.04 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:15:08,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1424733.3333333333, ans=0.1
2023-11-21 08:15:24,587 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.88 vs. limit=22.5
2023-11-21 08:15:31,886 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.285e+01 7.980e+01 8.564e+01 9.426e+01 1.371e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 08:15:42,071 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.64 vs. limit=15.0
2023-11-21 08:15:57,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213750
2023-11-21 08:15:59,935 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9350, loss[loss=0.08707, simple_loss=0.1093, pruned_loss=0.02318, audio_tagging_loss=0.009254, over 15776.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09736, pruned_loss=0.01737, audio_tagging_loss=0.009571, over 3064159.05 frames. ], batch size: 60, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:16:04,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1425000.0, ans=0.125
2023-11-21 08:16:09,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1425000.0, ans=0.0
2023-11-21 08:16:12,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1425066.6666666667, ans=0.2
2023-11-21 08:16:12,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1425066.6666666667, ans=0.1
2023-11-21 08:16:19,451 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0
2023-11-21 08:16:20,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1425066.6666666667, ans=0.125
2023-11-21 08:16:26,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1425133.3333333333, ans=0.0
2023-11-21 08:16:43,883 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:16:51,620 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:16:56,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1425266.6666666667, ans=0.1
2023-11-21 08:17:02,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213800
2023-11-21 08:17:05,137 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9400, loss[loss=0.06583, simple_loss=0.08485, pruned_loss=0.0129, audio_tagging_loss=0.01051, over 14987.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09709, pruned_loss=0.01746, audio_tagging_loss=0.00972, over 3054268.52 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:17:30,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1425400.0, ans=0.2
2023-11-21 08:17:39,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.36 vs. limit=15.0
2023-11-21 08:17:43,420 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 8.186e+01 8.902e+01 9.704e+01 1.344e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 08:17:56,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1425600.0, ans=0.0
2023-11-21 08:18:05,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1425600.0, ans=0.0
2023-11-21 08:18:08,521 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:18:08,592 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213850
2023-11-21 08:18:10,873 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9450, loss[loss=0.09245, simple_loss=0.1154, pruned_loss=0.02454, audio_tagging_loss=0.01021, over 15357.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09687, pruned_loss=0.01751, audio_tagging_loss=0.009778, over 3058250.82 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:18:20,178 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.75 vs. limit=15.0
2023-11-21 08:18:24,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1425733.3333333333, ans=0.0
2023-11-21 08:18:57,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1425866.6666666667, ans=0.0
2023-11-21 08:19:03,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1425933.3333333333, ans=0.125
2023-11-21 08:19:05,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1425933.3333333333, ans=0.0
2023-11-21 08:19:13,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213900
2023-11-21 08:19:16,295 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9500, loss[loss=0.08775, simple_loss=0.1134, pruned_loss=0.01995, audio_tagging_loss=0.01112, over 15380.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.09562, pruned_loss=0.01716, audio_tagging_loss=0.009849, over 3052953.65 frames. ], batch size: 53, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:19:21,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1426000.0, ans=0.125
2023-11-21 08:19:53,771 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.280e+01 9.020e+01 9.623e+01 1.395e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-21 08:19:54,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1426200.0, ans=0.125
2023-11-21 08:19:58,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1426200.0, ans=0.0
2023-11-21 08:20:02,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1426200.0, ans=0.125
2023-11-21 08:20:09,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1426266.6666666667, ans=0.125
2023-11-21 08:20:10,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1426266.6666666667, ans=10.0
2023-11-21 08:20:16,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1426266.6666666667, ans=0.125
2023-11-21 08:20:17,984 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 213950
2023-11-21 08:20:20,264 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9550, loss[loss=0.0718, simple_loss=0.0872, pruned_loss=0.01728, audio_tagging_loss=0.01092, over 15776.00 frames. ], tot_loss[loss=0.07576, simple_loss=0.0965, pruned_loss=0.01754, audio_tagging_loss=0.009967, over 3050954.36 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:20:26,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1426333.3333333333, ans=0.2
2023-11-21 08:20:37,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1426400.0, ans=0.2
2023-11-21 08:20:48,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1426466.6666666667, ans=0.1
2023-11-21 08:20:56,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1426466.6666666667, ans=0.125
2023-11-21 08:21:04,122 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.17 vs. limit=15.0
2023-11-21 08:21:09,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1426533.3333333333, ans=0.125
2023-11-21 08:21:10,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1426533.3333333333, ans=0.125
2023-11-21 08:21:11,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1426600.0, ans=0.0
2023-11-21 08:21:22,677 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214000
2023-11-21 08:21:25,355 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9600, loss[loss=0.07609, simple_loss=0.1062, pruned_loss=0.01415, audio_tagging_loss=0.008848, over 14378.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.09753, pruned_loss=0.01751, audio_tagging_loss=0.009934, over 3054381.54 frames. ], batch size: 54, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:21:40,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1426733.3333333333, ans=0.2
2023-11-21 08:21:50,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1426800.0, ans=0.1
2023-11-21 08:21:57,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1426800.0, ans=0.125
2023-11-21 08:21:58,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.57 vs. limit=15.0
2023-11-21 08:21:59,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1426800.0, ans=0.125
2023-11-21 08:22:02,942 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.086e+01 8.937e+01 9.718e+01 1.374e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-21 08:22:13,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.81 vs. limit=15.0
2023-11-21 08:22:29,418 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214050
2023-11-21 08:22:31,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1427000.0, ans=22.5
2023-11-21 08:22:31,877 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9650, loss[loss=0.07677, simple_loss=0.0999, pruned_loss=0.01856, audio_tagging_loss=0.008266, over 15124.00 frames. ], tot_loss[loss=0.07622, simple_loss=0.09761, pruned_loss=0.01753, audio_tagging_loss=0.009879, over 3053426.38 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:22:57,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1427133.3333333333, ans=0.1
2023-11-21 08:22:58,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1427133.3333333333, ans=0.125
2023-11-21 08:23:07,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1427133.3333333333, ans=0.05
2023-11-21 08:23:07,969 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.04 vs. limit=15.0
2023-11-21 08:23:23,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.93 vs. limit=15.0
2023-11-21 08:23:34,111 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214100
2023-11-21 08:23:36,427 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9700, loss[loss=0.0941, simple_loss=0.1332, pruned_loss=0.01987, audio_tagging_loss=0.007619, over 15242.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09721, pruned_loss=0.01736, audio_tagging_loss=0.009762, over 3049896.74 frames. ], batch size: 54, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:24:16,178 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.525e+01 8.096e+01 8.550e+01 9.320e+01 1.224e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-21 08:24:21,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1427533.3333333333, ans=0.125
2023-11-21 08:24:38,980 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214150
2023-11-21 08:24:40,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1427666.6666666667, ans=0.0
2023-11-21 08:24:41,412 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9750, loss[loss=0.07568, simple_loss=0.09619, pruned_loss=0.01962, audio_tagging_loss=0.007966, over 15184.00 frames. ], tot_loss[loss=0.07503, simple_loss=0.09622, pruned_loss=0.01728, audio_tagging_loss=0.009644, over 3046063.19 frames. ], batch size: 61, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:24:44,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-21 08:24:54,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1427733.3333333333, ans=0.125
2023-11-21 08:25:06,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1427733.3333333333, ans=0.125
2023-11-21 08:25:16,366 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.44 vs. limit=22.5
2023-11-21 08:25:18,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1427800.0, ans=0.125
2023-11-21 08:25:31,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1427866.6666666667, ans=0.2
2023-11-21 08:25:32,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1427933.3333333333, ans=0.0
2023-11-21 08:25:38,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1427933.3333333333, ans=0.2
2023-11-21 08:25:41,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1427933.3333333333, ans=0.125
2023-11-21 08:25:42,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1427933.3333333333, ans=0.125
2023-11-21 08:25:45,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214200
2023-11-21 08:25:47,816 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9800, loss[loss=0.07604, simple_loss=0.1018, pruned_loss=0.01479, audio_tagging_loss=0.01033, over 14965.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.09668, pruned_loss=0.01741, audio_tagging_loss=0.009501, over 3043865.34 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:26:13,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1428133.3333333333, ans=0.05
2023-11-21 08:26:16,681 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.11 vs. limit=10.0
2023-11-21 08:26:24,458 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.73 vs. limit=12.0
2023-11-21 08:26:25,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1428200.0, ans=0.1
2023-11-21 08:26:26,811 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.204e+01 8.669e+01 9.345e+01 1.280e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 08:26:44,304 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:26:49,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1428266.6666666667, ans=0.0
2023-11-21 08:26:50,521 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214250
2023-11-21 08:26:53,003 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9850, loss[loss=0.05829, simple_loss=0.07486, pruned_loss=0.01005, audio_tagging_loss=0.01081, over 14291.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.0965, pruned_loss=0.0172, audio_tagging_loss=0.009505, over 3044504.43 frames. ], batch size: 54, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:27:15,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1428400.0, ans=0.2
2023-11-21 08:27:30,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1428466.6666666667, ans=0.1
2023-11-21 08:27:38,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1428533.3333333333, ans=0.015
2023-11-21 08:27:55,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214300
2023-11-21 08:27:57,977 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9900, loss[loss=0.08119, simple_loss=0.1014, pruned_loss=0.01835, audio_tagging_loss=0.01214, over 14524.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.09743, pruned_loss=0.01734, audio_tagging_loss=0.009437, over 3048192.31 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:28:37,141 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.117e+01 8.820e+01 9.379e+01 1.222e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 08:28:43,022 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.76 vs. limit=15.0
2023-11-21 08:28:43,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1428866.6666666667, ans=0.125
2023-11-21 08:28:44,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1428866.6666666667, ans=0.0
2023-11-21 08:28:47,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1428866.6666666667, ans=0.1
2023-11-21 08:28:52,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1428933.3333333333, ans=0.0
2023-11-21 08:29:01,232 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214350
2023-11-21 08:29:03,559 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 9950, loss[loss=0.05051, simple_loss=0.06317, pruned_loss=0.007771, audio_tagging_loss=0.01116, over 13250.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09703, pruned_loss=0.01728, audio_tagging_loss=0.009402, over 3048562.06 frames. ], batch size: 52, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:29:08,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1429000.0, ans=0.2
2023-11-21 08:29:15,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1429066.6666666667, ans=0.125
2023-11-21 08:29:19,591 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.52 vs. limit=15.0
2023-11-21 08:29:28,445 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.15 vs. limit=10.0
2023-11-21 08:29:33,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1429133.3333333333, ans=0.125
2023-11-21 08:30:06,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214400
2023-11-21 08:30:07,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1429333.3333333333, ans=0.125
2023-11-21 08:30:08,810 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10000, loss[loss=0.09425, simple_loss=0.1306, pruned_loss=0.02043, audio_tagging_loss=0.008501, over 15468.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09701, pruned_loss=0.01715, audio_tagging_loss=0.009446, over 3045241.49 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:30:29,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1429400.0, ans=0.125
2023-11-21 08:30:47,793 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.046e+01 8.852e+01 9.481e+01 1.142e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 08:30:50,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1429533.3333333333, ans=0.125
2023-11-21 08:30:52,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.19 vs. limit=22.5
2023-11-21 08:30:53,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.52 vs. limit=15.0
2023-11-21 08:30:59,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1429600.0, ans=0.125
2023-11-21 08:31:10,698 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214450
2023-11-21 08:31:12,994 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10050, loss[loss=0.07886, simple_loss=0.1066, pruned_loss=0.01726, audio_tagging_loss=0.008303, over 15493.00 frames. ], tot_loss[loss=0.07523, simple_loss=0.09704, pruned_loss=0.01721, audio_tagging_loss=0.009497, over 3046530.70 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:31:14,767 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.39 vs. limit=12.0
2023-11-21 08:31:20,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1429666.6666666667, ans=0.125
2023-11-21 08:31:43,998 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.06 vs. limit=10.0
2023-11-21 08:31:57,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1429866.6666666667, ans=0.125
2023-11-21 08:32:00,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1429866.6666666667, ans=0.0
2023-11-21 08:32:16,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214500
2023-11-21 08:32:19,118 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10100, loss[loss=0.07234, simple_loss=0.08355, pruned_loss=0.01714, audio_tagging_loss=0.01342, over 14899.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09788, pruned_loss=0.01736, audio_tagging_loss=0.009567, over 3052697.13 frames. ], batch size: 59, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:32:41,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1430066.6666666667, ans=0.1
2023-11-21 08:32:41,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1430066.6666666667, ans=0.125
2023-11-21 08:32:58,372 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.203e+01 8.739e+01 9.472e+01 1.226e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 08:33:08,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1430200.0, ans=0.125
2023-11-21 08:33:09,616 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:33:20,717 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214550
2023-11-21 08:33:22,992 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10150, loss[loss=0.0899, simple_loss=0.1154, pruned_loss=0.02315, audio_tagging_loss=0.009027, over 16902.00 frames. ], tot_loss[loss=0.07582, simple_loss=0.09781, pruned_loss=0.01728, audio_tagging_loss=0.009633, over 3057055.31 frames. ], batch size: 62, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:33:48,158 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.43 vs. limit=15.0
2023-11-21 08:33:51,634 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:33:53,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1430466.6666666667, ans=0.125
2023-11-21 08:34:02,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1430533.3333333333, ans=10.0
2023-11-21 08:34:07,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1430533.3333333333, ans=0.125
2023-11-21 08:34:12,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1430533.3333333333, ans=0.125
2023-11-21 08:34:13,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.31 vs. limit=15.0
2023-11-21 08:34:13,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1430600.0, ans=0.125
2023-11-21 08:34:25,589 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214600
2023-11-21 08:34:28,258 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10200, loss[loss=0.07557, simple_loss=0.0954, pruned_loss=0.017, audio_tagging_loss=0.01087, over 14869.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09764, pruned_loss=0.01723, audio_tagging_loss=0.009624, over 3052368.38 frames. ], batch size: 54, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:34:29,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1430666.6666666667, ans=0.0
2023-11-21 08:34:39,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1430666.6666666667, ans=0.0
2023-11-21 08:34:51,442 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:35:02,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1430800.0, ans=0.0
2023-11-21 08:35:03,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1430800.0, ans=0.125
2023-11-21 08:35:08,568 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.283e+01 8.036e+01 8.614e+01 9.487e+01 2.302e+02, threshold=1.723e+02, percent-clipped=1.0
2023-11-21 08:35:11,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1430866.6666666667, ans=0.125
2023-11-21 08:35:16,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.12 vs. limit=15.0
2023-11-21 08:35:31,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214650
2023-11-21 08:35:32,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1431000.0, ans=0.5
2023-11-21 08:35:33,533 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10250, loss[loss=0.06648, simple_loss=0.09054, pruned_loss=0.012, audio_tagging_loss=0.009212, over 15242.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09763, pruned_loss=0.01708, audio_tagging_loss=0.009583, over 3062323.24 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:35:53,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1431066.6666666667, ans=0.125
2023-11-21 08:35:55,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1431066.6666666667, ans=0.125
2023-11-21 08:36:36,443 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214700
2023-11-21 08:36:38,830 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10300, loss[loss=0.06692, simple_loss=0.07546, pruned_loss=0.01668, audio_tagging_loss=0.01251, over 13552.00 frames. ], tot_loss[loss=0.07597, simple_loss=0.09811, pruned_loss=0.01726, audio_tagging_loss=0.009662, over 3065534.91 frames. ], batch size: 52, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:36:54,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1431400.0, ans=0.07
2023-11-21 08:36:59,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1431400.0, ans=0.125
2023-11-21 08:37:06,673 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.12 vs. limit=15.0
2023-11-21 08:37:11,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1431466.6666666667, ans=0.125
2023-11-21 08:37:19,815 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.659e+01 8.199e+01 8.798e+01 9.773e+01 1.568e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 08:37:36,307 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:37:37,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1431600.0, ans=0.125
2023-11-21 08:37:41,037 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214750
2023-11-21 08:37:43,449 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10350, loss[loss=0.07133, simple_loss=0.08622, pruned_loss=0.015, audio_tagging_loss=0.01322, over 14965.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09872, pruned_loss=0.01729, audio_tagging_loss=0.009725, over 3063703.26 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:37:43,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1431666.6666666667, ans=0.0
2023-11-21 08:37:53,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1431666.6666666667, ans=0.125
2023-11-21 08:38:01,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1431733.3333333333, ans=0.125
2023-11-21 08:38:16,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1431800.0, ans=0.0
2023-11-21 08:38:30,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1431866.6666666667, ans=0.125
2023-11-21 08:38:32,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1431866.6666666667, ans=0.125
2023-11-21 08:38:46,809 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214800
2023-11-21 08:38:49,888 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10400, loss[loss=0.04806, simple_loss=0.05724, pruned_loss=0.009636, audio_tagging_loss=0.009801, over 14302.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09832, pruned_loss=0.01724, audio_tagging_loss=0.009736, over 3053866.10 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:39:00,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1432000.0, ans=0.07
2023-11-21 08:39:16,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1432133.3333333333, ans=0.025
2023-11-21 08:39:24,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.33 vs. limit=12.0
2023-11-21 08:39:31,171 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.245e+01 8.104e+01 8.738e+01 9.653e+01 1.281e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 08:39:42,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1432266.6666666667, ans=0.125
2023-11-21 08:39:52,337 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214850
2023-11-21 08:39:54,710 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10450, loss[loss=0.07235, simple_loss=0.08481, pruned_loss=0.01485, audio_tagging_loss=0.01509, over 15916.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09815, pruned_loss=0.01718, audio_tagging_loss=0.009795, over 3050502.81 frames. ], batch size: 59, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:40:02,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1432333.3333333333, ans=0.0
2023-11-21 08:40:13,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1432400.0, ans=0.125
2023-11-21 08:40:13,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1432400.0, ans=0.125
2023-11-21 08:40:27,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1432466.6666666667, ans=0.0
2023-11-21 08:40:34,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1432533.3333333333, ans=0.04949747468305833
2023-11-21 08:40:39,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1432533.3333333333, ans=0.125
2023-11-21 08:40:56,650 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214900
2023-11-21 08:40:58,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1432666.6666666667, ans=0.0
2023-11-21 08:40:59,118 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10500, loss[loss=0.06292, simple_loss=0.07152, pruned_loss=0.01478, audio_tagging_loss=0.01238, over 14827.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.0973, pruned_loss=0.01714, audio_tagging_loss=0.009752, over 3046798.40 frames. ], batch size: 54, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:41:06,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1432666.6666666667, ans=0.0
2023-11-21 08:41:13,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1432733.3333333333, ans=0.1
2023-11-21 08:41:40,105 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.754e+01 7.879e+01 8.434e+01 9.185e+01 1.259e+02, threshold=1.687e+02, percent-clipped=0.0
2023-11-21 08:41:47,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1432866.6666666667, ans=0.125
2023-11-21 08:41:55,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.56 vs. limit=6.0
2023-11-21 08:42:01,012 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 214950
2023-11-21 08:42:03,546 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10550, loss[loss=0.06823, simple_loss=0.09347, pruned_loss=0.01343, audio_tagging_loss=0.00806, over 14686.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09815, pruned_loss=0.01747, audio_tagging_loss=0.009615, over 3050047.51 frames. ], batch size: 54, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:42:16,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1433066.6666666667, ans=0.125
2023-11-21 08:42:25,212 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.61 vs. limit=15.0
2023-11-21 08:42:50,575 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.62 vs. limit=15.0
2023-11-21 08:42:56,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1433266.6666666667, ans=0.0
2023-11-21 08:43:06,205 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215000
2023-11-21 08:43:06,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1433266.6666666667, ans=0.1
2023-11-21 08:43:08,941 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10600, loss[loss=0.09264, simple_loss=0.1259, pruned_loss=0.02237, audio_tagging_loss=0.007332, over 15079.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09877, pruned_loss=0.01748, audio_tagging_loss=0.009501, over 3057041.80 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:43:20,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1433400.0, ans=0.125
2023-11-21 08:43:25,086 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:43:31,339 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.84 vs. limit=22.5
2023-11-21 08:43:33,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1433466.6666666667, ans=0.0
2023-11-21 08:43:43,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1433466.6666666667, ans=0.1
2023-11-21 08:43:51,458 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.274e+01 8.887e+01 9.806e+01 1.375e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 08:43:51,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1433533.3333333333, ans=0.2
2023-11-21 08:44:01,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1433600.0, ans=0.07
2023-11-21 08:44:09,897 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215050
2023-11-21 08:44:12,276 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10650, loss[loss=0.05131, simple_loss=0.05857, pruned_loss=0.00897, audio_tagging_loss=0.01305, over 16777.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.09829, pruned_loss=0.0176, audio_tagging_loss=0.009439, over 3057483.02 frames. ], batch size: 65, lr: 3.77e-03, grad_scale: 8.0
2023-11-21 08:44:13,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1433666.6666666667, ans=0.125
2023-11-21 08:44:15,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1433666.6666666667, ans=0.0
2023-11-21 08:44:23,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1433733.3333333333, ans=0.2
2023-11-21 08:44:29,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1433733.3333333333, ans=0.125
2023-11-21 08:45:07,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1433933.3333333333, ans=0.2
2023-11-21 08:45:15,005 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215100
2023-11-21 08:45:17,249 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10700, loss[loss=0.06286, simple_loss=0.07957, pruned_loss=0.01166, audio_tagging_loss=0.01141, over 15915.00 frames. ], tot_loss[loss=0.07689, simple_loss=0.0995, pruned_loss=0.01783, audio_tagging_loss=0.009312, over 3058357.26 frames. ], batch size: 60, lr: 3.77e-03, grad_scale: 8.0
2023-11-21 08:45:18,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1434000.0, ans=0.0
2023-11-21 08:45:39,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1434066.6666666667, ans=0.0
2023-11-21 08:45:59,558 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.05 vs. limit=12.0
2023-11-21 08:46:00,183 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.074e+01 8.707e+01 9.700e+01 1.286e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-21 08:46:00,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1434200.0, ans=0.125
2023-11-21 08:46:16,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1434266.6666666667, ans=0.125
2023-11-21 08:46:19,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1434266.6666666667, ans=0.125
2023-11-21 08:46:21,380 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215150
2023-11-21 08:46:23,789 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10750, loss[loss=0.091, simple_loss=0.1154, pruned_loss=0.02683, audio_tagging_loss=0.006492, over 15716.00 frames. ], tot_loss[loss=0.07648, simple_loss=0.09896, pruned_loss=0.01758, audio_tagging_loss=0.009413, over 3060148.22 frames. ], batch size: 57, lr: 3.77e-03, grad_scale: 8.0
2023-11-21 08:46:24,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1434333.3333333333, ans=0.1
2023-11-21 08:46:41,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1434400.0, ans=0.125
2023-11-21 08:46:52,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1434466.6666666667, ans=0.1
2023-11-21 08:46:57,517 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.68 vs. limit=15.0
2023-11-21 08:47:20,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1434600.0, ans=0.07
2023-11-21 08:47:25,635 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215200
2023-11-21 08:47:28,315 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10800, loss[loss=0.09135, simple_loss=0.1164, pruned_loss=0.0235, audio_tagging_loss=0.00964, over 15109.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09818, pruned_loss=0.0173, audio_tagging_loss=0.009468, over 3059590.78 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:47:44,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1434733.3333333333, ans=0.2
2023-11-21 08:47:54,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1434800.0, ans=0.1
2023-11-21 08:48:11,101 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.512e+01 8.195e+01 8.985e+01 9.543e+01 1.112e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-21 08:48:17,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1434866.6666666667, ans=0.1
2023-11-21 08:48:20,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1434933.3333333333, ans=0.125
2023-11-21 08:48:22,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1434933.3333333333, ans=0.0
2023-11-21 08:48:29,582 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215250
2023-11-21 08:48:32,529 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10850, loss[loss=0.0702, simple_loss=0.0936, pruned_loss=0.01302, audio_tagging_loss=0.01038, over 15106.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09866, pruned_loss=0.01729, audio_tagging_loss=0.009506, over 3055636.81 frames. ], batch size: 58, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:48:43,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.81 vs. limit=22.5
2023-11-21 08:48:58,742 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.80 vs. limit=6.0
2023-11-21 08:48:59,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1435133.3333333333, ans=0.125
2023-11-21 08:49:15,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1435200.0, ans=0.0
2023-11-21 08:49:33,031 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:49:35,531 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215300
2023-11-21 08:49:38,444 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10900, loss[loss=0.08974, simple_loss=0.1174, pruned_loss=0.02344, audio_tagging_loss=0.007625, over 14012.00 frames. ], tot_loss[loss=0.07595, simple_loss=0.09857, pruned_loss=0.01718, audio_tagging_loss=0.009484, over 3052003.74 frames. ], batch size: 53, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:49:54,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1435400.0, ans=0.125
2023-11-21 08:49:56,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1435400.0, ans=0.125
2023-11-21 08:49:56,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1435400.0, ans=0.0
2023-11-21 08:50:12,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1435466.6666666667, ans=0.0
2023-11-21 08:50:13,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.76 vs. limit=12.0
2023-11-21 08:50:19,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.01 vs. limit=15.0
2023-11-21 08:50:20,531 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.007e+01 7.936e+01 8.755e+01 9.807e+01 1.231e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 08:50:27,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1435533.3333333333, ans=0.125
2023-11-21 08:50:29,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1435600.0, ans=0.5
2023-11-21 08:50:32,574 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.93 vs. limit=22.5
2023-11-21 08:50:40,642 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215350
2023-11-21 08:50:42,942 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 10950, loss[loss=0.08198, simple_loss=0.1147, pruned_loss=0.01836, audio_tagging_loss=0.006263, over 16068.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09814, pruned_loss=0.01708, audio_tagging_loss=0.00963, over 3059566.40 frames. ], batch size: 59, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:51:44,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215400
2023-11-21 08:51:47,125 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11000, loss[loss=0.06727, simple_loss=0.08148, pruned_loss=0.01325, audio_tagging_loss=0.01328, over 15212.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09872, pruned_loss=0.01712, audio_tagging_loss=0.009655, over 3068941.31 frames. ], batch size: 60, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:51:48,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.57 vs. limit=10.0
2023-11-21 08:51:56,401 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:51:57,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1436000.0, ans=10.0
2023-11-21 08:52:01,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1436066.6666666667, ans=0.125
2023-11-21 08:52:17,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.44 vs. limit=15.0
2023-11-21 08:52:30,019 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.054e+01 8.831e+01 9.756e+01 2.281e+02, threshold=1.766e+02, percent-clipped=1.0
2023-11-21 08:52:30,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1436200.0, ans=0.1
2023-11-21 08:52:45,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1436266.6666666667, ans=0.0
2023-11-21 08:52:50,282 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215450
2023-11-21 08:52:51,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1436333.3333333333, ans=0.0
2023-11-21 08:52:52,534 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11050, loss[loss=0.09106, simple_loss=0.1167, pruned_loss=0.02606, audio_tagging_loss=0.006631, over 14887.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09848, pruned_loss=0.01715, audio_tagging_loss=0.009661, over 3064681.08 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:53:11,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1436400.0, ans=0.125
2023-11-21 08:53:21,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1436466.6666666667, ans=0.125
2023-11-21 08:53:30,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1436533.3333333333, ans=0.0
2023-11-21 08:53:31,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1436533.3333333333, ans=0.0
2023-11-21 08:53:51,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1436600.0, ans=0.125
2023-11-21 08:53:54,360 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215500
2023-11-21 08:53:56,860 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11100, loss[loss=0.08907, simple_loss=0.1081, pruned_loss=0.02425, audio_tagging_loss=0.01076, over 16753.00 frames. ], tot_loss[loss=0.07607, simple_loss=0.09838, pruned_loss=0.01715, audio_tagging_loss=0.009722, over 3062938.00 frames. ], batch size: 62, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:53:59,845 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.98 vs. limit=15.0
2023-11-21 08:54:02,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1436666.6666666667, ans=0.2
2023-11-21 08:54:15,439 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:54:17,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1436733.3333333333, ans=0.125
2023-11-21 08:54:36,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1436866.6666666667, ans=0.125
2023-11-21 08:54:39,003 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.330e+01 8.983e+01 9.923e+01 1.426e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-21 08:54:47,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1436933.3333333333, ans=0.125
2023-11-21 08:54:52,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1436933.3333333333, ans=0.0
2023-11-21 08:54:58,165 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215550
2023-11-21 08:55:00,449 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11150, loss[loss=0.06584, simple_loss=0.0841, pruned_loss=0.01384, audio_tagging_loss=0.009957, over 15167.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09772, pruned_loss=0.01709, audio_tagging_loss=0.009831, over 3057378.97 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:55:31,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1437133.3333333333, ans=0.0
2023-11-21 08:55:38,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1437200.0, ans=0.0
2023-11-21 08:55:39,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.10 vs. limit=22.5
2023-11-21 08:55:45,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1437200.0, ans=0.1
2023-11-21 08:56:02,976 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215600
2023-11-21 08:56:06,627 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11200, loss[loss=0.104, simple_loss=0.1451, pruned_loss=0.02409, audio_tagging_loss=0.007308, over 15841.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09707, pruned_loss=0.01696, audio_tagging_loss=0.009984, over 3051774.62 frames. ], batch size: 54, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:56:23,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1437400.0, ans=0.0
2023-11-21 08:56:33,728 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.94 vs. limit=22.5
2023-11-21 08:56:48,511 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.237e+01 8.761e+01 9.554e+01 1.513e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 08:57:07,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1437600.0, ans=0.2
2023-11-21 08:57:08,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215650
2023-11-21 08:57:09,927 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:57:10,697 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11250, loss[loss=0.06346, simple_loss=0.08305, pruned_loss=0.01353, audio_tagging_loss=0.008407, over 15827.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09636, pruned_loss=0.01684, audio_tagging_loss=0.009886, over 3055584.36 frames. ], batch size: 58, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:57:14,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.02 vs. limit=6.0
2023-11-21 08:57:26,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1437733.3333333333, ans=0.125
2023-11-21 08:57:31,797 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.28 vs. limit=15.0
2023-11-21 08:57:40,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1437800.0, ans=0.125
2023-11-21 08:57:53,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1437866.6666666667, ans=0.1
2023-11-21 08:57:55,704 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.04 vs. limit=15.0
2023-11-21 08:58:05,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1437933.3333333333, ans=0.1
2023-11-21 08:58:05,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1437933.3333333333, ans=0.0
2023-11-21 08:58:12,897 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215700
2023-11-21 08:58:15,335 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11300, loss[loss=0.07731, simple_loss=0.09671, pruned_loss=0.01556, audio_tagging_loss=0.01339, over 15058.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.09603, pruned_loss=0.01668, audio_tagging_loss=0.009812, over 3064058.00 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:58:41,290 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:58:48,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1438133.3333333333, ans=0.1
2023-11-21 08:58:50,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1438133.3333333333, ans=0.125
2023-11-21 08:58:57,862 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 8.122e+01 8.636e+01 9.244e+01 1.479e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-21 08:59:05,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1438266.6666666667, ans=0.2
2023-11-21 08:59:17,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215750
2023-11-21 08:59:20,448 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11350, loss[loss=0.0724, simple_loss=0.09678, pruned_loss=0.01561, audio_tagging_loss=0.008394, over 15131.00 frames. ], tot_loss[loss=0.07513, simple_loss=0.0969, pruned_loss=0.01702, audio_tagging_loss=0.009654, over 3054821.38 frames. ], batch size: 59, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:59:20,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1438333.3333333333, ans=0.0
2023-11-21 08:59:42,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1438400.0, ans=0.125
2023-11-21 08:59:42,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1438400.0, ans=0.1
2023-11-21 08:59:57,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1438533.3333333333, ans=0.09899494936611666
2023-11-21 08:59:57,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1438533.3333333333, ans=0.125
2023-11-21 09:00:14,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1438600.0, ans=0.2
2023-11-21 09:00:15,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.75 vs. limit=10.0
2023-11-21 09:00:22,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215800
2023-11-21 09:00:24,846 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11400, loss[loss=0.07681, simple_loss=0.09853, pruned_loss=0.01725, audio_tagging_loss=0.0103, over 16214.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09665, pruned_loss=0.017, audio_tagging_loss=0.009568, over 3050695.89 frames. ], batch size: 60, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 09:00:27,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1438666.6666666667, ans=0.0
2023-11-21 09:00:29,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.56 vs. limit=22.5
2023-11-21 09:00:36,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1438733.3333333333, ans=0.0
2023-11-21 09:00:37,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1438733.3333333333, ans=0.0
2023-11-21 09:00:43,394 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.53 vs. limit=15.0
2023-11-21 09:01:07,981 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.283e+01 9.027e+01 9.488e+01 1.297e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-21 09:01:09,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1438866.6666666667, ans=0.0
2023-11-21 09:01:26,721 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215850
2023-11-21 09:01:29,047 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11450, loss[loss=0.07162, simple_loss=0.09289, pruned_loss=0.01517, audio_tagging_loss=0.01001, over 15061.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09632, pruned_loss=0.01678, audio_tagging_loss=0.009546, over 3053463.17 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 09:01:34,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.20 vs. limit=6.0
2023-11-21 09:02:32,337 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215900
2023-11-21 09:02:34,729 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11500, loss[loss=0.06723, simple_loss=0.08756, pruned_loss=0.01479, audio_tagging_loss=0.00866, over 15045.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09587, pruned_loss=0.01674, audio_tagging_loss=0.009568, over 3055727.43 frames. ], batch size: 57, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 09:03:09,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1439466.6666666667, ans=0.125
2023-11-21 09:03:10,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=12.0
2023-11-21 09:03:17,260 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.959e+01 7.996e+01 8.548e+01 9.424e+01 1.295e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-21 09:03:27,984 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.40 vs. limit=15.0
2023-11-21 09:03:31,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1439600.0, ans=0.0
2023-11-21 09:03:33,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1439600.0, ans=0.5
2023-11-21 09:03:37,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 215950
2023-11-21 09:03:39,559 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.52 vs. limit=22.5
2023-11-21 09:03:40,092 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11550, loss[loss=0.07313, simple_loss=0.09739, pruned_loss=0.01501, audio_tagging_loss=0.009422, over 15532.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09576, pruned_loss=0.01677, audio_tagging_loss=0.009652, over 3057461.12 frames. ], batch size: 57, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:03:56,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1439733.3333333333, ans=0.0
2023-11-21 09:04:01,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1439733.3333333333, ans=0.125
2023-11-21 09:04:15,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1439800.0, ans=0.0
2023-11-21 09:04:18,476 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:04:30,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1439866.6666666667, ans=0.125
2023-11-21 09:04:36,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1439933.3333333333, ans=0.125
2023-11-21 09:04:39,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1439933.3333333333, ans=0.1
2023-11-21 09:04:42,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216000
2023-11-21 09:04:47,540 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11600, loss[loss=0.06781, simple_loss=0.08629, pruned_loss=0.01364, audio_tagging_loss=0.01102, over 14856.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.0968, pruned_loss=0.01696, audio_tagging_loss=0.009632, over 3058204.00 frames. ], batch size: 57, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:04:51,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1440000.0, ans=0.125
2023-11-21 09:04:53,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1440000.0, ans=0.125
2023-11-21 09:04:55,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1440000.0, ans=10.0
2023-11-21 09:05:29,710 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.268e+01 8.853e+01 9.480e+01 1.180e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 09:05:48,800 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216050
2023-11-21 09:05:51,736 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11650, loss[loss=0.06036, simple_loss=0.0756, pruned_loss=0.01311, audio_tagging_loss=0.009454, over 16698.00 frames. ], tot_loss[loss=0.07513, simple_loss=0.09726, pruned_loss=0.01695, audio_tagging_loss=0.009542, over 3056382.18 frames. ], batch size: 64, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:06:03,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1440400.0, ans=0.0
2023-11-21 09:06:24,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1440466.6666666667, ans=0.125
2023-11-21 09:06:40,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1440533.3333333333, ans=0.05
2023-11-21 09:06:46,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1440600.0, ans=0.125
2023-11-21 09:06:54,295 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216100
2023-11-21 09:06:56,685 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11700, loss[loss=0.07207, simple_loss=0.0947, pruned_loss=0.01722, audio_tagging_loss=0.007509, over 15084.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09757, pruned_loss=0.01716, audio_tagging_loss=0.009537, over 3048362.74 frames. ], batch size: 55, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:07:02,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1440666.6666666667, ans=0.2
2023-11-21 09:07:10,986 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.40 vs. limit=15.0
2023-11-21 09:07:22,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.22 vs. limit=15.0
2023-11-21 09:07:38,772 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.159e+01 8.719e+01 9.346e+01 1.199e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 09:07:57,243 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216150
2023-11-21 09:07:57,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1440933.3333333333, ans=0.125
2023-11-21 09:07:59,717 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11750, loss[loss=0.06702, simple_loss=0.08606, pruned_loss=0.01519, audio_tagging_loss=0.008797, over 14272.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09807, pruned_loss=0.01736, audio_tagging_loss=0.009634, over 3045582.26 frames. ], batch size: 56, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:08:15,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.07 vs. limit=15.0
2023-11-21 09:08:41,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1441200.0, ans=0.2
2023-11-21 09:08:45,578 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.19 vs. limit=15.0
2023-11-21 09:09:01,196 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216200
2023-11-21 09:09:02,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1441333.3333333333, ans=0.125
2023-11-21 09:09:04,071 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11800, loss[loss=0.08502, simple_loss=0.1187, pruned_loss=0.01834, audio_tagging_loss=0.007323, over 15142.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09771, pruned_loss=0.01729, audio_tagging_loss=0.009789, over 3045209.39 frames. ], batch size: 55, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:09:23,874 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.07 vs. limit=10.0
2023-11-21 09:09:24,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1441400.0, ans=0.125
2023-11-21 09:09:28,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1441400.0, ans=0.125
2023-11-21 09:09:28,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff3.min_abs, batch_count=1441400.0, ans=0.2
2023-11-21 09:09:33,865 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.38 vs. limit=6.0
2023-11-21 09:09:47,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1441533.3333333333, ans=0.1
2023-11-21 09:09:48,128 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 8.145e+01 8.665e+01 9.541e+01 1.453e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 09:09:52,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1441533.3333333333, ans=0.0
2023-11-21 09:09:57,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1441600.0, ans=0.125
2023-11-21 09:10:07,921 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216250
2023-11-21 09:10:10,317 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11850, loss[loss=0.07288, simple_loss=0.09796, pruned_loss=0.01594, audio_tagging_loss=0.007957, over 15140.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09762, pruned_loss=0.01721, audio_tagging_loss=0.009841, over 3050472.33 frames. ], batch size: 54, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:10:10,795 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:10:34,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1441800.0, ans=0.125
2023-11-21 09:10:49,114 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.34 vs. limit=22.5
2023-11-21 09:10:53,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1441866.6666666667, ans=0.2
2023-11-21 09:11:11,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216300
2023-11-21 09:11:14,274 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11900, loss[loss=0.07306, simple_loss=0.09608, pruned_loss=0.01519, audio_tagging_loss=0.009824, over 16043.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09712, pruned_loss=0.01696, audio_tagging_loss=0.009917, over 3050887.78 frames. ], batch size: 60, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:11:34,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.56 vs. limit=15.0
2023-11-21 09:11:57,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1442200.0, ans=0.1
2023-11-21 09:11:58,209 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.062e+01 8.187e+01 8.956e+01 9.688e+01 1.192e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 09:12:04,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1442266.6666666667, ans=0.1
2023-11-21 09:12:15,539 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216350
2023-11-21 09:12:17,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1442333.3333333333, ans=0.125
2023-11-21 09:12:18,013 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 11950, loss[loss=0.09349, simple_loss=0.1209, pruned_loss=0.02504, audio_tagging_loss=0.008008, over 14480.00 frames. ], tot_loss[loss=0.07517, simple_loss=0.09645, pruned_loss=0.01701, audio_tagging_loss=0.009931, over 3047353.10 frames. ], batch size: 54, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:12:23,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1442333.3333333333, ans=0.125
2023-11-21 09:12:29,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1442333.3333333333, ans=0.0
2023-11-21 09:12:34,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.34 vs. limit=15.0
2023-11-21 09:12:38,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1442400.0, ans=0.2
2023-11-21 09:13:02,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1442533.3333333333, ans=0.07
2023-11-21 09:13:04,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1442533.3333333333, ans=0.0
2023-11-21 09:13:11,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1442600.0, ans=0.0
2023-11-21 09:13:17,289 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216400
2023-11-21 09:13:19,969 INFO [train_asr.py:1221] (2/4) Epoch 18, batch 12000, loss[loss=0.05716, simple_loss=0.06055, pruned_loss=0.01265, audio_tagging_loss=0.01424, over 14793.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.0962, pruned_loss=0.01697, audio_tagging_loss=0.01002, over 3042927.26 frames. ], batch size: 58, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:13:19,970 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 09:14:01,470 INFO [train_asr.py:1253] (2/4) Epoch 18, validation: loss=0.06063, simple_loss=0.05245, pruned_loss=0.005328, audio_tagging_loss=0.02908, over 4681554.00 frames. 
2023-11-21 09:14:01,471 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 09:14:01,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1442666.6666666667, ans=0.125
2023-11-21 09:14:08,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1442666.6666666667, ans=0.2
2023-11-21 09:14:12,531 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.47 vs. limit=22.5
2023-11-21 09:14:25,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1442800.0, ans=0.0
2023-11-21 09:15:03,521 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 0, loss[loss=0.09315, simple_loss=0.1142, pruned_loss=0.01894, audio_tagging_loss=0.01711, over 16619.00 frames. ], tot_loss[loss=0.09315, simple_loss=0.1142, pruned_loss=0.01894, audio_tagging_loss=0.01711, over 16619.00 frames. ], batch size: 62, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:15:03,521 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 09:15:39,007 INFO [train_asr.py:1253] (2/4) Epoch 19, validation: loss=0.05975, simple_loss=0.05244, pruned_loss=0.005316, audio_tagging_loss=0.02822, over 4681554.00 frames. 
2023-11-21 09:15:39,008 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 09:15:45,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1442820.0, ans=0.125
2023-11-21 09:15:52,044 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.47 vs. limit=22.5
2023-11-21 09:15:52,327 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.140e+01 8.995e+01 9.841e+01 1.386e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-21 09:15:57,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.59 vs. limit=12.0
2023-11-21 09:16:09,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1442953.3333333333, ans=0.125
2023-11-21 09:16:11,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216450
2023-11-21 09:16:24,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1443020.0, ans=0.0
2023-11-21 09:16:36,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1443086.6666666667, ans=0.125
2023-11-21 09:16:38,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1443086.6666666667, ans=0.1
2023-11-21 09:16:43,264 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 50, loss[loss=0.0928, simple_loss=0.1064, pruned_loss=0.02217, audio_tagging_loss=0.01741, over 14986.00 frames. ], tot_loss[loss=0.08161, simple_loss=0.0949, pruned_loss=0.01599, audio_tagging_loss=0.01817, over 690834.59 frames. ], batch size: 57, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:16:43,824 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=5.03 vs. limit=15.0
2023-11-21 09:16:51,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1443153.3333333333, ans=0.125
2023-11-21 09:17:05,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1443220.0, ans=10.0
2023-11-21 09:17:15,877 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216500
2023-11-21 09:17:36,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1443420.0, ans=0.025
2023-11-21 09:17:46,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1443420.0, ans=0.2
2023-11-21 09:17:49,157 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 100, loss[loss=0.05972, simple_loss=0.07183, pruned_loss=0.00986, audio_tagging_loss=0.01394, over 15013.00 frames. ], tot_loss[loss=0.08219, simple_loss=0.09625, pruned_loss=0.01656, audio_tagging_loss=0.01751, over 1214140.91 frames. ], batch size: 57, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:17:49,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1443486.6666666667, ans=0.125
2023-11-21 09:17:54,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1443486.6666666667, ans=0.0
2023-11-21 09:18:00,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1443553.3333333333, ans=0.1
2023-11-21 09:18:01,578 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:18:01,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1443553.3333333333, ans=0.125
2023-11-21 09:18:02,451 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.866e+01 9.575e+01 1.050e+02 1.381e+02, threshold=1.915e+02, percent-clipped=0.0
2023-11-21 09:18:04,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.51 vs. limit=15.0
2023-11-21 09:18:16,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1443620.0, ans=0.125
2023-11-21 09:18:19,836 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216550
2023-11-21 09:18:23,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1443620.0, ans=0.1
2023-11-21 09:18:41,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1443753.3333333333, ans=0.1
2023-11-21 09:18:52,662 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 150, loss[loss=0.06153, simple_loss=0.07709, pruned_loss=0.009497, audio_tagging_loss=0.01349, over 15378.00 frames. ], tot_loss[loss=0.07995, simple_loss=0.09541, pruned_loss=0.01632, audio_tagging_loss=0.01593, over 1615985.19 frames. ], batch size: 59, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:19:05,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1443886.6666666667, ans=0.125
2023-11-21 09:19:15,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1443886.6666666667, ans=0.125
2023-11-21 09:19:21,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1443953.3333333333, ans=0.05
2023-11-21 09:19:24,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1443953.3333333333, ans=0.125
2023-11-21 09:19:25,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216600
2023-11-21 09:19:25,406 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:19:33,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1444020.0, ans=0.125
2023-11-21 09:19:35,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1444020.0, ans=0.125
2023-11-21 09:19:38,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1444020.0, ans=0.0
2023-11-21 09:19:43,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1444086.6666666667, ans=0.125
2023-11-21 09:19:44,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1444086.6666666667, ans=0.125
2023-11-21 09:19:47,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1444086.6666666667, ans=0.0
2023-11-21 09:19:56,865 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 200, loss[loss=0.07589, simple_loss=0.1054, pruned_loss=0.01599, audio_tagging_loss=0.007211, over 15338.00 frames. ], tot_loss[loss=0.07908, simple_loss=0.09615, pruned_loss=0.01675, audio_tagging_loss=0.01425, over 1924913.10 frames. ], batch size: 57, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:20:12,110 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.223e+01 8.596e+01 9.461e+01 1.153e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 09:20:14,092 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.78 vs. limit=12.0
2023-11-21 09:20:14,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1444220.0, ans=0.125
2023-11-21 09:20:21,463 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:20:25,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1444286.6666666667, ans=0.0
2023-11-21 09:20:29,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.50 vs. limit=10.0
2023-11-21 09:20:29,869 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216650
2023-11-21 09:20:52,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1444420.0, ans=0.125
2023-11-21 09:21:02,346 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 250, loss[loss=0.05222, simple_loss=0.06124, pruned_loss=0.008395, audio_tagging_loss=0.0132, over 14522.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09565, pruned_loss=0.01687, audio_tagging_loss=0.01301, over 2172665.46 frames. ], batch size: 56, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:21:02,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1444486.6666666667, ans=0.1
2023-11-21 09:21:27,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1444620.0, ans=0.0
2023-11-21 09:21:33,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216700
2023-11-21 09:21:36,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1444620.0, ans=0.0
2023-11-21 09:21:39,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1444686.6666666667, ans=0.2
2023-11-21 09:21:41,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1444686.6666666667, ans=0.125
2023-11-21 09:21:49,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1444686.6666666667, ans=0.125
2023-11-21 09:22:00,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1444753.3333333333, ans=0.1
2023-11-21 09:22:06,600 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 300, loss[loss=0.08308, simple_loss=0.1116, pruned_loss=0.01981, audio_tagging_loss=0.007484, over 14817.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09645, pruned_loss=0.01709, audio_tagging_loss=0.01199, over 2361386.65 frames. ], batch size: 55, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:22:10,702 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:22:19,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1444886.6666666667, ans=0.0
2023-11-21 09:22:19,963 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.255e+01 8.343e+01 8.887e+01 9.711e+01 1.312e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 09:22:29,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1444886.6666666667, ans=0.0
2023-11-21 09:22:38,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216750
2023-11-21 09:22:42,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1444953.3333333333, ans=0.125
2023-11-21 09:22:43,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1445020.0, ans=0.125
2023-11-21 09:22:51,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1445020.0, ans=0.0
2023-11-21 09:23:06,797 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.09 vs. limit=12.0
2023-11-21 09:23:09,897 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 350, loss[loss=0.06013, simple_loss=0.07452, pruned_loss=0.01197, audio_tagging_loss=0.0109, over 16476.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09589, pruned_loss=0.01687, audio_tagging_loss=0.01135, over 2509100.74 frames. ], batch size: 61, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:23:36,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1445286.6666666667, ans=0.025
2023-11-21 09:23:42,984 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216800
2023-11-21 09:23:47,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1445286.6666666667, ans=0.0
2023-11-21 09:23:48,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1445353.3333333333, ans=0.125
2023-11-21 09:23:55,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1445353.3333333333, ans=0.1
2023-11-21 09:24:00,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1445420.0, ans=0.0
2023-11-21 09:24:01,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1445420.0, ans=0.0
2023-11-21 09:24:07,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1445420.0, ans=0.07
2023-11-21 09:24:15,662 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 400, loss[loss=0.08901, simple_loss=0.126, pruned_loss=0.02107, audio_tagging_loss=0.004942, over 14843.00 frames. ], tot_loss[loss=0.07635, simple_loss=0.09678, pruned_loss=0.0172, audio_tagging_loss=0.01076, over 2632043.22 frames. ], batch size: 55, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:24:29,297 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.137e+01 8.063e+01 8.732e+01 9.531e+01 1.428e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 09:24:35,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=22.5
2023-11-21 09:24:38,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1445553.3333333333, ans=0.0
2023-11-21 09:24:40,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1445620.0, ans=0.0
2023-11-21 09:24:47,461 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216850
2023-11-21 09:24:47,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1445620.0, ans=0.0
2023-11-21 09:25:06,906 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.74 vs. limit=15.0
2023-11-21 09:25:17,950 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.40 vs. limit=15.0
2023-11-21 09:25:19,742 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 450, loss[loss=0.07915, simple_loss=0.1056, pruned_loss=0.01702, audio_tagging_loss=0.009335, over 15547.00 frames. ], tot_loss[loss=0.0755, simple_loss=0.09595, pruned_loss=0.01711, audio_tagging_loss=0.01042, over 2729399.73 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:25:25,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1445820.0, ans=0.04949747468305833
2023-11-21 09:25:28,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1445820.0, ans=0.125
2023-11-21 09:25:28,669 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.44 vs. limit=22.5
2023-11-21 09:25:36,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1445886.6666666667, ans=0.2
2023-11-21 09:25:37,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1445886.6666666667, ans=0.0
2023-11-21 09:25:43,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1445886.6666666667, ans=0.1
2023-11-21 09:25:47,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1445953.3333333333, ans=0.0
2023-11-21 09:25:50,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1445953.3333333333, ans=0.125
2023-11-21 09:25:53,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216900
2023-11-21 09:26:18,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1446086.6666666667, ans=0.5
2023-11-21 09:26:24,332 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 500, loss[loss=0.07637, simple_loss=0.09743, pruned_loss=0.01839, audio_tagging_loss=0.009266, over 14532.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09564, pruned_loss=0.01701, audio_tagging_loss=0.01019, over 2800140.28 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:26:41,040 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.147e+01 8.789e+01 9.630e+01 1.210e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 09:26:48,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1446220.0, ans=0.1
2023-11-21 09:26:54,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1446286.6666666667, ans=0.125
2023-11-21 09:26:56,998 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 216950
2023-11-21 09:26:57,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=1446286.6666666667, ans=15.0
2023-11-21 09:27:06,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0
2023-11-21 09:27:16,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1446420.0, ans=0.125
2023-11-21 09:27:18,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1446420.0, ans=0.125
2023-11-21 09:27:29,711 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 550, loss[loss=0.07454, simple_loss=0.09471, pruned_loss=0.01687, audio_tagging_loss=0.01031, over 15464.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09615, pruned_loss=0.01697, audio_tagging_loss=0.01001, over 2863106.49 frames. ], batch size: 59, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:27:54,837 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.68 vs. limit=22.5
2023-11-21 09:28:00,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217000
2023-11-21 09:28:00,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1446620.0, ans=0.0
2023-11-21 09:28:02,644 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.84 vs. limit=15.0
2023-11-21 09:28:02,735 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=15.0
2023-11-21 09:28:06,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1446686.6666666667, ans=0.1
2023-11-21 09:28:15,271 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.94 vs. limit=10.0
2023-11-21 09:28:19,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1446686.6666666667, ans=0.125
2023-11-21 09:28:22,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1446753.3333333333, ans=0.125
2023-11-21 09:28:26,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1446753.3333333333, ans=0.2
2023-11-21 09:28:33,553 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 600, loss[loss=0.06524, simple_loss=0.08622, pruned_loss=0.01424, audio_tagging_loss=0.007889, over 15755.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09669, pruned_loss=0.01686, audio_tagging_loss=0.009914, over 2907404.12 frames. ], batch size: 60, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:28:38,777 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:28:49,014 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.367e+01 7.902e+01 8.372e+01 9.097e+01 1.242e+02, threshold=1.674e+02, percent-clipped=0.0
2023-11-21 09:28:55,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1446886.6666666667, ans=0.0
2023-11-21 09:28:57,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1446886.6666666667, ans=0.07
2023-11-21 09:29:07,214 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217050
2023-11-21 09:29:16,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1447020.0, ans=0.125
2023-11-21 09:29:25,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1447086.6666666667, ans=0.125
2023-11-21 09:29:26,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=12.0
2023-11-21 09:29:33,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1447086.6666666667, ans=0.2
2023-11-21 09:29:38,229 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 650, loss[loss=0.07546, simple_loss=0.1062, pruned_loss=0.01426, audio_tagging_loss=0.008106, over 15776.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09698, pruned_loss=0.01698, audio_tagging_loss=0.00985, over 2932830.66 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:29:54,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1447220.0, ans=0.125
2023-11-21 09:29:54,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1447220.0, ans=0.125
2023-11-21 09:30:00,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.68 vs. limit=15.0
2023-11-21 09:30:10,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217100
2023-11-21 09:30:31,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1447420.0, ans=0.125
2023-11-21 09:30:40,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1447420.0, ans=0.0
2023-11-21 09:30:42,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1447486.6666666667, ans=0.0
2023-11-21 09:30:44,269 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 700, loss[loss=0.07013, simple_loss=0.09659, pruned_loss=0.01385, audio_tagging_loss=0.007992, over 15566.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.0969, pruned_loss=0.01691, audio_tagging_loss=0.009856, over 2957736.72 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:30:49,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1447486.6666666667, ans=0.2
2023-11-21 09:30:50,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1447486.6666666667, ans=0.0
2023-11-21 09:30:54,766 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.74 vs. limit=22.5
2023-11-21 09:30:58,916 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.199e+01 8.920e+01 9.625e+01 1.204e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 09:31:02,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1447553.3333333333, ans=0.0
2023-11-21 09:31:08,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1447620.0, ans=0.1
2023-11-21 09:31:14,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217150
2023-11-21 09:31:20,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1447686.6666666667, ans=0.04949747468305833
2023-11-21 09:31:24,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1447686.6666666667, ans=0.125
2023-11-21 09:31:30,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1447686.6666666667, ans=0.2
2023-11-21 09:31:47,780 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 750, loss[loss=0.08143, simple_loss=0.09651, pruned_loss=0.02138, audio_tagging_loss=0.01179, over 15882.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09707, pruned_loss=0.01702, audio_tagging_loss=0.009922, over 2978578.04 frames. ], batch size: 61, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:32:20,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217200
2023-11-21 09:32:49,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1448086.6666666667, ans=0.125
2023-11-21 09:32:51,796 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 800, loss[loss=0.08969, simple_loss=0.1201, pruned_loss=0.02121, audio_tagging_loss=0.008442, over 14769.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09697, pruned_loss=0.01703, audio_tagging_loss=0.009925, over 2987275.14 frames. ], batch size: 54, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:33:07,433 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.21 vs. limit=15.0
2023-11-21 09:33:08,188 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.219e+01 8.917e+01 9.925e+01 1.296e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 09:33:09,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1448220.0, ans=0.125
2023-11-21 09:33:14,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1448220.0, ans=0.025
2023-11-21 09:33:24,907 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217250
2023-11-21 09:33:46,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1448420.0, ans=0.125
2023-11-21 09:33:57,393 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 850, loss[loss=0.1089, simple_loss=0.1473, pruned_loss=0.02566, audio_tagging_loss=0.00961, over 15655.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09699, pruned_loss=0.01705, audio_tagging_loss=0.009981, over 3008369.20 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:34:03,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1448486.6666666667, ans=0.1
2023-11-21 09:34:06,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1448486.6666666667, ans=0.2
2023-11-21 09:34:12,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1448553.3333333333, ans=0.125
2023-11-21 09:34:19,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1448553.3333333333, ans=0.0
2023-11-21 09:34:28,615 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217300
2023-11-21 09:34:36,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1448686.6666666667, ans=0.0
2023-11-21 09:34:40,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1448686.6666666667, ans=0.0
2023-11-21 09:34:41,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1448686.6666666667, ans=0.125
2023-11-21 09:34:48,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1448753.3333333333, ans=0.125
2023-11-21 09:34:55,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1448753.3333333333, ans=0.0
2023-11-21 09:35:00,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1448820.0, ans=0.09899494936611666
2023-11-21 09:35:00,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1448820.0, ans=0.125
2023-11-21 09:35:01,259 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 900, loss[loss=0.06623, simple_loss=0.08274, pruned_loss=0.01391, audio_tagging_loss=0.01095, over 14562.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09649, pruned_loss=0.01689, audio_tagging_loss=0.009958, over 3016648.72 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:35:15,734 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.060e+01 7.979e+01 8.801e+01 9.598e+01 1.201e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 09:35:33,202 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217350
2023-11-21 09:35:39,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1449020.0, ans=0.125
2023-11-21 09:35:48,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1449020.0, ans=0.0
2023-11-21 09:35:51,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.53 vs. limit=15.0
2023-11-21 09:36:02,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.95 vs. limit=15.0
2023-11-21 09:36:04,293 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 950, loss[loss=0.07763, simple_loss=0.1019, pruned_loss=0.01785, audio_tagging_loss=0.008833, over 15324.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09591, pruned_loss=0.01681, audio_tagging_loss=0.009871, over 3020748.51 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:36:13,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1449153.3333333333, ans=0.125
2023-11-21 09:36:17,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1449220.0, ans=0.07
2023-11-21 09:36:25,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1449220.0, ans=0.2
2023-11-21 09:36:28,756 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.82 vs. limit=6.0
2023-11-21 09:36:36,780 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217400
2023-11-21 09:36:44,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1449353.3333333333, ans=0.125
2023-11-21 09:37:05,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1449420.0, ans=0.125
2023-11-21 09:37:08,033 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1000, loss[loss=0.06361, simple_loss=0.07173, pruned_loss=0.01679, audio_tagging_loss=0.01095, over 16561.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.09572, pruned_loss=0.01691, audio_tagging_loss=0.009764, over 3037631.80 frames. ], batch size: 63, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:37:25,113 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.132e+01 7.890e+01 8.682e+01 9.562e+01 1.074e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 09:37:31,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1449553.3333333333, ans=0.125
2023-11-21 09:37:35,082 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:37:39,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217450
2023-11-21 09:37:39,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1449620.0, ans=0.1
2023-11-21 09:37:40,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1449620.0, ans=0.125
2023-11-21 09:38:03,129 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.18 vs. limit=22.5
2023-11-21 09:38:08,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1449753.3333333333, ans=0.07
2023-11-21 09:38:12,111 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1050, loss[loss=0.08786, simple_loss=0.118, pruned_loss=0.02107, audio_tagging_loss=0.007798, over 16241.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09506, pruned_loss=0.01672, audio_tagging_loss=0.009604, over 3035259.72 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:38:16,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1449820.0, ans=0.0
2023-11-21 09:38:16,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.14 vs. limit=12.0
2023-11-21 09:38:23,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1449886.6666666667, ans=0.0
2023-11-21 09:38:27,286 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=15.0
2023-11-21 09:38:36,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1449953.3333333333, ans=0.125
2023-11-21 09:38:43,239 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217500
2023-11-21 09:38:48,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1450020.0, ans=0.0
2023-11-21 09:39:14,694 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1100, loss[loss=0.07959, simple_loss=0.1022, pruned_loss=0.01859, audio_tagging_loss=0.009891, over 15565.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09511, pruned_loss=0.01675, audio_tagging_loss=0.00956, over 3036980.07 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:39:16,349 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.66 vs. limit=15.0
2023-11-21 09:39:18,234 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:39:31,325 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.109e+01 8.788e+01 9.339e+01 1.148e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 09:39:43,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.60 vs. limit=22.5
2023-11-21 09:39:47,989 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217550
2023-11-21 09:40:06,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1450420.0, ans=0.0
2023-11-21 09:40:07,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1450420.0, ans=0.125
2023-11-21 09:40:17,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1450486.6666666667, ans=15.0
2023-11-21 09:40:18,395 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1150, loss[loss=0.07811, simple_loss=0.1028, pruned_loss=0.0178, audio_tagging_loss=0.008902, over 14648.00 frames. ], tot_loss[loss=0.07411, simple_loss=0.09537, pruned_loss=0.0168, audio_tagging_loss=0.009623, over 3037821.69 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:40:27,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1450486.6666666667, ans=0.2
2023-11-21 09:40:31,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1450553.3333333333, ans=0.125
2023-11-21 09:40:44,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1450620.0, ans=0.125
2023-11-21 09:40:47,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1450620.0, ans=0.0
2023-11-21 09:40:51,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217600
2023-11-21 09:41:00,747 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.61 vs. limit=22.5
2023-11-21 09:41:14,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1450753.3333333333, ans=0.125
2023-11-21 09:41:24,283 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1200, loss[loss=0.0829, simple_loss=0.1134, pruned_loss=0.01749, audio_tagging_loss=0.008697, over 14944.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09519, pruned_loss=0.01658, audio_tagging_loss=0.009584, over 3038936.33 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:41:27,449 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.77 vs. limit=6.0
2023-11-21 09:41:33,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1450820.0, ans=0.0
2023-11-21 09:41:40,312 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.126e+01 8.918e+01 9.759e+01 1.159e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 09:41:41,151 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=24.74 vs. limit=22.5
2023-11-21 09:41:55,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217650
2023-11-21 09:42:17,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1451086.6666666667, ans=0.125
2023-11-21 09:42:22,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1451086.6666666667, ans=0.0
2023-11-21 09:42:28,262 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1250, loss[loss=0.06797, simple_loss=0.09245, pruned_loss=0.01235, audio_tagging_loss=0.009385, over 14838.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09519, pruned_loss=0.01656, audio_tagging_loss=0.009512, over 3039793.89 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:42:29,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1451153.3333333333, ans=0.125
2023-11-21 09:42:30,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1451153.3333333333, ans=0.125
2023-11-21 09:42:34,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1451153.3333333333, ans=0.0
2023-11-21 09:42:35,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1451153.3333333333, ans=0.125
2023-11-21 09:43:00,187 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217700
2023-11-21 09:43:05,007 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.33 vs. limit=22.5
2023-11-21 09:43:19,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1451420.0, ans=0.125
2023-11-21 09:43:21,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1451420.0, ans=0.125
2023-11-21 09:43:24,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1451420.0, ans=0.2
2023-11-21 09:43:31,383 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1300, loss[loss=0.06682, simple_loss=0.08664, pruned_loss=0.01232, audio_tagging_loss=0.01118, over 14048.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09548, pruned_loss=0.0166, audio_tagging_loss=0.009576, over 3036369.65 frames. ], batch size: 53, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:43:48,499 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.540e+01 8.140e+01 8.673e+01 9.454e+01 1.388e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 09:43:58,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1451620.0, ans=0.1
2023-11-21 09:44:03,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217750
2023-11-21 09:44:04,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1451620.0, ans=0.125
2023-11-21 09:44:06,446 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:44:15,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1451686.6666666667, ans=0.0
2023-11-21 09:44:15,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1451686.6666666667, ans=0.125
2023-11-21 09:44:35,293 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1350, loss[loss=0.06291, simple_loss=0.08487, pruned_loss=0.01138, audio_tagging_loss=0.009094, over 15613.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09641, pruned_loss=0.01686, audio_tagging_loss=0.009652, over 3043981.69 frames. ], batch size: 61, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:44:38,451 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.67 vs. limit=15.0
2023-11-21 09:44:48,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1451886.6666666667, ans=0.125
2023-11-21 09:44:58,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1451886.6666666667, ans=0.0
2023-11-21 09:45:06,577 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217800
2023-11-21 09:45:16,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1452020.0, ans=0.125
2023-11-21 09:45:21,507 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:45:27,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.94 vs. limit=15.0
2023-11-21 09:45:38,995 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1400, loss[loss=0.08002, simple_loss=0.1024, pruned_loss=0.01995, audio_tagging_loss=0.008876, over 14934.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09626, pruned_loss=0.01677, audio_tagging_loss=0.009658, over 3044443.25 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:45:46,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1452153.3333333333, ans=0.95
2023-11-21 09:45:55,279 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:45:55,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1452220.0, ans=0.125
2023-11-21 09:45:56,184 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.285e+01 9.066e+01 9.966e+01 1.256e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-21 09:46:10,899 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217850
2023-11-21 09:46:13,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1452286.6666666667, ans=0.125
2023-11-21 09:46:18,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1452353.3333333333, ans=0.025
2023-11-21 09:46:26,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1452353.3333333333, ans=0.125
2023-11-21 09:46:32,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1452420.0, ans=0.1
2023-11-21 09:46:40,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1452486.6666666667, ans=0.125
2023-11-21 09:46:41,621 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1450, loss[loss=0.07881, simple_loss=0.09892, pruned_loss=0.02005, audio_tagging_loss=0.009299, over 14513.00 frames. ], tot_loss[loss=0.07461, simple_loss=0.09631, pruned_loss=0.01681, audio_tagging_loss=0.00965, over 3041780.23 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:46:53,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1452553.3333333333, ans=0.0
2023-11-21 09:47:00,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1452553.3333333333, ans=0.125
2023-11-21 09:47:03,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1452553.3333333333, ans=0.1
2023-11-21 09:47:14,060 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217900
2023-11-21 09:47:21,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1452686.6666666667, ans=0.125
2023-11-21 09:47:37,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.13 vs. limit=15.0
2023-11-21 09:47:39,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1452753.3333333333, ans=0.0
2023-11-21 09:47:46,027 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1500, loss[loss=0.06663, simple_loss=0.07682, pruned_loss=0.01828, audio_tagging_loss=0.009931, over 15488.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09651, pruned_loss=0.01695, audio_tagging_loss=0.009664, over 3034153.75 frames. ], batch size: 62, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:47:52,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1452820.0, ans=0.0
2023-11-21 09:47:54,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.31 vs. limit=12.0
2023-11-21 09:48:01,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1452886.6666666667, ans=0.1
2023-11-21 09:48:03,704 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.762e+01 8.061e+01 8.622e+01 9.502e+01 1.412e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-21 09:48:11,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.92 vs. limit=10.0
2023-11-21 09:48:13,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1452953.3333333333, ans=0.0
2023-11-21 09:48:17,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 217950
2023-11-21 09:48:23,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1453020.0, ans=0.1
2023-11-21 09:48:37,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.10 vs. limit=15.0
2023-11-21 09:48:49,050 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1550, loss[loss=0.08382, simple_loss=0.1132, pruned_loss=0.02006, audio_tagging_loss=0.00714, over 15533.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.0971, pruned_loss=0.01714, audio_tagging_loss=0.009703, over 3036894.81 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:49:16,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1453286.6666666667, ans=0.0
2023-11-21 09:49:21,865 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218000
2023-11-21 09:49:24,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1453286.6666666667, ans=0.1
2023-11-21 09:49:28,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1453353.3333333333, ans=0.2
2023-11-21 09:49:53,165 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1600, loss[loss=0.0602, simple_loss=0.0772, pruned_loss=0.01226, audio_tagging_loss=0.009333, over 14669.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09651, pruned_loss=0.01701, audio_tagging_loss=0.009736, over 3039833.47 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:50:02,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1453486.6666666667, ans=0.125
2023-11-21 09:50:11,476 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.287e+01 9.004e+01 9.532e+01 2.614e+02, threshold=1.801e+02, percent-clipped=1.0
2023-11-21 09:50:24,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1453620.0, ans=0.125
2023-11-21 09:50:25,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218050
2023-11-21 09:50:43,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1453753.3333333333, ans=10.0
2023-11-21 09:50:44,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1453753.3333333333, ans=0.1
2023-11-21 09:50:44,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1453753.3333333333, ans=0.125
2023-11-21 09:50:57,623 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1650, loss[loss=0.06191, simple_loss=0.07797, pruned_loss=0.01475, audio_tagging_loss=0.008172, over 14735.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09608, pruned_loss=0.01679, audio_tagging_loss=0.009829, over 3045987.18 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:51:08,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1453886.6666666667, ans=0.0
2023-11-21 09:51:10,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1453886.6666666667, ans=0.0
2023-11-21 09:51:25,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.23 vs. limit=15.0
2023-11-21 09:51:29,047 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218100
2023-11-21 09:51:33,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1453953.3333333333, ans=0.125
2023-11-21 09:51:42,697 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.82 vs. limit=12.0
2023-11-21 09:52:01,505 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1700, loss[loss=0.05223, simple_loss=0.06297, pruned_loss=0.0111, audio_tagging_loss=0.009642, over 16121.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09625, pruned_loss=0.01682, audio_tagging_loss=0.009805, over 3041573.72 frames. ], batch size: 62, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:52:06,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1454153.3333333333, ans=0.0
2023-11-21 09:52:17,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1454220.0, ans=0.0
2023-11-21 09:52:18,990 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.81 vs. limit=15.0
2023-11-21 09:52:19,398 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.104e+01 8.594e+01 9.360e+01 1.130e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 09:52:23,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1454220.0, ans=0.125
2023-11-21 09:52:26,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.89 vs. limit=15.0
2023-11-21 09:52:33,603 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218150
2023-11-21 09:53:00,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1454420.0, ans=0.125
2023-11-21 09:53:05,414 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1750, loss[loss=0.07088, simple_loss=0.09058, pruned_loss=0.01692, audio_tagging_loss=0.008675, over 14659.00 frames. ], tot_loss[loss=0.07429, simple_loss=0.09564, pruned_loss=0.01668, audio_tagging_loss=0.009795, over 3041946.17 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:53:14,406 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.87 vs. limit=15.0
2023-11-21 09:53:21,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1454553.3333333333, ans=0.125
2023-11-21 09:53:25,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1454553.3333333333, ans=0.125
2023-11-21 09:53:37,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218200
2023-11-21 09:53:42,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1454686.6666666667, ans=0.125
2023-11-21 09:54:09,255 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1800, loss[loss=0.07348, simple_loss=0.09619, pruned_loss=0.01761, audio_tagging_loss=0.00777, over 15600.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09666, pruned_loss=0.01669, audio_tagging_loss=0.009603, over 3040432.79 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:54:16,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1454820.0, ans=0.125
2023-11-21 09:54:24,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1454886.6666666667, ans=0.0
2023-11-21 09:54:27,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.60 vs. limit=15.0
2023-11-21 09:54:28,218 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 7.963e+01 8.409e+01 9.319e+01 2.355e+02, threshold=1.682e+02, percent-clipped=0.0
2023-11-21 09:54:30,243 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.81 vs. limit=22.5
2023-11-21 09:54:36,382 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-21 09:54:41,233 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218250
2023-11-21 09:54:49,905 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.387e-01
2023-11-21 09:55:10,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.31 vs. limit=15.0
2023-11-21 09:55:13,335 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1850, loss[loss=0.0643, simple_loss=0.07659, pruned_loss=0.01418, audio_tagging_loss=0.01183, over 15968.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.09695, pruned_loss=0.01684, audio_tagging_loss=0.009598, over 3044166.15 frames. ], batch size: 64, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:55:32,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1455220.0, ans=0.125
2023-11-21 09:55:34,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1455220.0, ans=0.125
2023-11-21 09:55:37,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1455286.6666666667, ans=0.2
2023-11-21 09:55:45,233 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218300
2023-11-21 09:56:04,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1455420.0, ans=10.0
2023-11-21 09:56:04,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1455420.0, ans=0.07
2023-11-21 09:56:07,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.52 vs. limit=15.0
2023-11-21 09:56:16,175 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1900, loss[loss=0.08104, simple_loss=0.1086, pruned_loss=0.01817, audio_tagging_loss=0.008564, over 15537.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09678, pruned_loss=0.01681, audio_tagging_loss=0.009522, over 3059566.16 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:56:35,901 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.400e+01 8.910e+01 9.667e+01 1.226e+02, threshold=1.782e+02, percent-clipped=1.0
2023-11-21 09:56:38,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1455553.3333333333, ans=0.0
2023-11-21 09:56:48,870 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218350
2023-11-21 09:56:50,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1455620.0, ans=0.1
2023-11-21 09:57:20,931 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 1950, loss[loss=0.0871, simple_loss=0.1221, pruned_loss=0.01885, audio_tagging_loss=0.007181, over 14717.00 frames. ], tot_loss[loss=0.07519, simple_loss=0.09745, pruned_loss=0.01705, audio_tagging_loss=0.009422, over 3054389.21 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:57:22,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1455820.0, ans=0.125
2023-11-21 09:57:29,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.38 vs. limit=15.0
2023-11-21 09:57:35,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1455886.6666666667, ans=0.05
2023-11-21 09:57:52,294 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218400
2023-11-21 09:58:07,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1456020.0, ans=0.2
2023-11-21 09:58:24,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1456153.3333333333, ans=0.0
2023-11-21 09:58:25,230 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2000, loss[loss=0.05936, simple_loss=0.08135, pruned_loss=0.007851, audio_tagging_loss=0.01084, over 15783.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.09678, pruned_loss=0.01706, audio_tagging_loss=0.009469, over 3052366.37 frames. ], batch size: 60, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:58:40,289 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:58:43,539 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.117e+01 8.612e+01 9.115e+01 1.173e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 09:58:57,079 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218450
2023-11-21 09:58:57,631 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.11 vs. limit=6.0
2023-11-21 09:59:16,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1456420.0, ans=0.0
2023-11-21 09:59:17,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1456420.0, ans=0.125
2023-11-21 09:59:21,325 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:59:21,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.14 vs. limit=15.0
2023-11-21 09:59:28,368 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2050, loss[loss=0.08231, simple_loss=0.1127, pruned_loss=0.01934, audio_tagging_loss=0.006636, over 16210.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.09584, pruned_loss=0.0169, audio_tagging_loss=0.009403, over 3046729.81 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:59:29,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1456486.6666666667, ans=0.125
2023-11-21 09:59:30,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1456486.6666666667, ans=0.1
2023-11-21 09:59:38,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1456486.6666666667, ans=0.1
2023-11-21 09:59:38,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1456486.6666666667, ans=0.0
2023-11-21 09:59:50,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1456553.3333333333, ans=0.125
2023-11-21 09:59:53,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1456620.0, ans=0.2
2023-11-21 10:00:00,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218500
2023-11-21 10:00:05,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1456686.6666666667, ans=0.125
2023-11-21 10:00:12,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1456686.6666666667, ans=0.05
2023-11-21 10:00:31,895 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2100, loss[loss=0.05941, simple_loss=0.07328, pruned_loss=0.01097, audio_tagging_loss=0.0118, over 14983.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09591, pruned_loss=0.01674, audio_tagging_loss=0.009362, over 3042321.09 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:00:32,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1456820.0, ans=0.1
2023-11-21 10:00:33,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.80 vs. limit=15.0
2023-11-21 10:00:33,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.19 vs. limit=15.0
2023-11-21 10:00:52,769 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.577e+01 8.022e+01 8.596e+01 9.289e+01 1.125e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 10:01:04,094 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218550
2023-11-21 10:01:05,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1456953.3333333333, ans=0.035
2023-11-21 10:01:34,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1457086.6666666667, ans=0.0
2023-11-21 10:01:36,285 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2150, loss[loss=0.0688, simple_loss=0.08885, pruned_loss=0.01713, audio_tagging_loss=0.007241, over 15633.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09637, pruned_loss=0.01669, audio_tagging_loss=0.009381, over 3045874.53 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:01:42,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1457153.3333333333, ans=0.2
2023-11-21 10:01:45,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1457153.3333333333, ans=0.2
2023-11-21 10:02:05,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1457286.6666666667, ans=0.0
2023-11-21 10:02:05,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1457286.6666666667, ans=0.125
2023-11-21 10:02:06,994 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218600
2023-11-21 10:02:10,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1457286.6666666667, ans=0.125
2023-11-21 10:02:12,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1457286.6666666667, ans=0.125
2023-11-21 10:02:14,339 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:02:23,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1457353.3333333333, ans=0.0
2023-11-21 10:02:34,032 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.26 vs. limit=22.5
2023-11-21 10:02:39,308 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2200, loss[loss=0.07051, simple_loss=0.09285, pruned_loss=0.01303, audio_tagging_loss=0.01105, over 15044.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09694, pruned_loss=0.01689, audio_tagging_loss=0.009351, over 3052689.50 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:02:49,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1457486.6666666667, ans=0.0
2023-11-21 10:02:53,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1457553.3333333333, ans=0.5
2023-11-21 10:02:55,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1457553.3333333333, ans=10.0
2023-11-21 10:02:59,803 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.151e+01 8.877e+01 9.686e+01 1.220e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 10:03:04,888 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.63 vs. limit=6.0
2023-11-21 10:03:12,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218650
2023-11-21 10:03:16,763 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.53 vs. limit=22.5
2023-11-21 10:03:17,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1457686.6666666667, ans=0.125
2023-11-21 10:03:35,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1457753.3333333333, ans=0.1
2023-11-21 10:03:37,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1457753.3333333333, ans=0.125
2023-11-21 10:03:43,232 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2250, loss[loss=0.1062, simple_loss=0.111, pruned_loss=0.03738, audio_tagging_loss=0.01331, over 15143.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.0971, pruned_loss=0.01694, audio_tagging_loss=0.009443, over 3046833.87 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:03:48,526 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.13 vs. limit=15.0
2023-11-21 10:04:01,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1457886.6666666667, ans=0.1
2023-11-21 10:04:06,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1457886.6666666667, ans=0.0
2023-11-21 10:04:16,231 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218700
2023-11-21 10:04:20,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1457953.3333333333, ans=0.2
2023-11-21 10:04:45,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1458086.6666666667, ans=0.04949747468305833
2023-11-21 10:04:49,119 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2300, loss[loss=0.09152, simple_loss=0.1242, pruned_loss=0.02041, audio_tagging_loss=0.009014, over 14522.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.09681, pruned_loss=0.01705, audio_tagging_loss=0.009536, over 3043691.66 frames. ], batch size: 54, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:05:09,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1458220.0, ans=0.0
2023-11-21 10:05:10,249 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.131e+01 8.990e+01 9.981e+01 1.492e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-21 10:05:16,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1458286.6666666667, ans=0.2
2023-11-21 10:05:20,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218750
2023-11-21 10:05:25,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1458286.6666666667, ans=0.2
2023-11-21 10:05:45,511 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:05:52,969 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2350, loss[loss=0.09584, simple_loss=0.1289, pruned_loss=0.02461, audio_tagging_loss=0.006767, over 15800.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.0973, pruned_loss=0.01691, audio_tagging_loss=0.009534, over 3046228.84 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 8.0
2023-11-21 10:06:02,274 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.59 vs. limit=12.0
2023-11-21 10:06:05,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1458553.3333333333, ans=0.2
2023-11-21 10:06:26,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218800
2023-11-21 10:06:48,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1458753.3333333333, ans=0.125
2023-11-21 10:06:56,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1458820.0, ans=0.2
2023-11-21 10:06:57,865 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2400, loss[loss=0.06284, simple_loss=0.08279, pruned_loss=0.01219, audio_tagging_loss=0.009252, over 14307.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09739, pruned_loss=0.01676, audio_tagging_loss=0.009662, over 3043908.15 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:06:58,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1458820.0, ans=0.125
2023-11-21 10:07:04,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1458820.0, ans=0.0
2023-11-21 10:07:20,775 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.958e+01 7.990e+01 8.660e+01 9.489e+01 1.183e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 10:07:29,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1458953.3333333333, ans=0.015
2023-11-21 10:07:31,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218850
2023-11-21 10:07:55,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1459086.6666666667, ans=0.125
2023-11-21 10:07:57,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1459086.6666666667, ans=0.125
2023-11-21 10:08:03,734 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2450, loss[loss=0.08071, simple_loss=0.1039, pruned_loss=0.01671, audio_tagging_loss=0.01206, over 14725.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09781, pruned_loss=0.01664, audio_tagging_loss=0.009777, over 3043027.18 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:08:19,780 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.49 vs. limit=15.0
2023-11-21 10:08:35,273 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218900
2023-11-21 10:08:36,113 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.80 vs. limit=15.0
2023-11-21 10:08:43,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.55 vs. limit=15.0
2023-11-21 10:08:51,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1459353.3333333333, ans=0.125
2023-11-21 10:09:08,368 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2500, loss[loss=0.0564, simple_loss=0.07574, pruned_loss=0.009982, audio_tagging_loss=0.008552, over 15453.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09728, pruned_loss=0.01669, audio_tagging_loss=0.009874, over 3050201.73 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:09:17,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1459486.6666666667, ans=0.125
2023-11-21 10:09:29,585 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 8.078e+01 8.779e+01 9.710e+01 1.406e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-21 10:09:36,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.66 vs. limit=22.5
2023-11-21 10:09:36,839 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:09:41,151 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 218950
2023-11-21 10:09:50,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1459686.6666666667, ans=0.1
2023-11-21 10:10:00,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1459753.3333333333, ans=0.5
2023-11-21 10:10:12,696 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2550, loss[loss=0.07093, simple_loss=0.08936, pruned_loss=0.01645, audio_tagging_loss=0.009797, over 15450.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09796, pruned_loss=0.01697, audio_tagging_loss=0.009673, over 3045232.89 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:10:45,869 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219000
2023-11-21 10:11:17,702 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2600, loss[loss=0.07753, simple_loss=0.1053, pruned_loss=0.01792, audio_tagging_loss=0.006973, over 14405.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09745, pruned_loss=0.01672, audio_tagging_loss=0.009536, over 3040127.33 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:11:39,749 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.043e+01 8.700e+01 9.465e+01 1.335e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 10:11:42,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1460286.6666666667, ans=0.125
2023-11-21 10:11:47,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1460286.6666666667, ans=0.125
2023-11-21 10:11:48,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1460286.6666666667, ans=0.125
2023-11-21 10:11:49,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219050
2023-11-21 10:11:51,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1460286.6666666667, ans=0.0
2023-11-21 10:12:05,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1460353.3333333333, ans=0.125
2023-11-21 10:12:09,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1460420.0, ans=0.125
2023-11-21 10:12:22,990 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2650, loss[loss=0.07393, simple_loss=0.09965, pruned_loss=0.01447, audio_tagging_loss=0.009638, over 15562.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09682, pruned_loss=0.01665, audio_tagging_loss=0.009601, over 3035643.30 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:12:24,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1460486.6666666667, ans=0.1
2023-11-21 10:12:55,110 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219100
2023-11-21 10:12:59,707 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:13:06,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1460686.6666666667, ans=10.0
2023-11-21 10:13:18,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1460753.3333333333, ans=0.125
2023-11-21 10:13:19,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1460753.3333333333, ans=0.125
2023-11-21 10:13:26,503 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2700, loss[loss=0.07311, simple_loss=0.08555, pruned_loss=0.01779, audio_tagging_loss=0.01254, over 14855.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09739, pruned_loss=0.01689, audio_tagging_loss=0.009614, over 3035403.40 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:13:28,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1460820.0, ans=0.1
2023-11-21 10:13:48,928 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.184e+01 8.729e+01 9.389e+01 1.078e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 10:13:58,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219150
2023-11-21 10:14:08,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1461020.0, ans=0.125
2023-11-21 10:14:31,254 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2750, loss[loss=0.06478, simple_loss=0.08338, pruned_loss=0.01303, audio_tagging_loss=0.01006, over 15633.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09731, pruned_loss=0.01681, audio_tagging_loss=0.009625, over 3043787.25 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:14:44,742 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.28 vs. limit=15.0
2023-11-21 10:15:02,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219200
2023-11-21 10:15:09,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1461353.3333333333, ans=0.125
2023-11-21 10:15:26,985 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:15:35,659 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2800, loss[loss=0.08257, simple_loss=0.1052, pruned_loss=0.02038, audio_tagging_loss=0.009569, over 16122.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09798, pruned_loss=0.01715, audio_tagging_loss=0.009563, over 3048226.51 frames. ], batch size: 61, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 10:15:52,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1461553.3333333333, ans=0.125
2023-11-21 10:15:56,835 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.792e+01 8.334e+01 9.153e+01 1.021e+02 1.703e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-21 10:15:58,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1461553.3333333333, ans=0.125
2023-11-21 10:16:04,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1461620.0, ans=0.125
2023-11-21 10:16:08,010 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219250
2023-11-21 10:16:11,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1461620.0, ans=0.125
2023-11-21 10:16:32,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1461753.3333333333, ans=0.025
2023-11-21 10:16:39,941 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2850, loss[loss=0.0873, simple_loss=0.1134, pruned_loss=0.02196, audio_tagging_loss=0.008657, over 15133.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09754, pruned_loss=0.01705, audio_tagging_loss=0.009598, over 3045263.56 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:16:54,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1461886.6666666667, ans=0.125
2023-11-21 10:17:11,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1461953.3333333333, ans=0.0
2023-11-21 10:17:12,687 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219300
2023-11-21 10:17:20,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1462020.0, ans=0.0
2023-11-21 10:17:22,058 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.73 vs. limit=15.0
2023-11-21 10:17:45,354 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2900, loss[loss=0.05952, simple_loss=0.07323, pruned_loss=0.01255, audio_tagging_loss=0.01035, over 14673.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09785, pruned_loss=0.01733, audio_tagging_loss=0.009561, over 3037957.90 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:17:50,520 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:17:55,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1462153.3333333333, ans=0.125
2023-11-21 10:18:04,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1462220.0, ans=0.5
2023-11-21 10:18:08,042 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.530e+01 8.282e+01 8.695e+01 9.389e+01 1.119e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 10:18:17,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219350
2023-11-21 10:18:38,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1462420.0, ans=0.125
2023-11-21 10:18:47,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1462420.0, ans=0.2
2023-11-21 10:18:49,477 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 2950, loss[loss=0.09659, simple_loss=0.122, pruned_loss=0.0243, audio_tagging_loss=0.01129, over 14347.00 frames. ], tot_loss[loss=0.07575, simple_loss=0.09791, pruned_loss=0.01727, audio_tagging_loss=0.009527, over 3040128.42 frames. ], batch size: 53, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:18:55,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1462486.6666666667, ans=0.0
2023-11-21 10:19:07,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1462553.3333333333, ans=0.125
2023-11-21 10:19:14,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.09 vs. limit=22.5
2023-11-21 10:19:21,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219400
2023-11-21 10:19:35,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1462686.6666666667, ans=0.125
2023-11-21 10:19:47,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.17 vs. limit=15.0
2023-11-21 10:19:53,234 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3000, loss[loss=0.09117, simple_loss=0.111, pruned_loss=0.02743, audio_tagging_loss=0.008262, over 14764.00 frames. ], tot_loss[loss=0.07655, simple_loss=0.09892, pruned_loss=0.01758, audio_tagging_loss=0.009509, over 3039948.77 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:19:53,234 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 10:20:11,238 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.4082, 3.2227, 3.2562, 2.7177, 3.4201, 3.5614, 3.4446, 3.5052],
       device='cuda:2')
2023-11-21 10:20:16,420 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6048, 3.5471, 3.8304, 3.4124], device='cuda:2')
2023-11-21 10:20:28,748 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7542, 5.8081, 5.8674, 5.8503], device='cuda:2')
2023-11-21 10:20:29,265 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.1041, 3.0782, 3.4054, 2.9616, 3.7459, 3.7525, 3.3282, 3.1672],
       device='cuda:2')
2023-11-21 10:20:32,683 INFO [train_asr.py:1253] (2/4) Epoch 19, validation: loss=0.05961, simple_loss=0.05235, pruned_loss=0.005214, audio_tagging_loss=0.02822, over 4681554.00 frames. 
2023-11-21 10:20:32,684 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 10:20:45,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1462886.6666666667, ans=0.0
2023-11-21 10:20:55,279 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.194e+01 8.910e+01 9.878e+01 1.451e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 10:21:00,379 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:21:00,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1462953.3333333333, ans=0.125
2023-11-21 10:21:03,889 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219450
2023-11-21 10:21:07,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1462953.3333333333, ans=0.125
2023-11-21 10:21:09,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1463020.0, ans=0.125
2023-11-21 10:21:20,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1463020.0, ans=0.0
2023-11-21 10:21:37,029 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3050, loss[loss=0.0502, simple_loss=0.05625, pruned_loss=0.009426, audio_tagging_loss=0.01265, over 15906.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.09858, pruned_loss=0.01735, audio_tagging_loss=0.009686, over 3041824.57 frames. ], batch size: 62, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:21:37,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1463153.3333333333, ans=0.0
2023-11-21 10:21:53,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1463220.0, ans=0.125
2023-11-21 10:21:57,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1463220.0, ans=0.2
2023-11-21 10:22:07,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.38 vs. limit=10.0
2023-11-21 10:22:09,619 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219500
2023-11-21 10:22:15,113 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:22:40,805 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3100, loss[loss=0.08009, simple_loss=0.1068, pruned_loss=0.01717, audio_tagging_loss=0.009503, over 15412.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09879, pruned_loss=0.01739, audio_tagging_loss=0.009716, over 3041117.90 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:22:59,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1463553.3333333333, ans=0.5
2023-11-21 10:23:05,415 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.202e+01 8.849e+01 9.632e+01 1.395e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 10:23:14,143 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219550
2023-11-21 10:23:22,318 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.23 vs. limit=10.0
2023-11-21 10:23:32,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1463753.3333333333, ans=0.125
2023-11-21 10:23:34,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1463753.3333333333, ans=0.125
2023-11-21 10:23:45,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1463820.0, ans=0.0
2023-11-21 10:23:46,192 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3150, loss[loss=0.09733, simple_loss=0.1404, pruned_loss=0.02002, audio_tagging_loss=0.007111, over 15257.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09931, pruned_loss=0.01732, audio_tagging_loss=0.009745, over 3043781.02 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:24:17,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219600
2023-11-21 10:24:50,995 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3200, loss[loss=0.06293, simple_loss=0.08436, pruned_loss=0.01283, audio_tagging_loss=0.007914, over 14411.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.09888, pruned_loss=0.01712, audio_tagging_loss=0.00974, over 3046051.37 frames. ], batch size: 53, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:24:51,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1464153.3333333333, ans=0.125
2023-11-21 10:24:55,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.29 vs. limit=15.0
2023-11-21 10:25:08,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1464220.0, ans=0.125
2023-11-21 10:25:11,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1464220.0, ans=0.2
2023-11-21 10:25:13,499 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.202e+01 8.879e+01 9.509e+01 1.631e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 10:25:20,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1464286.6666666667, ans=0.125
2023-11-21 10:25:23,378 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219650
2023-11-21 10:25:33,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1464353.3333333333, ans=0.0
2023-11-21 10:25:54,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten.whitening_limit, batch_count=1464486.6666666667, ans=15.0
2023-11-21 10:25:55,589 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3250, loss[loss=0.1046, simple_loss=0.12, pruned_loss=0.03297, audio_tagging_loss=0.0116, over 16066.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.09864, pruned_loss=0.01714, audio_tagging_loss=0.009727, over 3046862.63 frames. ], batch size: 59, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:26:08,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.94 vs. limit=15.0
2023-11-21 10:26:10,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1464553.3333333333, ans=0.1
2023-11-21 10:26:12,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1464553.3333333333, ans=0.125
2023-11-21 10:26:16,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1464553.3333333333, ans=0.1
2023-11-21 10:26:28,027 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219700
2023-11-21 10:26:28,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1464620.0, ans=0.1
2023-11-21 10:26:41,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=1464686.6666666667, ans=0.02
2023-11-21 10:26:45,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1464753.3333333333, ans=0.5
2023-11-21 10:26:47,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1464753.3333333333, ans=0.0
2023-11-21 10:26:54,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1464753.3333333333, ans=0.0
2023-11-21 10:26:55,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1464753.3333333333, ans=0.0
2023-11-21 10:26:58,970 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3300, loss[loss=0.07817, simple_loss=0.095, pruned_loss=0.01848, audio_tagging_loss=0.0122, over 15271.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.09835, pruned_loss=0.01725, audio_tagging_loss=0.009762, over 3044057.69 frames. ], batch size: 59, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:27:07,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1464820.0, ans=0.125
2023-11-21 10:27:22,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1464886.6666666667, ans=0.0
2023-11-21 10:27:24,140 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.128e+01 8.736e+01 9.388e+01 1.641e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 10:27:29,989 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.87 vs. limit=12.0
2023-11-21 10:27:32,976 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219750
2023-11-21 10:28:06,508 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3350, loss[loss=0.08078, simple_loss=0.1107, pruned_loss=0.01665, audio_tagging_loss=0.008776, over 15189.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09734, pruned_loss=0.01702, audio_tagging_loss=0.009743, over 3048111.67 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:28:06,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1465153.3333333333, ans=0.125
2023-11-21 10:28:20,484 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.76 vs. limit=22.5
2023-11-21 10:28:37,921 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219800
2023-11-21 10:28:43,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1465286.6666666667, ans=0.2
2023-11-21 10:29:11,225 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3400, loss[loss=0.07119, simple_loss=0.1065, pruned_loss=0.01238, audio_tagging_loss=0.005567, over 15449.00 frames. ], tot_loss[loss=0.07547, simple_loss=0.09769, pruned_loss=0.01701, audio_tagging_loss=0.009615, over 3050950.30 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:29:35,140 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.502e+01 9.051e+01 9.796e+01 1.205e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-21 10:29:35,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1465553.3333333333, ans=0.125
2023-11-21 10:29:37,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0
2023-11-21 10:29:43,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1465620.0, ans=0.0
2023-11-21 10:29:45,037 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219850
2023-11-21 10:29:52,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1465686.6666666667, ans=0.1
2023-11-21 10:29:56,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1465686.6666666667, ans=0.1
2023-11-21 10:30:06,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1465753.3333333333, ans=0.125
2023-11-21 10:30:14,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1465820.0, ans=0.125
2023-11-21 10:30:15,864 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3450, loss[loss=0.08152, simple_loss=0.1057, pruned_loss=0.01882, audio_tagging_loss=0.009839, over 15272.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.09693, pruned_loss=0.01701, audio_tagging_loss=0.009518, over 3050348.10 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:30:34,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1465886.6666666667, ans=0.2
2023-11-21 10:30:49,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219900
2023-11-21 10:31:00,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1466020.0, ans=0.125
2023-11-21 10:31:07,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1466086.6666666667, ans=0.125
2023-11-21 10:31:22,087 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3500, loss[loss=0.06391, simple_loss=0.08207, pruned_loss=0.01338, audio_tagging_loss=0.009496, over 14415.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09731, pruned_loss=0.01706, audio_tagging_loss=0.009377, over 3053758.14 frames. ], batch size: 54, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:31:44,237 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.587e+01 8.045e+01 8.820e+01 9.717e+01 1.244e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 10:31:48,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1466286.6666666667, ans=0.0
2023-11-21 10:31:51,228 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.08 vs. limit=15.0
2023-11-21 10:31:52,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 219950
2023-11-21 10:31:53,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1466286.6666666667, ans=0.1
2023-11-21 10:31:54,004 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:32:02,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1466353.3333333333, ans=0.125
2023-11-21 10:32:14,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1466420.0, ans=0.025
2023-11-21 10:32:19,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1466420.0, ans=0.125
2023-11-21 10:32:24,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1466420.0, ans=0.09899494936611666
2023-11-21 10:32:26,293 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3550, loss[loss=0.06697, simple_loss=0.08206, pruned_loss=0.01691, audio_tagging_loss=0.009032, over 15482.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09585, pruned_loss=0.01693, audio_tagging_loss=0.0094, over 3050898.52 frames. ], batch size: 60, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:32:41,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1466553.3333333333, ans=0.2
2023-11-21 10:32:56,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.98 vs. limit=6.0
2023-11-21 10:32:57,083 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.66 vs. limit=22.5
2023-11-21 10:32:59,008 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220000
2023-11-21 10:33:09,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1466686.6666666667, ans=0.0
2023-11-21 10:33:16,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.92 vs. limit=15.0
2023-11-21 10:33:34,062 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3600, loss[loss=0.06823, simple_loss=0.09029, pruned_loss=0.01511, audio_tagging_loss=0.00798, over 15597.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09659, pruned_loss=0.01706, audio_tagging_loss=0.00937, over 3050896.70 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:33:54,036 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.72 vs. limit=15.0
2023-11-21 10:34:00,230 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.124e+01 7.919e+01 8.702e+01 9.494e+01 1.216e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 10:34:06,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1466953.3333333333, ans=0.1
2023-11-21 10:34:07,749 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220050
2023-11-21 10:34:22,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1467020.0, ans=0.0
2023-11-21 10:34:40,917 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3650, loss[loss=0.07256, simple_loss=0.09012, pruned_loss=0.0166, audio_tagging_loss=0.0109, over 14992.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09629, pruned_loss=0.01696, audio_tagging_loss=0.009441, over 3053478.55 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:34:51,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1467153.3333333333, ans=0.125
2023-11-21 10:34:56,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1467220.0, ans=0.1
2023-11-21 10:35:08,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1467286.6666666667, ans=0.1
2023-11-21 10:35:12,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220100
2023-11-21 10:35:18,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1467353.3333333333, ans=0.1
2023-11-21 10:35:25,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1467353.3333333333, ans=0.125
2023-11-21 10:35:44,983 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3700, loss[loss=0.07103, simple_loss=0.09764, pruned_loss=0.0121, audio_tagging_loss=0.01012, over 16002.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09598, pruned_loss=0.01686, audio_tagging_loss=0.009543, over 3060300.73 frames. ], batch size: 60, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:35:48,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1467486.6666666667, ans=0.125
2023-11-21 10:35:52,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1467486.6666666667, ans=0.1
2023-11-21 10:35:58,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1467553.3333333333, ans=0.2
2023-11-21 10:36:08,777 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.151e+01 8.696e+01 9.414e+01 1.695e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 10:36:16,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1467620.0, ans=0.125
2023-11-21 10:36:17,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220150
2023-11-21 10:36:28,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1467686.6666666667, ans=0.125
2023-11-21 10:36:32,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1467686.6666666667, ans=0.125
2023-11-21 10:36:41,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1467753.3333333333, ans=0.0
2023-11-21 10:36:49,685 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3750, loss[loss=0.05899, simple_loss=0.07732, pruned_loss=0.01083, audio_tagging_loss=0.009496, over 16161.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09686, pruned_loss=0.01702, audio_tagging_loss=0.009437, over 3065577.98 frames. ], batch size: 60, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:36:53,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1467820.0, ans=0.125
2023-11-21 10:36:54,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1467820.0, ans=0.125
2023-11-21 10:37:23,609 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220200
2023-11-21 10:37:35,352 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:37:57,245 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3800, loss[loss=0.08432, simple_loss=0.1136, pruned_loss=0.02104, audio_tagging_loss=0.006472, over 15537.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09725, pruned_loss=0.01733, audio_tagging_loss=0.009475, over 3056409.05 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:37:58,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1468153.3333333333, ans=0.0
2023-11-21 10:38:22,691 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.173e+01 8.825e+01 9.624e+01 1.863e+02, threshold=1.765e+02, percent-clipped=1.0
2023-11-21 10:38:25,455 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:38:28,986 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220250
2023-11-21 10:38:35,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1468353.3333333333, ans=0.125
2023-11-21 10:38:35,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1468353.3333333333, ans=0.125
2023-11-21 10:39:01,427 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3850, loss[loss=0.07863, simple_loss=0.1036, pruned_loss=0.01625, audio_tagging_loss=0.01058, over 15506.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09617, pruned_loss=0.01701, audio_tagging_loss=0.009627, over 3050981.53 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:39:19,694 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:39:34,146 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220300
2023-11-21 10:39:34,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1468620.0, ans=0.0
2023-11-21 10:39:39,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1468686.6666666667, ans=0.0
2023-11-21 10:39:52,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1468753.3333333333, ans=0.125
2023-11-21 10:40:02,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.07 vs. limit=12.0
2023-11-21 10:40:06,825 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3900, loss[loss=0.07623, simple_loss=0.107, pruned_loss=0.01562, audio_tagging_loss=0.00711, over 14472.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09613, pruned_loss=0.01711, audio_tagging_loss=0.009627, over 3037525.93 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:40:09,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1468820.0, ans=0.2
2023-11-21 10:40:09,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1468820.0, ans=0.0
2023-11-21 10:40:33,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1468953.3333333333, ans=0.1
2023-11-21 10:40:34,105 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.413e+01 7.996e+01 8.695e+01 9.569e+01 1.427e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 10:40:40,599 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220350
2023-11-21 10:40:42,362 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.31 vs. limit=12.0
2023-11-21 10:41:01,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1469086.6666666667, ans=0.1
2023-11-21 10:41:13,866 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 3950, loss[loss=0.06188, simple_loss=0.07586, pruned_loss=0.01286, audio_tagging_loss=0.01108, over 16221.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09591, pruned_loss=0.01695, audio_tagging_loss=0.009717, over 3039473.15 frames. ], batch size: 61, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:41:18,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1469153.3333333333, ans=0.125
2023-11-21 10:41:20,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1469153.3333333333, ans=0.125
2023-11-21 10:41:22,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1469153.3333333333, ans=0.125
2023-11-21 10:41:46,266 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220400
2023-11-21 10:42:04,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1469353.3333333333, ans=0.1
2023-11-21 10:42:10,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1469420.0, ans=0.125
2023-11-21 10:42:15,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.49 vs. limit=15.0
2023-11-21 10:42:19,413 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4000, loss[loss=0.09708, simple_loss=0.1261, pruned_loss=0.0242, audio_tagging_loss=0.009822, over 15846.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09579, pruned_loss=0.01701, audio_tagging_loss=0.009736, over 3046146.93 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:42:19,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1469486.6666666667, ans=0.07
2023-11-21 10:42:24,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1469486.6666666667, ans=0.125
2023-11-21 10:42:44,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.206e+01 8.898e+01 9.673e+01 1.616e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 10:42:52,375 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220450
2023-11-21 10:43:07,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1469686.6666666667, ans=0.05
2023-11-21 10:43:15,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1469753.3333333333, ans=0.125
2023-11-21 10:43:24,147 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4050, loss[loss=0.08035, simple_loss=0.1026, pruned_loss=0.01574, audio_tagging_loss=0.01333, over 15557.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09572, pruned_loss=0.01689, audio_tagging_loss=0.009841, over 3041155.20 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:43:26,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.88 vs. limit=15.0
2023-11-21 10:43:26,824 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:43:45,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1469886.6666666667, ans=0.125
2023-11-21 10:43:55,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1469953.3333333333, ans=0.125
2023-11-21 10:43:56,963 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220500
2023-11-21 10:44:07,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.87 vs. limit=15.0
2023-11-21 10:44:29,578 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4100, loss[loss=0.06983, simple_loss=0.0866, pruned_loss=0.01482, audio_tagging_loss=0.0117, over 16218.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09617, pruned_loss=0.01682, audio_tagging_loss=0.009811, over 3041918.10 frames. ], batch size: 62, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:44:36,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1470153.3333333333, ans=0.0
2023-11-21 10:44:45,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1470220.0, ans=0.125
2023-11-21 10:44:47,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1470220.0, ans=0.125
2023-11-21 10:44:49,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1470220.0, ans=0.1
2023-11-21 10:44:55,160 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.123e+01 8.556e+01 9.611e+01 1.349e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 10:44:57,108 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.43 vs. limit=15.0
2023-11-21 10:45:01,829 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220550
2023-11-21 10:45:02,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1470286.6666666667, ans=0.0
2023-11-21 10:45:08,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1470353.3333333333, ans=0.0
2023-11-21 10:45:16,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1470353.3333333333, ans=0.125
2023-11-21 10:45:34,939 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.03 vs. limit=22.5
2023-11-21 10:45:35,381 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4150, loss[loss=0.0588, simple_loss=0.0721, pruned_loss=0.01228, audio_tagging_loss=0.01048, over 13772.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09585, pruned_loss=0.01681, audio_tagging_loss=0.009673, over 3044978.83 frames. ], batch size: 54, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:45:50,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1470553.3333333333, ans=0.2
2023-11-21 10:45:50,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.11 vs. limit=15.0
2023-11-21 10:45:51,785 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.51 vs. limit=12.0
2023-11-21 10:46:06,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1470620.0, ans=0.125
2023-11-21 10:46:08,155 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220600
2023-11-21 10:46:09,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1470620.0, ans=0.1
2023-11-21 10:46:11,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1470620.0, ans=0.0
2023-11-21 10:46:12,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1470620.0, ans=0.0
2023-11-21 10:46:14,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1470686.6666666667, ans=0.125
2023-11-21 10:46:22,596 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:46:25,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1470686.6666666667, ans=0.125
2023-11-21 10:46:27,067 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0
2023-11-21 10:46:27,850 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:46:37,023 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.41 vs. limit=6.0
2023-11-21 10:46:39,652 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4200, loss[loss=0.07432, simple_loss=0.09536, pruned_loss=0.01748, audio_tagging_loss=0.00916, over 13993.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09704, pruned_loss=0.0171, audio_tagging_loss=0.00956, over 3047727.54 frames. ], batch size: 53, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:46:39,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1470820.0, ans=0.0
2023-11-21 10:46:47,071 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.32 vs. limit=22.5
2023-11-21 10:47:06,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.111e+01 8.714e+01 9.250e+01 1.135e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 10:47:12,913 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220650
2023-11-21 10:47:30,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1471086.6666666667, ans=0.125
2023-11-21 10:47:39,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1471086.6666666667, ans=0.0
2023-11-21 10:47:44,739 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4250, loss[loss=0.06951, simple_loss=0.08849, pruned_loss=0.01473, audio_tagging_loss=0.01053, over 14859.00 frames. ], tot_loss[loss=0.07526, simple_loss=0.09729, pruned_loss=0.01718, audio_tagging_loss=0.00944, over 3051652.86 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:47:45,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1471153.3333333333, ans=0.125
2023-11-21 10:48:14,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1471286.6666666667, ans=0.125
2023-11-21 10:48:17,314 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220700
2023-11-21 10:48:26,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1471353.3333333333, ans=0.125
2023-11-21 10:48:46,342 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.25 vs. limit=15.0
2023-11-21 10:48:50,615 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4300, loss[loss=0.08442, simple_loss=0.1076, pruned_loss=0.02176, audio_tagging_loss=0.008842, over 15935.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09826, pruned_loss=0.01728, audio_tagging_loss=0.009403, over 3053061.45 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:49:01,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.76 vs. limit=22.5
2023-11-21 10:49:15,458 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.061e+01 8.689e+01 9.437e+01 1.140e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 10:49:20,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1471620.0, ans=0.2
2023-11-21 10:49:21,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1471620.0, ans=0.2
2023-11-21 10:49:22,420 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220750
2023-11-21 10:49:28,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1471686.6666666667, ans=0.5
2023-11-21 10:49:48,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1471753.3333333333, ans=0.0
2023-11-21 10:49:50,456 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.57 vs. limit=15.0
2023-11-21 10:49:54,734 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4350, loss[loss=0.06605, simple_loss=0.08939, pruned_loss=0.01281, audio_tagging_loss=0.008544, over 15987.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09613, pruned_loss=0.01683, audio_tagging_loss=0.009469, over 3053967.38 frames. ], batch size: 60, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:49:57,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1471820.0, ans=0.125
2023-11-21 10:50:11,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1471886.6666666667, ans=0.1
2023-11-21 10:50:27,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1471953.3333333333, ans=0.125
2023-11-21 10:50:28,523 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220800
2023-11-21 10:50:29,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.83 vs. limit=15.0
2023-11-21 10:50:32,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1471953.3333333333, ans=0.0
2023-11-21 10:50:39,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1472020.0, ans=0.0
2023-11-21 10:51:00,779 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4400, loss[loss=0.05856, simple_loss=0.07407, pruned_loss=0.009514, audio_tagging_loss=0.01201, over 15545.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09649, pruned_loss=0.01687, audio_tagging_loss=0.009427, over 3052874.93 frames. ], batch size: 59, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:51:06,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1472153.3333333333, ans=0.125
2023-11-21 10:51:09,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1472153.3333333333, ans=0.0
2023-11-21 10:51:15,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1472220.0, ans=0.125
2023-11-21 10:51:27,110 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.481e+01 8.151e+01 8.586e+01 9.427e+01 1.147e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-21 10:51:33,383 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220850
2023-11-21 10:51:49,658 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:52:06,493 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4450, loss[loss=0.07316, simple_loss=0.09654, pruned_loss=0.01531, audio_tagging_loss=0.009579, over 15500.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09747, pruned_loss=0.01712, audio_tagging_loss=0.00948, over 3055126.70 frames. ], batch size: 60, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:52:07,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.95 vs. limit=22.5
2023-11-21 10:52:29,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1472553.3333333333, ans=0.0
2023-11-21 10:52:31,957 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:52:38,096 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220900
2023-11-21 10:53:01,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1472753.3333333333, ans=0.2
2023-11-21 10:53:11,305 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4500, loss[loss=0.06092, simple_loss=0.06695, pruned_loss=0.01243, audio_tagging_loss=0.01501, over 15764.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09644, pruned_loss=0.01675, audio_tagging_loss=0.009515, over 3055588.74 frames. ], batch size: 61, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:53:15,370 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:53:29,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1472886.6666666667, ans=0.125
2023-11-21 10:53:38,077 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.114e+01 8.781e+01 9.396e+01 1.182e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-21 10:53:39,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1472953.3333333333, ans=0.125
2023-11-21 10:53:45,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 220950
2023-11-21 10:54:06,083 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.46 vs. limit=10.0
2023-11-21 10:54:15,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.03 vs. limit=15.0
2023-11-21 10:54:16,245 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4550, loss[loss=0.05331, simple_loss=0.06625, pruned_loss=0.01066, audio_tagging_loss=0.009522, over 15357.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09666, pruned_loss=0.01688, audio_tagging_loss=0.009489, over 3055009.89 frames. ], batch size: 61, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:54:19,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1473153.3333333333, ans=0.125
2023-11-21 10:54:47,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1473286.6666666667, ans=0.0
2023-11-21 10:54:49,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221000
2023-11-21 10:54:56,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1473353.3333333333, ans=0.0
2023-11-21 10:54:57,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1473353.3333333333, ans=0.1
2023-11-21 10:54:58,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1473353.3333333333, ans=0.125
2023-11-21 10:55:03,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1473353.3333333333, ans=0.0
2023-11-21 10:55:05,207 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:55:07,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1473420.0, ans=0.125
2023-11-21 10:55:07,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1473420.0, ans=0.125
2023-11-21 10:55:16,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1473420.0, ans=10.0
2023-11-21 10:55:20,168 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.63 vs. limit=6.0
2023-11-21 10:55:22,530 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4600, loss[loss=0.04532, simple_loss=0.0521, pruned_loss=0.00654, audio_tagging_loss=0.01273, over 14328.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.09576, pruned_loss=0.0166, audio_tagging_loss=0.009552, over 3051988.19 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:55:30,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1473486.6666666667, ans=10.0
2023-11-21 10:55:48,766 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.454e+01 8.163e+01 8.661e+01 9.433e+01 1.192e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 10:55:53,777 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221050
2023-11-21 10:56:12,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1473686.6666666667, ans=0.0
2023-11-21 10:56:18,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1473753.3333333333, ans=0.125
2023-11-21 10:56:22,252 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:56:27,011 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4650, loss[loss=0.07305, simple_loss=0.09497, pruned_loss=0.01935, audio_tagging_loss=0.006215, over 14758.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09604, pruned_loss=0.01664, audio_tagging_loss=0.009657, over 3048981.44 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:56:45,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1473886.6666666667, ans=0.125
2023-11-21 10:56:59,998 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221100
2023-11-21 10:57:03,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1473953.3333333333, ans=0.2
2023-11-21 10:57:16,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1474020.0, ans=0.125
2023-11-21 10:57:31,377 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4700, loss[loss=0.08099, simple_loss=0.09623, pruned_loss=0.01778, audio_tagging_loss=0.0151, over 14690.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09657, pruned_loss=0.01669, audio_tagging_loss=0.009692, over 3056938.84 frames. ], batch size: 54, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:57:32,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1474153.3333333333, ans=0.035
2023-11-21 10:57:44,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1474220.0, ans=0.2
2023-11-21 10:57:45,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1474220.0, ans=0.1
2023-11-21 10:57:59,449 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.726e+01 8.412e+01 8.839e+01 9.940e+01 1.268e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 10:58:04,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221150
2023-11-21 10:58:12,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1474353.3333333333, ans=0.125
2023-11-21 10:58:37,199 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4750, loss[loss=0.08283, simple_loss=0.1068, pruned_loss=0.01923, audio_tagging_loss=0.01022, over 15349.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09618, pruned_loss=0.01668, audio_tagging_loss=0.009766, over 3052657.29 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:58:45,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1474486.6666666667, ans=15.0
2023-11-21 10:59:09,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221200
2023-11-21 10:59:39,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1474753.3333333333, ans=0.1
2023-11-21 10:59:43,345 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4800, loss[loss=0.08775, simple_loss=0.1231, pruned_loss=0.01833, audio_tagging_loss=0.007881, over 13781.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09549, pruned_loss=0.01659, audio_tagging_loss=0.009923, over 3045166.14 frames. ], batch size: 52, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 11:00:04,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1474886.6666666667, ans=0.0
2023-11-21 11:00:10,203 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.162e+01 8.841e+01 9.779e+01 1.301e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 11:00:10,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1474953.3333333333, ans=0.1
2023-11-21 11:00:13,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1474953.3333333333, ans=0.2
2023-11-21 11:00:15,882 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221250
2023-11-21 11:00:21,971 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.87 vs. limit=12.0
2023-11-21 11:00:22,821 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:00:29,733 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.52 vs. limit=15.0
2023-11-21 11:00:47,765 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4850, loss[loss=0.07279, simple_loss=0.09762, pruned_loss=0.01413, audio_tagging_loss=0.009857, over 14796.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.09535, pruned_loss=0.0164, audio_tagging_loss=0.009954, over 3038531.51 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:01:08,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1475220.0, ans=0.2
2023-11-21 11:01:19,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1475286.6666666667, ans=0.0
2023-11-21 11:01:21,615 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221300
2023-11-21 11:01:52,958 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4900, loss[loss=0.08716, simple_loss=0.1195, pruned_loss=0.01987, audio_tagging_loss=0.007524, over 14589.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09493, pruned_loss=0.01625, audio_tagging_loss=0.009987, over 3035309.79 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:02:20,531 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.27 vs. limit=12.0
2023-11-21 11:02:20,760 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.034e+01 8.624e+01 9.399e+01 1.492e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 11:02:24,619 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221350
2023-11-21 11:02:24,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1475620.0, ans=0.125
2023-11-21 11:02:49,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1475753.3333333333, ans=0.2
2023-11-21 11:02:57,370 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 4950, loss[loss=0.06932, simple_loss=0.08845, pruned_loss=0.01579, audio_tagging_loss=0.009306, over 14694.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09461, pruned_loss=0.01614, audio_tagging_loss=0.009807, over 3036383.68 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:02:59,430 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.91 vs. limit=15.0
2023-11-21 11:03:12,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1475886.6666666667, ans=0.95
2023-11-21 11:03:13,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1475886.6666666667, ans=0.2
2023-11-21 11:03:18,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1475886.6666666667, ans=0.0
2023-11-21 11:03:20,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1475886.6666666667, ans=0.1
2023-11-21 11:03:24,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.70 vs. limit=15.0
2023-11-21 11:03:25,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1475953.3333333333, ans=0.0
2023-11-21 11:03:26,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1475953.3333333333, ans=0.1
2023-11-21 11:03:29,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221400
2023-11-21 11:03:35,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1476020.0, ans=0.5
2023-11-21 11:04:00,610 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=15.15 vs. limit=15.0
2023-11-21 11:04:02,073 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5000, loss[loss=0.07413, simple_loss=0.09153, pruned_loss=0.01594, audio_tagging_loss=0.01243, over 15088.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.09389, pruned_loss=0.01597, audio_tagging_loss=0.009743, over 3037745.61 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:04:02,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1476153.3333333333, ans=0.2
2023-11-21 11:04:15,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1476220.0, ans=0.0
2023-11-21 11:04:31,462 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.334e+01 8.366e+01 9.036e+01 1.004e+02 1.239e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 11:04:35,416 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221450
2023-11-21 11:04:39,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1476286.6666666667, ans=0.2
2023-11-21 11:04:44,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1476353.3333333333, ans=0.0
2023-11-21 11:04:46,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1476353.3333333333, ans=0.125
2023-11-21 11:04:50,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1476353.3333333333, ans=0.125
2023-11-21 11:04:59,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1476420.0, ans=0.0
2023-11-21 11:05:07,097 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5050, loss[loss=0.07264, simple_loss=0.09591, pruned_loss=0.01609, audio_tagging_loss=0.008597, over 13598.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09486, pruned_loss=0.01617, audio_tagging_loss=0.009614, over 3036496.72 frames. ], batch size: 53, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:05:08,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.53 vs. limit=15.0
2023-11-21 11:05:11,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1476486.6666666667, ans=0.2
2023-11-21 11:05:19,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.48 vs. limit=8.0
2023-11-21 11:05:29,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1476553.3333333333, ans=0.2
2023-11-21 11:05:39,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221500
2023-11-21 11:06:02,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1476753.3333333333, ans=0.125
2023-11-21 11:06:12,964 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5100, loss[loss=0.06313, simple_loss=0.08482, pruned_loss=0.01144, audio_tagging_loss=0.009278, over 15915.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.0952, pruned_loss=0.01629, audio_tagging_loss=0.009542, over 3040457.69 frames. ], batch size: 62, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:06:14,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1476820.0, ans=0.0
2023-11-21 11:06:15,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1476820.0, ans=0.0
2023-11-21 11:06:15,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1476820.0, ans=0.0
2023-11-21 11:06:30,596 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.67 vs. limit=15.0
2023-11-21 11:06:34,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.27 vs. limit=10.0
2023-11-21 11:06:40,830 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.039e+01 8.516e+01 9.144e+01 1.339e+02, threshold=1.703e+02, percent-clipped=0.0
2023-11-21 11:06:45,271 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221550
2023-11-21 11:06:56,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1477020.0, ans=0.0
2023-11-21 11:07:08,565 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-21 11:07:18,250 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5150, loss[loss=0.1121, simple_loss=0.1545, pruned_loss=0.02727, audio_tagging_loss=0.007544, over 15790.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09564, pruned_loss=0.01652, audio_tagging_loss=0.009458, over 3037259.88 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:07:32,420 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.70 vs. limit=6.0
2023-11-21 11:07:35,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.83 vs. limit=15.0
2023-11-21 11:07:51,192 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221600
2023-11-21 11:08:06,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1477353.3333333333, ans=0.125
2023-11-21 11:08:23,351 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5200, loss[loss=0.05626, simple_loss=0.07164, pruned_loss=0.01173, audio_tagging_loss=0.008709, over 15274.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09672, pruned_loss=0.01678, audio_tagging_loss=0.009342, over 3040173.09 frames. ], batch size: 60, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 11:08:30,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1477486.6666666667, ans=0.0
2023-11-21 11:08:37,712 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.27 vs. limit=15.0
2023-11-21 11:08:46,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1477553.3333333333, ans=0.125
2023-11-21 11:08:49,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1477620.0, ans=0.0
2023-11-21 11:08:51,500 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.310e+01 8.141e+01 8.886e+01 9.437e+01 1.227e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 11:08:51,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1477620.0, ans=0.0
2023-11-21 11:08:54,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1477620.0, ans=0.0
2023-11-21 11:08:55,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221650
2023-11-21 11:09:12,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1477686.6666666667, ans=0.2
2023-11-21 11:09:28,801 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5250, loss[loss=0.06975, simple_loss=0.0882, pruned_loss=0.01445, audio_tagging_loss=0.0112, over 14686.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09685, pruned_loss=0.01711, audio_tagging_loss=0.009338, over 3045143.32 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 11:09:32,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1477820.0, ans=10.0
2023-11-21 11:09:34,459 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.50 vs. limit=15.0
2023-11-21 11:09:38,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1477820.0, ans=0.125
2023-11-21 11:09:50,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1477886.6666666667, ans=0.1
2023-11-21 11:09:54,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.86 vs. limit=15.0
2023-11-21 11:10:00,083 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221700
2023-11-21 11:10:03,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1477953.3333333333, ans=0.1
2023-11-21 11:10:07,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1478020.0, ans=0.125
2023-11-21 11:10:09,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1478020.0, ans=0.09899494936611666
2023-11-21 11:10:19,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1478086.6666666667, ans=0.1
2023-11-21 11:10:24,181 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:10:26,607 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:10:32,526 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5300, loss[loss=0.07586, simple_loss=0.1013, pruned_loss=0.01642, audio_tagging_loss=0.008758, over 15984.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09718, pruned_loss=0.01703, audio_tagging_loss=0.009314, over 3042892.25 frames. ], batch size: 60, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:10:42,542 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=15.0
2023-11-21 11:10:55,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1478220.0, ans=0.125
2023-11-21 11:11:01,610 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.119e+01 8.657e+01 9.101e+01 1.165e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 11:11:06,040 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221750
2023-11-21 11:11:32,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1478420.0, ans=0.0
2023-11-21 11:11:36,917 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5350, loss[loss=0.07475, simple_loss=0.1029, pruned_loss=0.01575, audio_tagging_loss=0.007541, over 14137.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09666, pruned_loss=0.01698, audio_tagging_loss=0.009421, over 3035414.26 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:11:49,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.80 vs. limit=6.0
2023-11-21 11:12:09,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221800
2023-11-21 11:12:21,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1478686.6666666667, ans=0.1
2023-11-21 11:12:21,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1478686.6666666667, ans=0.05
2023-11-21 11:12:39,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1478753.3333333333, ans=0.0
2023-11-21 11:12:42,779 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5400, loss[loss=0.05886, simple_loss=0.07656, pruned_loss=0.01126, audio_tagging_loss=0.009322, over 15712.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09714, pruned_loss=0.01711, audio_tagging_loss=0.00934, over 3044877.40 frames. ], batch size: 60, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:12:48,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1478820.0, ans=0.125
2023-11-21 11:12:54,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.13 vs. limit=15.0
2023-11-21 11:13:11,529 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.155e+01 8.706e+01 9.355e+01 1.764e+02, threshold=1.741e+02, percent-clipped=1.0
2023-11-21 11:13:14,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221850
2023-11-21 11:13:27,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1479020.0, ans=0.125
2023-11-21 11:13:32,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1479020.0, ans=0.125
2023-11-21 11:13:46,682 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5450, loss[loss=0.07573, simple_loss=0.1072, pruned_loss=0.0153, audio_tagging_loss=0.006823, over 14943.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09656, pruned_loss=0.017, audio_tagging_loss=0.009475, over 3044645.26 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:13:50,672 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:14:02,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1479220.0, ans=0.0
2023-11-21 11:14:05,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1479220.0, ans=0.125
2023-11-21 11:14:17,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1479286.6666666667, ans=0.0
2023-11-21 11:14:19,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221900
2023-11-21 11:14:30,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=22.5
2023-11-21 11:14:35,865 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:14:50,958 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5500, loss[loss=0.07399, simple_loss=0.09536, pruned_loss=0.01605, audio_tagging_loss=0.01026, over 16395.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.0969, pruned_loss=0.01709, audio_tagging_loss=0.009524, over 3047391.84 frames. ], batch size: 61, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:15:07,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1479553.3333333333, ans=0.125
2023-11-21 11:15:11,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1479553.3333333333, ans=0.0
2023-11-21 11:15:21,344 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.123e+01 9.057e+01 9.951e+01 1.227e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-21 11:15:23,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 221950
2023-11-21 11:15:39,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1479686.6666666667, ans=0.125
2023-11-21 11:15:50,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1479753.3333333333, ans=0.07
2023-11-21 11:15:56,238 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5550, loss[loss=0.06202, simple_loss=0.08107, pruned_loss=0.01327, audio_tagging_loss=0.008217, over 14883.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09595, pruned_loss=0.01699, audio_tagging_loss=0.009616, over 3050806.16 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:15:56,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1479820.0, ans=0.125
2023-11-21 11:16:04,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1479820.0, ans=0.125
2023-11-21 11:16:10,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0
2023-11-21 11:16:27,021 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222000
2023-11-21 11:16:31,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1479953.3333333333, ans=0.1
2023-11-21 11:16:44,202 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.36 vs. limit=22.5
2023-11-21 11:16:48,353 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.11 vs. limit=10.0
2023-11-21 11:16:52,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1480086.6666666667, ans=0.0
2023-11-21 11:17:00,034 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5600, loss[loss=0.05179, simple_loss=0.06458, pruned_loss=0.009514, audio_tagging_loss=0.009987, over 14595.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09725, pruned_loss=0.01723, audio_tagging_loss=0.009724, over 3055691.51 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:17:00,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1480153.3333333333, ans=0.05
2023-11-21 11:17:04,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1480153.3333333333, ans=0.125
2023-11-21 11:17:12,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1480220.0, ans=0.0
2023-11-21 11:17:13,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1480220.0, ans=0.1
2023-11-21 11:17:14,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.31 vs. limit=10.0
2023-11-21 11:17:16,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1480220.0, ans=0.1
2023-11-21 11:17:20,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1480220.0, ans=0.07
2023-11-21 11:17:21,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1480220.0, ans=0.125
2023-11-21 11:17:30,534 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.739e+01 7.938e+01 8.660e+01 9.316e+01 1.150e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 11:17:31,883 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222050
2023-11-21 11:17:44,696 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 11:17:45,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.09 vs. limit=22.5
2023-11-21 11:18:02,765 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5650, loss[loss=0.08014, simple_loss=0.1089, pruned_loss=0.01598, audio_tagging_loss=0.009717, over 15795.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09691, pruned_loss=0.01713, audio_tagging_loss=0.009771, over 3051223.60 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:18:07,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1480486.6666666667, ans=0.125
2023-11-21 11:18:09,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1480486.6666666667, ans=0.2
2023-11-21 11:18:34,760 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:18:35,855 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222100
2023-11-21 11:18:57,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1480753.3333333333, ans=0.125
2023-11-21 11:19:07,292 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5700, loss[loss=0.08441, simple_loss=0.1119, pruned_loss=0.02065, audio_tagging_loss=0.007802, over 14745.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.09767, pruned_loss=0.01722, audio_tagging_loss=0.009741, over 3046475.23 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:19:15,703 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.82 vs. limit=22.5
2023-11-21 11:19:23,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1480886.6666666667, ans=0.2
2023-11-21 11:19:24,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1480886.6666666667, ans=0.0
2023-11-21 11:19:29,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1480886.6666666667, ans=0.125
2023-11-21 11:19:36,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1480953.3333333333, ans=0.2
2023-11-21 11:19:37,368 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.400e+01 9.034e+01 9.970e+01 1.319e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 11:19:38,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222150
2023-11-21 11:19:40,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.69 vs. limit=15.0
2023-11-21 11:19:51,742 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=6.755e-02
2023-11-21 11:19:53,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1481020.0, ans=6.0
2023-11-21 11:19:54,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1481020.0, ans=0.125
2023-11-21 11:20:09,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.82 vs. limit=15.0
2023-11-21 11:20:11,590 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5750, loss[loss=0.05762, simple_loss=0.06444, pruned_loss=0.01606, audio_tagging_loss=0.00934, over 14428.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09636, pruned_loss=0.01701, audio_tagging_loss=0.009692, over 3045445.31 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:20:37,768 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:20:43,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222200
2023-11-21 11:20:47,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1481286.6666666667, ans=0.2
2023-11-21 11:21:04,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1481420.0, ans=0.0
2023-11-21 11:21:09,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1481420.0, ans=0.125
2023-11-21 11:21:14,843 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5800, loss[loss=0.08618, simple_loss=0.1173, pruned_loss=0.01853, audio_tagging_loss=0.009024, over 14633.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.09754, pruned_loss=0.01718, audio_tagging_loss=0.009544, over 3044552.81 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:21:16,706 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.16 vs. limit=12.0
2023-11-21 11:21:28,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1481553.3333333333, ans=0.125
2023-11-21 11:21:42,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1481620.0, ans=0.1
2023-11-21 11:21:46,278 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.053e+01 8.570e+01 9.375e+01 1.179e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 11:21:47,742 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222250
2023-11-21 11:21:50,828 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.29 vs. limit=6.0
2023-11-21 11:21:58,080 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.20 vs. limit=15.0
2023-11-21 11:22:00,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1481686.6666666667, ans=0.125
2023-11-21 11:22:01,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1481686.6666666667, ans=0.0
2023-11-21 11:22:15,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=1481753.3333333333, ans=10.0
2023-11-21 11:22:18,581 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5850, loss[loss=0.06404, simple_loss=0.08252, pruned_loss=0.01332, audio_tagging_loss=0.009459, over 14407.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09753, pruned_loss=0.01719, audio_tagging_loss=0.009577, over 3043145.03 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:22:25,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1481820.0, ans=0.2
2023-11-21 11:22:32,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1481886.6666666667, ans=0.07
2023-11-21 11:22:34,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1481886.6666666667, ans=0.1
2023-11-21 11:22:50,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222300
2023-11-21 11:22:59,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1482020.0, ans=0.1
2023-11-21 11:23:06,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1482020.0, ans=0.2
2023-11-21 11:23:11,828 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:23:22,444 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5900, loss[loss=0.07596, simple_loss=0.1018, pruned_loss=0.01618, audio_tagging_loss=0.008878, over 15526.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.09681, pruned_loss=0.017, audio_tagging_loss=0.009548, over 3038695.28 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:23:23,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1482153.3333333333, ans=0.125
2023-11-21 11:23:28,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.03 vs. limit=10.0
2023-11-21 11:23:34,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1482220.0, ans=0.07
2023-11-21 11:23:38,368 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=5.045e-03
2023-11-21 11:23:51,653 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.184e+01 8.891e+01 9.441e+01 1.396e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 11:23:52,968 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222350
2023-11-21 11:24:11,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1482420.0, ans=0.125
2023-11-21 11:24:16,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.98 vs. limit=10.0
2023-11-21 11:24:24,462 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 5950, loss[loss=0.07754, simple_loss=0.1045, pruned_loss=0.0183, audio_tagging_loss=0.006973, over 14692.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.0967, pruned_loss=0.01696, audio_tagging_loss=0.009551, over 3041054.91 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:24:34,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1482486.6666666667, ans=0.125
2023-11-21 11:24:52,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1482620.0, ans=0.125
2023-11-21 11:24:57,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222400
2023-11-21 11:25:12,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1482686.6666666667, ans=0.0
2023-11-21 11:25:27,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1482820.0, ans=0.125
2023-11-21 11:25:28,518 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6000, loss[loss=0.1033, simple_loss=0.1414, pruned_loss=0.02495, audio_tagging_loss=0.007642, over 15549.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09573, pruned_loss=0.01675, audio_tagging_loss=0.009592, over 3042719.36 frames. ], batch size: 59, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:25:28,518 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 11:26:10,018 INFO [train_asr.py:1253] (2/4) Epoch 19, validation: loss=0.05983, simple_loss=0.05236, pruned_loss=0.005293, audio_tagging_loss=0.02835, over 4681554.00 frames. 
2023-11-21 11:26:10,019 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 11:26:10,821 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.15 vs. limit=22.5
2023-11-21 11:26:14,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1482820.0, ans=0.0
2023-11-21 11:26:27,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1482886.6666666667, ans=0.0
2023-11-21 11:26:30,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1482886.6666666667, ans=0.2
2023-11-21 11:26:33,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1482953.3333333333, ans=10.0
2023-11-21 11:26:38,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1482953.3333333333, ans=0.125
2023-11-21 11:26:40,197 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.474e+01 7.915e+01 8.568e+01 9.449e+01 1.142e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 11:26:41,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222450
2023-11-21 11:26:43,220 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.78 vs. limit=12.0
2023-11-21 11:26:55,904 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 11:26:57,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1483020.0, ans=0.125
2023-11-21 11:26:59,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1483086.6666666667, ans=0.0
2023-11-21 11:27:12,935 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6050, loss[loss=0.09202, simple_loss=0.1225, pruned_loss=0.02466, audio_tagging_loss=0.006113, over 16665.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09684, pruned_loss=0.0168, audio_tagging_loss=0.009497, over 3055987.53 frames. ], batch size: 60, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:27:45,800 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222500
2023-11-21 11:27:52,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1483353.3333333333, ans=0.125
2023-11-21 11:28:05,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1483420.0, ans=0.035
2023-11-21 11:28:16,885 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6100, loss[loss=0.08182, simple_loss=0.1171, pruned_loss=0.01672, audio_tagging_loss=0.006541, over 14649.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09642, pruned_loss=0.0168, audio_tagging_loss=0.009487, over 3047289.07 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:28:32,930 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.38 vs. limit=15.0
2023-11-21 11:28:47,589 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.060e+01 8.165e+01 8.686e+01 9.448e+01 1.325e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 11:28:48,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222550
2023-11-21 11:29:00,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.23 vs. limit=15.0
2023-11-21 11:29:21,025 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6150, loss[loss=0.062, simple_loss=0.07032, pruned_loss=0.01324, audio_tagging_loss=0.0136, over 15012.00 frames. ], tot_loss[loss=0.07442, simple_loss=0.09621, pruned_loss=0.01675, audio_tagging_loss=0.009565, over 3044859.69 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:29:23,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1483820.0, ans=0.95
2023-11-21 11:29:52,913 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222600
2023-11-21 11:29:59,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1484020.0, ans=0.1
2023-11-21 11:30:00,303 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:30:12,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1484086.6666666667, ans=0.125
2023-11-21 11:30:12,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1484086.6666666667, ans=10.0
2023-11-21 11:30:25,249 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6200, loss[loss=0.06538, simple_loss=0.07821, pruned_loss=0.01363, audio_tagging_loss=0.01265, over 15395.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09636, pruned_loss=0.01681, audio_tagging_loss=0.009682, over 3043470.36 frames. ], batch size: 59, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:30:26,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1484153.3333333333, ans=0.0
2023-11-21 11:30:29,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1484153.3333333333, ans=0.04949747468305833
2023-11-21 11:30:55,991 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.122e+01 8.795e+01 9.552e+01 1.328e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 11:30:57,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222650
2023-11-21 11:31:02,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.89 vs. limit=15.0
2023-11-21 11:31:17,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1484420.0, ans=0.2
2023-11-21 11:31:19,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1484420.0, ans=0.125
2023-11-21 11:31:28,749 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6250, loss[loss=0.1124, simple_loss=0.1591, pruned_loss=0.02763, audio_tagging_loss=0.005244, over 15839.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09543, pruned_loss=0.01671, audio_tagging_loss=0.009893, over 3046487.23 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:31:33,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1484486.6666666667, ans=0.125
2023-11-21 11:32:01,093 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222700
2023-11-21 11:32:02,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1484620.0, ans=0.125
2023-11-21 11:32:07,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1484686.6666666667, ans=0.125
2023-11-21 11:32:33,437 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6300, loss[loss=0.07108, simple_loss=0.0872, pruned_loss=0.01833, audio_tagging_loss=0.009152, over 13511.00 frames. ], tot_loss[loss=0.07477, simple_loss=0.09614, pruned_loss=0.01683, audio_tagging_loss=0.009868, over 3048745.20 frames. ], batch size: 52, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:32:33,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1484820.0, ans=0.125
2023-11-21 11:32:42,459 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.17 vs. limit=15.0
2023-11-21 11:32:42,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.68 vs. limit=15.0
2023-11-21 11:33:03,727 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.093e+01 8.788e+01 9.540e+01 1.292e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 11:33:05,083 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222750
2023-11-21 11:33:09,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1484953.3333333333, ans=0.0
2023-11-21 11:33:37,374 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6350, loss[loss=0.07711, simple_loss=0.08795, pruned_loss=0.02127, audio_tagging_loss=0.01186, over 15884.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09668, pruned_loss=0.01691, audio_tagging_loss=0.009863, over 3053215.90 frames. ], batch size: 62, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:33:59,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1485220.0, ans=0.0
2023-11-21 11:34:09,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222800
2023-11-21 11:34:41,974 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6400, loss[loss=0.07812, simple_loss=0.09967, pruned_loss=0.0163, audio_tagging_loss=0.01199, over 14935.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09641, pruned_loss=0.01683, audio_tagging_loss=0.009902, over 3047131.58 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:34:53,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1485553.3333333333, ans=0.0
2023-11-21 11:35:06,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1485620.0, ans=0.125
2023-11-21 11:35:07,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=20.01 vs. limit=15.0
2023-11-21 11:35:12,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.327e+01 9.131e+01 1.015e+02 1.534e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-21 11:35:13,768 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222850
2023-11-21 11:35:36,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1485753.3333333333, ans=0.1
2023-11-21 11:35:46,474 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6450, loss[loss=0.07812, simple_loss=0.09801, pruned_loss=0.01777, audio_tagging_loss=0.01134, over 16802.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09597, pruned_loss=0.01682, audio_tagging_loss=0.01007, over 3051956.20 frames. ], batch size: 65, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:35:47,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1485820.0, ans=0.125
2023-11-21 11:35:56,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1485820.0, ans=0.0
2023-11-21 11:35:58,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1485886.6666666667, ans=0.125
2023-11-21 11:36:00,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1485886.6666666667, ans=0.1
2023-11-21 11:36:18,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222900
2023-11-21 11:36:32,763 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.71 vs. limit=15.0
2023-11-21 11:36:49,990 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6500, loss[loss=0.1134, simple_loss=0.1464, pruned_loss=0.03152, audio_tagging_loss=0.008699, over 15171.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09509, pruned_loss=0.01665, audio_tagging_loss=0.01005, over 3050500.65 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:37:12,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1486220.0, ans=0.0
2023-11-21 11:37:20,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.08 vs. limit=22.5
2023-11-21 11:37:23,024 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.393e+01 8.000e+01 8.623e+01 9.429e+01 1.650e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 11:37:23,175 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 222950
2023-11-21 11:37:32,474 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.85 vs. limit=12.0
2023-11-21 11:37:40,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1486420.0, ans=0.1
2023-11-21 11:37:41,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1486420.0, ans=0.125
2023-11-21 11:37:46,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1486420.0, ans=0.1
2023-11-21 11:37:48,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1486420.0, ans=0.2
2023-11-21 11:37:54,463 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6550, loss[loss=0.07057, simple_loss=0.09293, pruned_loss=0.01575, audio_tagging_loss=0.008351, over 15707.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09506, pruned_loss=0.01661, audio_tagging_loss=0.009932, over 3043046.88 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:37:57,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1486486.6666666667, ans=0.125
2023-11-21 11:38:23,030 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.23 vs. limit=15.0
2023-11-21 11:38:25,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.88 vs. limit=22.5
2023-11-21 11:38:27,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223000
2023-11-21 11:38:31,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1486620.0, ans=0.125
2023-11-21 11:38:35,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1486686.6666666667, ans=0.2
2023-11-21 11:38:36,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1486686.6666666667, ans=0.0
2023-11-21 11:38:42,453 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.89 vs. limit=15.0
2023-11-21 11:38:46,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.36 vs. limit=10.0
2023-11-21 11:38:54,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1486753.3333333333, ans=0.2
2023-11-21 11:39:00,737 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6600, loss[loss=0.09878, simple_loss=0.1379, pruned_loss=0.0219, audio_tagging_loss=0.007925, over 15403.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09604, pruned_loss=0.01666, audio_tagging_loss=0.009716, over 3048272.54 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:39:17,641 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.20 vs. limit=15.0
2023-11-21 11:39:32,154 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.360e+01 8.932e+01 9.626e+01 1.286e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 11:39:32,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223050
2023-11-21 11:39:32,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1486953.3333333333, ans=0.125
2023-11-21 11:39:45,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1487020.0, ans=22.5
2023-11-21 11:39:48,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1487020.0, ans=0.04949747468305833
2023-11-21 11:40:04,841 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6650, loss[loss=0.08355, simple_loss=0.1156, pruned_loss=0.0203, audio_tagging_loss=0.00545, over 15392.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.09653, pruned_loss=0.01665, audio_tagging_loss=0.009589, over 3052437.48 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:40:05,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1487153.3333333333, ans=0.125
2023-11-21 11:40:28,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1487220.0, ans=0.2
2023-11-21 11:40:36,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.61 vs. limit=5.0
2023-11-21 11:40:37,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223100
2023-11-21 11:40:42,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1487286.6666666667, ans=0.125
2023-11-21 11:40:42,652 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.16 vs. limit=15.0
2023-11-21 11:40:43,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1487353.3333333333, ans=0.1
2023-11-21 11:40:47,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1487353.3333333333, ans=0.0
2023-11-21 11:40:58,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=15.0
2023-11-21 11:41:09,295 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6700, loss[loss=0.07092, simple_loss=0.1009, pruned_loss=0.01395, audio_tagging_loss=0.006512, over 15464.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.09658, pruned_loss=0.01666, audio_tagging_loss=0.00948, over 3053730.28 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:41:41,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.97 vs. limit=6.0
2023-11-21 11:41:42,374 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.137e+01 8.757e+01 9.597e+01 1.374e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 11:41:42,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223150
2023-11-21 11:41:56,525 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.22 vs. limit=15.0
2023-11-21 11:42:14,857 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6750, loss[loss=0.06245, simple_loss=0.07846, pruned_loss=0.01294, audio_tagging_loss=0.01028, over 15914.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09608, pruned_loss=0.01669, audio_tagging_loss=0.009503, over 3044703.24 frames. ], batch size: 61, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:42:21,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1487820.0, ans=0.125
2023-11-21 11:42:24,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1487820.0, ans=0.125
2023-11-21 11:42:25,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1487820.0, ans=0.04949747468305833
2023-11-21 11:42:35,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.30 vs. limit=15.0
2023-11-21 11:42:36,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1487886.6666666667, ans=0.0
2023-11-21 11:42:46,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223200
2023-11-21 11:43:02,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1488020.0, ans=0.125
2023-11-21 11:43:02,736 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.79 vs. limit=15.0
2023-11-21 11:43:04,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1488020.0, ans=0.2
2023-11-21 11:43:10,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1488086.6666666667, ans=0.125
2023-11-21 11:43:19,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1488153.3333333333, ans=0.0
2023-11-21 11:43:19,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1488153.3333333333, ans=0.125
2023-11-21 11:43:20,366 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6800, loss[loss=0.0669, simple_loss=0.08726, pruned_loss=0.01293, audio_tagging_loss=0.01034, over 14894.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09498, pruned_loss=0.01643, audio_tagging_loss=0.009498, over 3045674.34 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:43:23,385 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.43 vs. limit=22.5
2023-11-21 11:43:36,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1488220.0, ans=0.125
2023-11-21 11:43:38,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1488220.0, ans=0.0
2023-11-21 11:43:42,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1488220.0, ans=0.2
2023-11-21 11:43:42,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1488220.0, ans=0.0
2023-11-21 11:43:51,772 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 8.121e+01 8.832e+01 9.588e+01 1.152e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 11:43:51,948 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223250
2023-11-21 11:44:03,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1488353.3333333333, ans=0.125
2023-11-21 11:44:08,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1488353.3333333333, ans=0.125
2023-11-21 11:44:15,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1488420.0, ans=0.125
2023-11-21 11:44:23,448 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6850, loss[loss=0.04796, simple_loss=0.06056, pruned_loss=0.007255, audio_tagging_loss=0.01042, over 13751.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09453, pruned_loss=0.01632, audio_tagging_loss=0.009444, over 3034221.56 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:44:37,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1488553.3333333333, ans=0.0
2023-11-21 11:44:40,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1488553.3333333333, ans=0.0
2023-11-21 11:44:48,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1488553.3333333333, ans=0.1
2023-11-21 11:44:56,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223300
2023-11-21 11:45:10,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1488686.6666666667, ans=0.125
2023-11-21 11:45:19,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1488753.3333333333, ans=0.2
2023-11-21 11:45:20,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1488753.3333333333, ans=0.1
2023-11-21 11:45:29,246 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6900, loss[loss=0.07174, simple_loss=0.09339, pruned_loss=0.01777, audio_tagging_loss=0.007279, over 15343.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09536, pruned_loss=0.0163, audio_tagging_loss=0.00938, over 3047503.17 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:45:29,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1488820.0, ans=0.125
2023-11-21 11:45:34,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1488820.0, ans=0.1
2023-11-21 11:45:36,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1488820.0, ans=0.1
2023-11-21 11:45:36,515 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.79 vs. limit=15.0
2023-11-21 11:45:39,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.71 vs. limit=15.0
2023-11-21 11:45:39,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1488820.0, ans=0.125
2023-11-21 11:45:51,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1488886.6666666667, ans=0.125
2023-11-21 11:45:51,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1488886.6666666667, ans=0.125
2023-11-21 11:46:00,726 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.015e+01 8.658e+01 9.630e+01 1.322e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 11:46:00,866 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223350
2023-11-21 11:46:02,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1488953.3333333333, ans=0.0
2023-11-21 11:46:07,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1489020.0, ans=0.125
2023-11-21 11:46:07,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1489020.0, ans=0.1
2023-11-21 11:46:19,849 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 11:46:33,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.02 vs. limit=10.0
2023-11-21 11:46:34,103 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 6950, loss[loss=0.06741, simple_loss=0.08904, pruned_loss=0.01295, audio_tagging_loss=0.009935, over 13685.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09571, pruned_loss=0.01638, audio_tagging_loss=0.009352, over 3037938.97 frames. ], batch size: 54, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:46:44,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1489153.3333333333, ans=0.0
2023-11-21 11:47:00,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1489286.6666666667, ans=0.0
2023-11-21 11:47:06,170 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223400
2023-11-21 11:47:06,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1489286.6666666667, ans=0.0
2023-11-21 11:47:06,756 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.79 vs. limit=15.0
2023-11-21 11:47:13,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1489353.3333333333, ans=0.95
2023-11-21 11:47:17,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1489353.3333333333, ans=0.125
2023-11-21 11:47:38,462 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7000, loss[loss=0.08101, simple_loss=0.0994, pruned_loss=0.0214, audio_tagging_loss=0.009912, over 15944.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09501, pruned_loss=0.01633, audio_tagging_loss=0.009538, over 3038978.03 frames. ], batch size: 60, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:47:59,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1489553.3333333333, ans=0.125
2023-11-21 11:48:10,556 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 7.954e+01 8.531e+01 9.123e+01 1.113e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 11:48:10,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223450
2023-11-21 11:48:13,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1489620.0, ans=0.125
2023-11-21 11:48:36,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1489753.3333333333, ans=0.125
2023-11-21 11:48:42,257 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7050, loss[loss=0.0788, simple_loss=0.1031, pruned_loss=0.01792, audio_tagging_loss=0.009339, over 14874.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09576, pruned_loss=0.01649, audio_tagging_loss=0.009549, over 3040996.19 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:48:46,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1489820.0, ans=0.1
2023-11-21 11:48:53,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1489820.0, ans=0.0
2023-11-21 11:48:54,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1489886.6666666667, ans=0.0
2023-11-21 11:48:58,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1489886.6666666667, ans=0.2
2023-11-21 11:49:00,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1489886.6666666667, ans=0.125
2023-11-21 11:49:01,002 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.84 vs. limit=6.0
2023-11-21 11:49:05,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1489886.6666666667, ans=0.125
2023-11-21 11:49:13,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223500
2023-11-21 11:49:15,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1489953.3333333333, ans=0.2
2023-11-21 11:49:46,555 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7100, loss[loss=0.08201, simple_loss=0.104, pruned_loss=0.01892, audio_tagging_loss=0.01108, over 16009.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09659, pruned_loss=0.01659, audio_tagging_loss=0.009569, over 3047078.98 frames. ], batch size: 61, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:49:49,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1490153.3333333333, ans=0.0
2023-11-21 11:49:57,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0
2023-11-21 11:50:01,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1490220.0, ans=10.0
2023-11-21 11:50:17,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223550
2023-11-21 11:50:19,150 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.140e+01 8.741e+01 9.619e+01 1.480e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 11:50:23,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1490353.3333333333, ans=0.125
2023-11-21 11:50:38,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1490420.0, ans=0.125
2023-11-21 11:50:42,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=1490420.0, ans=15.0
2023-11-21 11:50:45,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1490420.0, ans=0.2
2023-11-21 11:50:46,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.03 vs. limit=15.0
2023-11-21 11:50:49,621 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7150, loss[loss=0.07127, simple_loss=0.09163, pruned_loss=0.01464, audio_tagging_loss=0.01082, over 15717.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.0967, pruned_loss=0.01666, audio_tagging_loss=0.009617, over 3045713.69 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:50:51,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1490486.6666666667, ans=0.125
2023-11-21 11:51:10,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1490553.3333333333, ans=0.125
2023-11-21 11:51:22,533 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223600
2023-11-21 11:51:35,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.93 vs. limit=15.0
2023-11-21 11:51:54,009 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7200, loss[loss=0.0544, simple_loss=0.06166, pruned_loss=0.01068, audio_tagging_loss=0.0129, over 17034.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09677, pruned_loss=0.01671, audio_tagging_loss=0.009717, over 3051831.19 frames. ], batch size: 68, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:52:04,520 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.10 vs. limit=6.0
2023-11-21 11:52:09,381 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.65 vs. limit=15.0
2023-11-21 11:52:10,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1490886.6666666667, ans=0.125
2023-11-21 11:52:14,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.whiten.whitening_limit, batch_count=1490886.6666666667, ans=12.0
2023-11-21 11:52:26,395 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223650
2023-11-21 11:52:27,423 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.414e+01 8.953e+01 9.739e+01 1.219e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 11:52:27,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1490953.3333333333, ans=0.125
2023-11-21 11:52:58,687 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7250, loss[loss=0.07937, simple_loss=0.1101, pruned_loss=0.01539, audio_tagging_loss=0.008915, over 15620.00 frames. ], tot_loss[loss=0.07515, simple_loss=0.09711, pruned_loss=0.01679, audio_tagging_loss=0.009801, over 3053877.93 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:53:16,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1491220.0, ans=0.1
2023-11-21 11:53:30,031 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223700
2023-11-21 11:53:42,528 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.23 vs. limit=10.0
2023-11-21 11:54:02,169 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7300, loss[loss=0.06966, simple_loss=0.09153, pruned_loss=0.01597, audio_tagging_loss=0.007923, over 14825.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.0968, pruned_loss=0.01681, audio_tagging_loss=0.009713, over 3055694.99 frames. ], batch size: 54, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:54:03,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1491486.6666666667, ans=0.05
2023-11-21 11:54:20,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1491553.3333333333, ans=0.2
2023-11-21 11:54:25,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1491553.3333333333, ans=0.125
2023-11-21 11:54:31,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1491620.0, ans=0.125
2023-11-21 11:54:34,376 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223750
2023-11-21 11:54:35,395 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.195e+01 8.693e+01 9.496e+01 1.146e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 11:54:39,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1491686.6666666667, ans=0.125
2023-11-21 11:54:52,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1491753.3333333333, ans=0.2
2023-11-21 11:55:03,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1491753.3333333333, ans=0.2
2023-11-21 11:55:05,923 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7350, loss[loss=0.09148, simple_loss=0.1197, pruned_loss=0.02075, audio_tagging_loss=0.01088, over 15751.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.09633, pruned_loss=0.01674, audio_tagging_loss=0.009612, over 3050440.19 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:55:10,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1491820.0, ans=0.125
2023-11-21 11:55:27,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1491886.6666666667, ans=0.125
2023-11-21 11:55:35,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1491953.3333333333, ans=0.125
2023-11-21 11:55:38,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223800
2023-11-21 11:55:38,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1491953.3333333333, ans=22.5
2023-11-21 11:56:02,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1492086.6666666667, ans=15.0
2023-11-21 11:56:05,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.86 vs. limit=15.0
2023-11-21 11:56:11,492 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7400, loss[loss=0.1, simple_loss=0.1284, pruned_loss=0.02776, audio_tagging_loss=0.008104, over 15869.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.0956, pruned_loss=0.01654, audio_tagging_loss=0.009539, over 3042677.45 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:56:17,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.83 vs. limit=10.0
2023-11-21 11:56:23,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1492220.0, ans=0.1
2023-11-21 11:56:32,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.60 vs. limit=15.0
2023-11-21 11:56:41,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1492286.6666666667, ans=0.125
2023-11-21 11:56:43,075 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223850
2023-11-21 11:56:45,437 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 7.994e+01 8.788e+01 9.602e+01 2.162e+02, threshold=1.758e+02, percent-clipped=1.0
2023-11-21 11:56:51,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1492353.3333333333, ans=0.125
2023-11-21 11:56:51,430 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:57:06,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1492420.0, ans=0.125
2023-11-21 11:57:15,932 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7450, loss[loss=0.05734, simple_loss=0.06529, pruned_loss=0.01206, audio_tagging_loss=0.01263, over 15890.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09577, pruned_loss=0.01641, audio_tagging_loss=0.009601, over 3040505.03 frames. ], batch size: 61, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:57:38,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1492553.3333333333, ans=0.2
2023-11-21 11:57:47,913 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.35 vs. limit=15.0
2023-11-21 11:57:48,278 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223900
2023-11-21 11:58:18,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1492820.0, ans=0.05
2023-11-21 11:58:19,356 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7500, loss[loss=0.06944, simple_loss=0.08905, pruned_loss=0.01456, audio_tagging_loss=0.01036, over 14202.00 frames. ], tot_loss[loss=0.07351, simple_loss=0.09509, pruned_loss=0.01634, audio_tagging_loss=0.00962, over 3041808.86 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:58:19,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1492820.0, ans=0.125
2023-11-21 11:58:25,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1492820.0, ans=0.125
2023-11-21 11:58:34,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1492886.6666666667, ans=0.0
2023-11-21 11:58:43,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1492886.6666666667, ans=0.1
2023-11-21 11:58:44,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1492953.3333333333, ans=0.2
2023-11-21 11:58:51,659 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 223950
2023-11-21 11:58:54,580 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.233e+01 8.004e+01 8.893e+01 9.495e+01 1.256e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 11:59:14,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1493086.6666666667, ans=0.125
2023-11-21 11:59:24,162 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7550, loss[loss=0.06774, simple_loss=0.08572, pruned_loss=0.01678, audio_tagging_loss=0.008106, over 14706.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09459, pruned_loss=0.01646, audio_tagging_loss=0.009734, over 3036626.49 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:59:24,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1493153.3333333333, ans=0.125
2023-11-21 11:59:50,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1493286.6666666667, ans=0.0
2023-11-21 11:59:54,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.18 vs. limit=15.0
2023-11-21 11:59:55,173 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224000
2023-11-21 12:00:11,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1493353.3333333333, ans=0.0
2023-11-21 12:00:18,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1493420.0, ans=0.125
2023-11-21 12:00:29,978 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7600, loss[loss=0.05886, simple_loss=0.07352, pruned_loss=0.01264, audio_tagging_loss=0.009459, over 15479.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09551, pruned_loss=0.01678, audio_tagging_loss=0.009657, over 3039092.45 frames. ], batch size: 61, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 12:00:35,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1493486.6666666667, ans=0.0
2023-11-21 12:00:46,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1493553.3333333333, ans=0.0
2023-11-21 12:01:02,911 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224050
2023-11-21 12:01:05,237 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.103e+01 8.665e+01 9.260e+01 1.158e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 12:01:11,615 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:01:28,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1493753.3333333333, ans=0.1
2023-11-21 12:01:33,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1493820.0, ans=0.0
2023-11-21 12:01:34,033 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7650, loss[loss=0.07629, simple_loss=0.1013, pruned_loss=0.01734, audio_tagging_loss=0.008304, over 15626.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09565, pruned_loss=0.0168, audio_tagging_loss=0.009683, over 3036287.63 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 12:01:34,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1493820.0, ans=0.95
2023-11-21 12:01:40,010 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=22.27 vs. limit=22.5
2023-11-21 12:01:48,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1493886.6666666667, ans=0.125
2023-11-21 12:01:51,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1493886.6666666667, ans=0.035
2023-11-21 12:02:06,324 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224100
2023-11-21 12:02:19,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1494020.0, ans=0.125
2023-11-21 12:02:38,308 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7700, loss[loss=0.08314, simple_loss=0.1122, pruned_loss=0.01905, audio_tagging_loss=0.007978, over 15375.00 frames. ], tot_loss[loss=0.07423, simple_loss=0.09566, pruned_loss=0.01677, audio_tagging_loss=0.009631, over 3041700.53 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 12:03:09,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224150
2023-11-21 12:03:12,166 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.308e+01 8.854e+01 9.589e+01 1.258e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 12:03:19,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1494353.3333333333, ans=0.125
2023-11-21 12:03:23,236 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:03:38,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1494420.0, ans=0.125
2023-11-21 12:03:41,680 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7750, loss[loss=0.06427, simple_loss=0.0817, pruned_loss=0.01413, audio_tagging_loss=0.009284, over 14488.00 frames. ], tot_loss[loss=0.07519, simple_loss=0.09699, pruned_loss=0.01718, audio_tagging_loss=0.009515, over 3036979.77 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:04:06,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.87 vs. limit=15.0
2023-11-21 12:04:09,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1494620.0, ans=0.125
2023-11-21 12:04:10,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1494620.0, ans=0.125
2023-11-21 12:04:14,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224200
2023-11-21 12:04:34,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1494753.3333333333, ans=0.035
2023-11-21 12:04:38,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.25 vs. limit=12.0
2023-11-21 12:04:40,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1494753.3333333333, ans=0.125
2023-11-21 12:04:45,933 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7800, loss[loss=0.07298, simple_loss=0.09715, pruned_loss=0.01666, audio_tagging_loss=0.007749, over 14950.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09746, pruned_loss=0.01726, audio_tagging_loss=0.009453, over 3038297.19 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:04:58,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1494886.6666666667, ans=0.2
2023-11-21 12:05:03,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.33 vs. limit=15.0
2023-11-21 12:05:18,296 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224250
2023-11-21 12:05:21,778 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.303e+01 8.766e+01 9.637e+01 1.238e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 12:05:22,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1494953.3333333333, ans=0.125
2023-11-21 12:05:33,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1495020.0, ans=0.125
2023-11-21 12:05:41,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1495086.6666666667, ans=0.0
2023-11-21 12:05:42,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.19 vs. limit=22.5
2023-11-21 12:05:50,430 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7850, loss[loss=0.08005, simple_loss=0.1068, pruned_loss=0.01723, audio_tagging_loss=0.009406, over 15235.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09802, pruned_loss=0.01725, audio_tagging_loss=0.009475, over 3042847.72 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:06:16,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1495286.6666666667, ans=0.125
2023-11-21 12:06:16,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1495286.6666666667, ans=0.2
2023-11-21 12:06:21,434 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224300
2023-11-21 12:06:21,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1495286.6666666667, ans=0.1
2023-11-21 12:06:24,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1495286.6666666667, ans=0.125
2023-11-21 12:06:31,395 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.69 vs. limit=6.0
2023-11-21 12:06:43,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.45 vs. limit=10.0
2023-11-21 12:06:45,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1495420.0, ans=0.125
2023-11-21 12:06:51,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1495420.0, ans=0.1
2023-11-21 12:06:53,990 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7900, loss[loss=0.08241, simple_loss=0.1083, pruned_loss=0.01905, audio_tagging_loss=0.009193, over 15055.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09671, pruned_loss=0.01683, audio_tagging_loss=0.009606, over 3047808.50 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:07:18,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1495620.0, ans=0.0
2023-11-21 12:07:21,632 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.77 vs. limit=15.0
2023-11-21 12:07:26,399 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224350
2023-11-21 12:07:29,889 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.237e+01 8.708e+01 9.758e+01 1.228e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 12:07:32,438 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.19 vs. limit=22.5
2023-11-21 12:07:35,948 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.29 vs. limit=15.0
2023-11-21 12:07:40,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.33 vs. limit=15.0
2023-11-21 12:07:42,036 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.14 vs. limit=22.5
2023-11-21 12:07:56,993 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 7950, loss[loss=0.1026, simple_loss=0.1283, pruned_loss=0.03187, audio_tagging_loss=0.006602, over 15130.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.09725, pruned_loss=0.01705, audio_tagging_loss=0.009711, over 3039398.39 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:08:14,112 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:08:17,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1495886.6666666667, ans=0.125
2023-11-21 12:08:29,278 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224400
2023-11-21 12:08:32,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.92 vs. limit=22.5
2023-11-21 12:08:50,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1496086.6666666667, ans=0.05
2023-11-21 12:09:01,137 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8000, loss[loss=0.08442, simple_loss=0.117, pruned_loss=0.01831, audio_tagging_loss=0.007617, over 16738.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09596, pruned_loss=0.01664, audio_tagging_loss=0.009941, over 3041902.03 frames. ], batch size: 59, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:09:03,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1496153.3333333333, ans=0.1
2023-11-21 12:09:05,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1496153.3333333333, ans=0.125
2023-11-21 12:09:25,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1496286.6666666667, ans=0.125
2023-11-21 12:09:31,535 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=12.0
2023-11-21 12:09:32,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224450
2023-11-21 12:09:35,522 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.187e+01 8.210e+01 8.871e+01 9.654e+01 1.306e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 12:09:41,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.96 vs. limit=15.0
2023-11-21 12:09:53,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.39 vs. limit=22.5
2023-11-21 12:09:54,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1496420.0, ans=0.2
2023-11-21 12:09:56,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1496420.0, ans=0.2
2023-11-21 12:10:04,407 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8050, loss[loss=0.07018, simple_loss=0.09313, pruned_loss=0.01464, audio_tagging_loss=0.008979, over 14609.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09604, pruned_loss=0.0166, audio_tagging_loss=0.01, over 3049355.33 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:10:09,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1496486.6666666667, ans=0.125
2023-11-21 12:10:15,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.77 vs. limit=15.0
2023-11-21 12:10:31,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1496620.0, ans=10.0
2023-11-21 12:10:35,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224500
2023-11-21 12:10:55,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.40 vs. limit=15.0
2023-11-21 12:10:55,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.33 vs. limit=12.0
2023-11-21 12:11:07,048 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8100, loss[loss=0.07899, simple_loss=0.09889, pruned_loss=0.01761, audio_tagging_loss=0.01193, over 14781.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09654, pruned_loss=0.01677, audio_tagging_loss=0.009832, over 3044706.74 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:11:17,408 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.51 vs. limit=12.0
2023-11-21 12:11:37,783 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.22 vs. limit=15.0
2023-11-21 12:11:39,674 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224550
2023-11-21 12:11:43,198 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.553e+01 8.024e+01 8.529e+01 9.304e+01 1.158e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 12:11:44,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1497020.0, ans=0.0
2023-11-21 12:11:50,034 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.08 vs. limit=15.0
2023-11-21 12:12:10,664 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8150, loss[loss=0.08553, simple_loss=0.1121, pruned_loss=0.02181, audio_tagging_loss=0.007653, over 15906.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09676, pruned_loss=0.01663, audio_tagging_loss=0.009628, over 3037564.54 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:12:28,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.22 vs. limit=15.0
2023-11-21 12:12:42,560 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224600
2023-11-21 12:13:15,671 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8200, loss[loss=0.08807, simple_loss=0.1087, pruned_loss=0.02275, audio_tagging_loss=0.01098, over 15179.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09689, pruned_loss=0.01676, audio_tagging_loss=0.009521, over 3038477.77 frames. ], batch size: 53, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:13:15,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1497486.6666666667, ans=0.1
2023-11-21 12:13:16,948 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:13:34,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1497553.3333333333, ans=0.1
2023-11-21 12:13:37,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1497553.3333333333, ans=0.0
2023-11-21 12:13:41,939 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.68 vs. limit=22.5
2023-11-21 12:13:46,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224650
2023-11-21 12:13:52,063 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.039e+01 8.805e+01 9.521e+01 1.121e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 12:14:14,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1497753.3333333333, ans=0.0
2023-11-21 12:14:18,856 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8250, loss[loss=0.08257, simple_loss=0.1126, pruned_loss=0.01855, audio_tagging_loss=0.007733, over 14406.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09676, pruned_loss=0.01667, audio_tagging_loss=0.009585, over 3039070.34 frames. ], batch size: 53, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:14:31,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1497886.6666666667, ans=0.0
2023-11-21 12:14:33,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1497886.6666666667, ans=0.025
2023-11-21 12:14:39,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1497886.6666666667, ans=0.2
2023-11-21 12:14:39,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1497886.6666666667, ans=0.07
2023-11-21 12:14:46,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1497953.3333333333, ans=0.125
2023-11-21 12:14:47,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1497953.3333333333, ans=0.125
2023-11-21 12:14:51,173 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224700
2023-11-21 12:14:51,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1497953.3333333333, ans=0.125
2023-11-21 12:15:20,053 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.14 vs. limit=6.0
2023-11-21 12:15:22,112 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8300, loss[loss=0.07015, simple_loss=0.09084, pruned_loss=0.01096, audio_tagging_loss=0.01377, over 16338.00 frames. ], tot_loss[loss=0.0749, simple_loss=0.09716, pruned_loss=0.01682, audio_tagging_loss=0.009498, over 3048221.40 frames. ], batch size: 63, lr: 3.59e-03, grad_scale: 8.0
2023-11-21 12:15:30,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1498153.3333333333, ans=0.125
2023-11-21 12:15:46,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.59 vs. limit=22.5
2023-11-21 12:15:54,097 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224750
2023-11-21 12:15:59,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1498353.3333333333, ans=0.125
2023-11-21 12:16:00,001 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.024e+01 7.847e+01 8.529e+01 9.488e+01 1.210e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 12:16:27,144 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8350, loss[loss=0.07239, simple_loss=0.09703, pruned_loss=0.016, audio_tagging_loss=0.007876, over 15650.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.09761, pruned_loss=0.01685, audio_tagging_loss=0.009417, over 3050827.85 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 8.0
2023-11-21 12:16:41,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1498553.3333333333, ans=0.125
2023-11-21 12:16:44,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1498553.3333333333, ans=0.0
2023-11-21 12:16:50,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.98 vs. limit=12.0
2023-11-21 12:16:58,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224800
2023-11-21 12:17:10,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1498686.6666666667, ans=0.125
2023-11-21 12:17:13,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1498686.6666666667, ans=0.125
2023-11-21 12:17:23,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1498753.3333333333, ans=0.125
2023-11-21 12:17:29,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1498820.0, ans=0.0
2023-11-21 12:17:30,439 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8400, loss[loss=0.08232, simple_loss=0.1082, pruned_loss=0.01765, audio_tagging_loss=0.0106, over 14773.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09723, pruned_loss=0.01693, audio_tagging_loss=0.009393, over 3055332.95 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:17:43,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.96 vs. limit=15.0
2023-11-21 12:17:54,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1498886.6666666667, ans=0.125
2023-11-21 12:17:59,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1498953.3333333333, ans=0.125
2023-11-21 12:18:03,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224850
2023-11-21 12:18:09,638 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.252e+01 9.012e+01 9.570e+01 1.298e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-21 12:18:24,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1499086.6666666667, ans=0.125
2023-11-21 12:18:27,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1499086.6666666667, ans=0.125
2023-11-21 12:18:33,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1499153.3333333333, ans=0.035
2023-11-21 12:18:34,199 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8450, loss[loss=0.09704, simple_loss=0.1297, pruned_loss=0.02445, audio_tagging_loss=0.007773, over 15518.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09623, pruned_loss=0.0168, audio_tagging_loss=0.009567, over 3057650.98 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:18:44,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1499153.3333333333, ans=0.125
2023-11-21 12:18:44,691 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.78 vs. limit=15.0
2023-11-21 12:18:56,121 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.83 vs. limit=15.0
2023-11-21 12:19:03,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1499286.6666666667, ans=0.1
2023-11-21 12:19:06,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224900
2023-11-21 12:19:39,594 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8500, loss[loss=0.09267, simple_loss=0.119, pruned_loss=0.02221, audio_tagging_loss=0.01098, over 15974.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.09631, pruned_loss=0.01687, audio_tagging_loss=0.009507, over 3053276.90 frames. ], batch size: 59, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:20:01,688 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.67 vs. limit=22.5
2023-11-21 12:20:03,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1499620.0, ans=0.125
2023-11-21 12:20:03,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1499620.0, ans=0.95
2023-11-21 12:20:11,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 224950
2023-11-21 12:20:17,538 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.782e+01 8.392e+01 8.931e+01 9.765e+01 1.184e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 12:20:21,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1499686.6666666667, ans=0.125
2023-11-21 12:20:26,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0
2023-11-21 12:20:29,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1499686.6666666667, ans=0.125
2023-11-21 12:20:32,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1499753.3333333333, ans=0.1
2023-11-21 12:20:32,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1499753.3333333333, ans=0.1
2023-11-21 12:20:44,389 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8550, loss[loss=0.08928, simple_loss=0.1148, pruned_loss=0.0247, audio_tagging_loss=0.007194, over 15151.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09698, pruned_loss=0.01708, audio_tagging_loss=0.009491, over 3057404.74 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:20:50,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1499820.0, ans=0.1
2023-11-21 12:20:52,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1499820.0, ans=0.2
2023-11-21 12:20:57,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1499886.6666666667, ans=0.04949747468305833
2023-11-21 12:21:16,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225000
2023-11-21 12:21:23,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1500020.0, ans=0.125
2023-11-21 12:21:26,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1500020.0, ans=0.0
2023-11-21 12:21:27,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1500020.0, ans=0.0
2023-11-21 12:21:40,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1500086.6666666667, ans=0.125
2023-11-21 12:21:48,089 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8600, loss[loss=0.06561, simple_loss=0.08149, pruned_loss=0.01226, audio_tagging_loss=0.0126, over 14965.00 frames. ], tot_loss[loss=0.07547, simple_loss=0.09761, pruned_loss=0.01714, audio_tagging_loss=0.009525, over 3056868.56 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:22:21,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225050
2023-11-21 12:22:27,331 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.106e+01 8.713e+01 9.527e+01 1.348e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 12:22:39,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1500420.0, ans=0.125
2023-11-21 12:22:45,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1500420.0, ans=0.125
2023-11-21 12:22:52,982 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8650, loss[loss=0.08082, simple_loss=0.1079, pruned_loss=0.01641, audio_tagging_loss=0.01044, over 15045.00 frames. ], tot_loss[loss=0.07526, simple_loss=0.09757, pruned_loss=0.01694, audio_tagging_loss=0.009542, over 3062214.80 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:23:06,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1500553.3333333333, ans=0.0
2023-11-21 12:23:25,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225100
2023-11-21 12:23:31,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1500686.6666666667, ans=0.1
2023-11-21 12:23:42,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1500686.6666666667, ans=0.1
2023-11-21 12:23:57,071 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8700, loss[loss=0.05358, simple_loss=0.07039, pruned_loss=0.009903, audio_tagging_loss=0.008482, over 15227.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09595, pruned_loss=0.01643, audio_tagging_loss=0.009744, over 3062277.04 frames. ], batch size: 60, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:24:02,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1500820.0, ans=0.125
2023-11-21 12:24:11,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1500886.6666666667, ans=0.125
2023-11-21 12:24:27,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1500953.3333333333, ans=0.0
2023-11-21 12:24:29,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225150
2023-11-21 12:24:35,996 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.442e+01 8.072e+01 8.843e+01 9.223e+01 1.204e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 12:24:51,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1501086.6666666667, ans=0.125
2023-11-21 12:24:55,342 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.81 vs. limit=15.0
2023-11-21 12:24:58,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1501086.6666666667, ans=10.0
2023-11-21 12:25:00,576 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8750, loss[loss=0.06416, simple_loss=0.07994, pruned_loss=0.01532, audio_tagging_loss=0.008862, over 14568.00 frames. ], tot_loss[loss=0.07468, simple_loss=0.09665, pruned_loss=0.0166, audio_tagging_loss=0.009747, over 3063172.48 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:25:00,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1501153.3333333333, ans=0.125
2023-11-21 12:25:10,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1501153.3333333333, ans=0.0
2023-11-21 12:25:32,778 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225200
2023-11-21 12:25:35,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1501286.6666666667, ans=0.125
2023-11-21 12:25:37,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1501286.6666666667, ans=0.125
2023-11-21 12:25:57,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1501420.0, ans=0.125
2023-11-21 12:26:05,340 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8800, loss[loss=0.06361, simple_loss=0.08155, pruned_loss=0.01329, audio_tagging_loss=0.009538, over 14805.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09627, pruned_loss=0.01657, audio_tagging_loss=0.009877, over 3056422.83 frames. ], batch size: 54, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:26:07,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1501486.6666666667, ans=0.125
2023-11-21 12:26:31,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1501620.0, ans=0.125
2023-11-21 12:26:37,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225250
2023-11-21 12:26:43,239 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.062e+01 8.940e+01 9.452e+01 1.337e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 12:26:44,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.09 vs. limit=15.0
2023-11-21 12:27:09,857 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8850, loss[loss=0.0687, simple_loss=0.08995, pruned_loss=0.01324, audio_tagging_loss=0.01048, over 15192.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09714, pruned_loss=0.0167, audio_tagging_loss=0.009847, over 3052426.86 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:27:16,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.45 vs. limit=15.0
2023-11-21 12:27:21,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1501886.6666666667, ans=0.125
2023-11-21 12:27:22,951 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:27:25,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1501886.6666666667, ans=0.125
2023-11-21 12:27:29,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1501886.6666666667, ans=0.09899494936611666
2023-11-21 12:27:42,730 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225300
2023-11-21 12:28:15,031 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8900, loss[loss=0.08123, simple_loss=0.103, pruned_loss=0.01909, audio_tagging_loss=0.01065, over 14563.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09723, pruned_loss=0.01674, audio_tagging_loss=0.009755, over 3054679.41 frames. ], batch size: 53, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:28:41,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1502286.6666666667, ans=0.125
2023-11-21 12:28:47,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225350
2023-11-21 12:28:54,894 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.032e+01 8.160e+01 8.777e+01 9.579e+01 1.339e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 12:29:01,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1502353.3333333333, ans=0.0
2023-11-21 12:29:20,409 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 8950, loss[loss=0.06352, simple_loss=0.0793, pruned_loss=0.01488, audio_tagging_loss=0.008994, over 14208.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09754, pruned_loss=0.01685, audio_tagging_loss=0.009603, over 3056938.14 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:29:36,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1502553.3333333333, ans=0.2
2023-11-21 12:29:39,054 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.60 vs. limit=15.0
2023-11-21 12:29:42,757 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.53 vs. limit=15.0
2023-11-21 12:29:44,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1502620.0, ans=0.125
2023-11-21 12:29:51,923 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225400
2023-11-21 12:29:53,279 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:29:53,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1502620.0, ans=0.125
2023-11-21 12:29:58,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1502686.6666666667, ans=0.0
2023-11-21 12:30:03,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1502686.6666666667, ans=0.125
2023-11-21 12:30:11,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1502753.3333333333, ans=0.1
2023-11-21 12:30:11,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1502753.3333333333, ans=0.125
2023-11-21 12:30:25,205 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9000, loss[loss=0.07201, simple_loss=0.09306, pruned_loss=0.01662, audio_tagging_loss=0.008851, over 15478.00 frames. ], tot_loss[loss=0.07527, simple_loss=0.0979, pruned_loss=0.01685, audio_tagging_loss=0.009466, over 3058709.48 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:30:25,205 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 12:31:06,245 INFO [train_asr.py:1253] (2/4) Epoch 19, validation: loss=0.06044, simple_loss=0.05233, pruned_loss=0.005297, audio_tagging_loss=0.02898, over 4681554.00 frames. 
2023-11-21 12:31:06,246 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 12:31:07,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1502820.0, ans=0.05
2023-11-21 12:31:24,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1502886.6666666667, ans=0.0
2023-11-21 12:31:28,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.40 vs. limit=15.0
2023-11-21 12:31:38,593 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225450
2023-11-21 12:31:42,843 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.89 vs. limit=15.0
2023-11-21 12:31:45,865 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.087e+01 8.929e+01 9.690e+01 1.260e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 12:32:05,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1503086.6666666667, ans=0.2
2023-11-21 12:32:10,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1503153.3333333333, ans=0.2
2023-11-21 12:32:11,479 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9050, loss[loss=0.06551, simple_loss=0.08055, pruned_loss=0.01546, audio_tagging_loss=0.009778, over 16651.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09762, pruned_loss=0.0169, audio_tagging_loss=0.009389, over 3059375.45 frames. ], batch size: 63, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:32:34,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1503220.0, ans=0.125
2023-11-21 12:32:42,875 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225500
2023-11-21 12:32:44,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1503286.6666666667, ans=0.1
2023-11-21 12:33:02,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1503420.0, ans=0.125
2023-11-21 12:33:13,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1503420.0, ans=0.09899494936611666
2023-11-21 12:33:15,758 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9100, loss[loss=0.07587, simple_loss=0.09814, pruned_loss=0.01876, audio_tagging_loss=0.008038, over 15694.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09714, pruned_loss=0.01679, audio_tagging_loss=0.009341, over 3056804.23 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:33:22,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1503486.6666666667, ans=0.2
2023-11-21 12:33:22,776 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.56 vs. limit=22.5
2023-11-21 12:33:29,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1503553.3333333333, ans=0.0
2023-11-21 12:33:30,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1503553.3333333333, ans=10.0
2023-11-21 12:33:40,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1503620.0, ans=0.1
2023-11-21 12:33:43,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1503620.0, ans=0.0
2023-11-21 12:33:48,306 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225550
2023-11-21 12:33:49,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1503620.0, ans=0.2
2023-11-21 12:33:55,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1503686.6666666667, ans=0.125
2023-11-21 12:33:55,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1503686.6666666667, ans=10.0
2023-11-21 12:33:56,025 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.533e+01 8.001e+01 8.590e+01 9.455e+01 1.329e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 12:33:56,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1503686.6666666667, ans=0.125
2023-11-21 12:33:57,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.26 vs. limit=15.0
2023-11-21 12:34:05,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1503686.6666666667, ans=0.1
2023-11-21 12:34:07,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1503753.3333333333, ans=0.0
2023-11-21 12:34:19,563 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9150, loss[loss=0.05393, simple_loss=0.06554, pruned_loss=0.01008, audio_tagging_loss=0.01108, over 15749.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09717, pruned_loss=0.01671, audio_tagging_loss=0.009376, over 3052954.76 frames. ], batch size: 61, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:34:52,118 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225600
2023-11-21 12:34:57,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1504020.0, ans=0.125
2023-11-21 12:34:58,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1504020.0, ans=0.0
2023-11-21 12:35:11,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1504086.6666666667, ans=0.125
2023-11-21 12:35:11,636 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.55 vs. limit=15.0
2023-11-21 12:35:21,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1504086.6666666667, ans=0.125
2023-11-21 12:35:24,754 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9200, loss[loss=0.07902, simple_loss=0.1042, pruned_loss=0.0192, audio_tagging_loss=0.007734, over 14591.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09633, pruned_loss=0.01665, audio_tagging_loss=0.009441, over 3053671.01 frames. ], batch size: 54, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:35:50,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1504286.6666666667, ans=10.0
2023-11-21 12:35:52,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1504286.6666666667, ans=0.125
2023-11-21 12:35:53,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1504286.6666666667, ans=0.0
2023-11-21 12:35:56,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225650
2023-11-21 12:35:57,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1504286.6666666667, ans=0.0
2023-11-21 12:36:03,554 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.007e+01 8.520e+01 9.210e+01 1.380e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 12:36:13,025 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:36:15,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1504420.0, ans=0.1
2023-11-21 12:36:29,514 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9250, loss[loss=0.08368, simple_loss=0.1061, pruned_loss=0.02132, audio_tagging_loss=0.009308, over 14580.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.0962, pruned_loss=0.01662, audio_tagging_loss=0.009455, over 3050223.62 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:36:33,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1504486.6666666667, ans=0.125
2023-11-21 12:36:50,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1504553.3333333333, ans=0.2
2023-11-21 12:36:54,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1504620.0, ans=0.125
2023-11-21 12:36:55,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1504620.0, ans=0.125
2023-11-21 12:36:55,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1504620.0, ans=0.04949747468305833
2023-11-21 12:37:01,635 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225700
2023-11-21 12:37:01,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1504620.0, ans=0.0
2023-11-21 12:37:16,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1504686.6666666667, ans=0.125
2023-11-21 12:37:16,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1504686.6666666667, ans=0.1
2023-11-21 12:37:22,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1504753.3333333333, ans=0.125
2023-11-21 12:37:22,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.92 vs. limit=10.0
2023-11-21 12:37:22,962 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.12 vs. limit=22.5
2023-11-21 12:37:33,055 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9300, loss[loss=0.063, simple_loss=0.07571, pruned_loss=0.01527, audio_tagging_loss=0.009878, over 14992.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09588, pruned_loss=0.01663, audio_tagging_loss=0.009513, over 3046402.94 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:37:52,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1504886.6666666667, ans=0.0
2023-11-21 12:38:05,506 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225750
2023-11-21 12:38:12,768 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 7.914e+01 8.650e+01 9.354e+01 1.480e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 12:38:13,084 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:38:14,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1505020.0, ans=0.125
2023-11-21 12:38:23,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1505086.6666666667, ans=0.0
2023-11-21 12:38:29,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1505086.6666666667, ans=0.1
2023-11-21 12:38:37,003 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9350, loss[loss=0.05105, simple_loss=0.05951, pruned_loss=0.007572, audio_tagging_loss=0.01372, over 15749.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09675, pruned_loss=0.01698, audio_tagging_loss=0.009496, over 3053516.41 frames. ], batch size: 62, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:38:38,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.98 vs. limit=15.0
2023-11-21 12:38:45,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1505153.3333333333, ans=0.0
2023-11-21 12:38:53,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1505220.0, ans=0.125
2023-11-21 12:39:00,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1505220.0, ans=0.125
2023-11-21 12:39:03,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1505286.6666666667, ans=0.1
2023-11-21 12:39:07,180 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.66 vs. limit=15.0
2023-11-21 12:39:09,037 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225800
2023-11-21 12:39:15,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.96 vs. limit=15.0
2023-11-21 12:39:40,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1505420.0, ans=0.2
2023-11-21 12:39:42,122 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9400, loss[loss=0.07647, simple_loss=0.1013, pruned_loss=0.0159, audio_tagging_loss=0.009922, over 15489.00 frames. ], tot_loss[loss=0.07461, simple_loss=0.09609, pruned_loss=0.01684, audio_tagging_loss=0.009727, over 3045928.88 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:40:10,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1505620.0, ans=0.1
2023-11-21 12:40:13,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225850
2023-11-21 12:40:14,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1505620.0, ans=0.2
2023-11-21 12:40:22,997 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.417e+01 9.121e+01 9.724e+01 1.188e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-21 12:40:27,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1505686.6666666667, ans=0.0
2023-11-21 12:40:28,652 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0
2023-11-21 12:40:41,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1505753.3333333333, ans=0.125
2023-11-21 12:40:42,781 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:40:45,233 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9450, loss[loss=0.07134, simple_loss=0.09535, pruned_loss=0.01443, audio_tagging_loss=0.009232, over 14667.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09712, pruned_loss=0.01704, audio_tagging_loss=0.009757, over 3050785.79 frames. ], batch size: 53, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:40:45,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1505820.0, ans=0.125
2023-11-21 12:41:06,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.16 vs. limit=15.0
2023-11-21 12:41:17,603 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225900
2023-11-21 12:41:31,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1506020.0, ans=0.125
2023-11-21 12:41:47,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1506153.3333333333, ans=0.0
2023-11-21 12:41:48,204 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9500, loss[loss=0.08903, simple_loss=0.1178, pruned_loss=0.02093, audio_tagging_loss=0.009172, over 15006.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.09727, pruned_loss=0.01682, audio_tagging_loss=0.009686, over 3059071.51 frames. ], batch size: 54, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:41:55,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1506153.3333333333, ans=0.1
2023-11-21 12:42:18,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1506286.6666666667, ans=0.2
2023-11-21 12:42:20,947 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 225950
2023-11-21 12:42:29,398 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.362e+01 9.191e+01 9.965e+01 1.276e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-21 12:42:37,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1506353.3333333333, ans=0.0
2023-11-21 12:42:43,759 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:42:44,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.58 vs. limit=15.0
2023-11-21 12:42:53,267 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9550, loss[loss=0.07449, simple_loss=0.1035, pruned_loss=0.01648, audio_tagging_loss=0.006239, over 14035.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09736, pruned_loss=0.01683, audio_tagging_loss=0.00969, over 3050260.35 frames. ], batch size: 53, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:42:57,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1506486.6666666667, ans=0.0
2023-11-21 12:43:03,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1506486.6666666667, ans=0.035
2023-11-21 12:43:09,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1506553.3333333333, ans=0.125
2023-11-21 12:43:18,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1506620.0, ans=0.1
2023-11-21 12:43:24,238 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226000
2023-11-21 12:43:24,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1506620.0, ans=0.125
2023-11-21 12:43:31,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1506686.6666666667, ans=0.05
2023-11-21 12:43:34,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1506686.6666666667, ans=0.09899494936611666
2023-11-21 12:43:44,522 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.14 vs. limit=10.0
2023-11-21 12:43:52,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1506753.3333333333, ans=0.1
2023-11-21 12:43:57,187 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9600, loss[loss=0.06952, simple_loss=0.08834, pruned_loss=0.01426, audio_tagging_loss=0.01108, over 15161.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09752, pruned_loss=0.01702, audio_tagging_loss=0.009813, over 3050899.58 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:43:59,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1506820.0, ans=0.2
2023-11-21 12:44:29,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226050
2023-11-21 12:44:38,332 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 7.983e+01 8.743e+01 9.613e+01 1.291e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 12:44:41,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0
2023-11-21 12:45:00,268 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9650, loss[loss=0.07207, simple_loss=0.08496, pruned_loss=0.01644, audio_tagging_loss=0.01315, over 16094.00 frames. ], tot_loss[loss=0.07551, simple_loss=0.09774, pruned_loss=0.01698, audio_tagging_loss=0.00966, over 3043922.80 frames. ], batch size: 62, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:45:04,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1507153.3333333333, ans=0.125
2023-11-21 12:45:10,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1507153.3333333333, ans=0.0
2023-11-21 12:45:22,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1507220.0, ans=0.125
2023-11-21 12:45:33,277 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226100
2023-11-21 12:45:51,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1507420.0, ans=0.1
2023-11-21 12:46:02,018 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.77 vs. limit=15.0
2023-11-21 12:46:04,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.14 vs. limit=15.0
2023-11-21 12:46:05,009 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9700, loss[loss=0.06472, simple_loss=0.08231, pruned_loss=0.01471, audio_tagging_loss=0.008852, over 14872.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09747, pruned_loss=0.01693, audio_tagging_loss=0.009447, over 3047031.51 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:46:12,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.95 vs. limit=6.0
2023-11-21 12:46:14,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1507486.6666666667, ans=0.125
2023-11-21 12:46:17,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1507553.3333333333, ans=0.0
2023-11-21 12:46:22,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1507553.3333333333, ans=0.125
2023-11-21 12:46:24,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1507553.3333333333, ans=0.025
2023-11-21 12:46:26,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1507553.3333333333, ans=0.125
2023-11-21 12:46:36,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226150
2023-11-21 12:46:46,070 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 7.992e+01 8.607e+01 9.446e+01 1.380e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 12:46:55,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1507753.3333333333, ans=0.125
2023-11-21 12:47:04,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1507753.3333333333, ans=0.1
2023-11-21 12:47:06,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1507753.3333333333, ans=0.125
2023-11-21 12:47:08,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1507820.0, ans=0.2
2023-11-21 12:47:09,288 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9750, loss[loss=0.06434, simple_loss=0.08436, pruned_loss=0.01403, audio_tagging_loss=0.008135, over 14247.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.09755, pruned_loss=0.01691, audio_tagging_loss=0.009362, over 3048670.27 frames. ], batch size: 54, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:47:41,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226200
2023-11-21 12:47:42,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1507953.3333333333, ans=0.125
2023-11-21 12:48:08,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1508086.6666666667, ans=0.1
2023-11-21 12:48:13,269 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9800, loss[loss=0.07553, simple_loss=0.09434, pruned_loss=0.0188, audio_tagging_loss=0.00956, over 15178.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09716, pruned_loss=0.01669, audio_tagging_loss=0.009369, over 3041429.14 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:48:13,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1508153.3333333333, ans=0.125
2023-11-21 12:48:14,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1508153.3333333333, ans=0.0
2023-11-21 12:48:22,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1508153.3333333333, ans=0.2
2023-11-21 12:48:25,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1508220.0, ans=0.125
2023-11-21 12:48:34,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1508220.0, ans=15.0
2023-11-21 12:48:37,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1508286.6666666667, ans=0.0
2023-11-21 12:48:45,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226250
2023-11-21 12:48:48,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.81 vs. limit=15.0
2023-11-21 12:48:53,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1508353.3333333333, ans=0.1
2023-11-21 12:48:53,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1508353.3333333333, ans=0.0
2023-11-21 12:48:53,958 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.494e+01 7.972e+01 8.418e+01 9.313e+01 1.119e+02, threshold=1.684e+02, percent-clipped=0.0
2023-11-21 12:48:55,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1508353.3333333333, ans=0.2
2023-11-21 12:49:04,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1508420.0, ans=0.1
2023-11-21 12:49:09,507 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:49:10,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.80 vs. limit=15.0
2023-11-21 12:49:17,414 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9850, loss[loss=0.06578, simple_loss=0.08427, pruned_loss=0.01503, audio_tagging_loss=0.008609, over 15443.00 frames. ], tot_loss[loss=0.07478, simple_loss=0.09742, pruned_loss=0.01685, audio_tagging_loss=0.009223, over 3035906.58 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:49:29,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1508553.3333333333, ans=0.1
2023-11-21 12:49:37,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1508553.3333333333, ans=0.125
2023-11-21 12:49:39,000 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:49:44,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1508620.0, ans=0.125
2023-11-21 12:49:48,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226300
2023-11-21 12:49:49,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1508620.0, ans=0.0
2023-11-21 12:49:57,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1508686.6666666667, ans=0.1
2023-11-21 12:50:09,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1508753.3333333333, ans=0.125
2023-11-21 12:50:16,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1508753.3333333333, ans=0.125
2023-11-21 12:50:19,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1508820.0, ans=0.125
2023-11-21 12:50:20,958 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9900, loss[loss=0.09536, simple_loss=0.1332, pruned_loss=0.02175, audio_tagging_loss=0.006998, over 15714.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09808, pruned_loss=0.01696, audio_tagging_loss=0.009237, over 3040765.59 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:50:53,691 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226350
2023-11-21 12:50:55,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1508953.3333333333, ans=0.1
2023-11-21 12:51:02,736 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.161e+01 8.727e+01 9.211e+01 1.112e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 12:51:13,184 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.01 vs. limit=15.0
2023-11-21 12:51:14,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1509086.6666666667, ans=0.125
2023-11-21 12:51:25,642 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 9950, loss[loss=0.06847, simple_loss=0.09082, pruned_loss=0.01396, audio_tagging_loss=0.009091, over 15124.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.097, pruned_loss=0.01666, audio_tagging_loss=0.009393, over 3042990.90 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:51:26,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.44 vs. limit=10.0
2023-11-21 12:51:42,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1509220.0, ans=0.125
2023-11-21 12:51:57,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226400
2023-11-21 12:51:59,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1509286.6666666667, ans=0.125
2023-11-21 12:52:02,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1509286.6666666667, ans=0.125
2023-11-21 12:52:03,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1509353.3333333333, ans=0.125
2023-11-21 12:52:12,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1509353.3333333333, ans=0.1
2023-11-21 12:52:14,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1509353.3333333333, ans=0.0
2023-11-21 12:52:18,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1509420.0, ans=0.125
2023-11-21 12:52:18,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.65 vs. limit=15.0
2023-11-21 12:52:29,855 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10000, loss[loss=0.08057, simple_loss=0.1005, pruned_loss=0.01938, audio_tagging_loss=0.01094, over 15181.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09691, pruned_loss=0.01677, audio_tagging_loss=0.009345, over 3038155.38 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:52:41,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1509553.3333333333, ans=0.125
2023-11-21 12:52:41,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1509553.3333333333, ans=0.0
2023-11-21 12:53:00,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1509620.0, ans=0.07
2023-11-21 12:53:01,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226450
2023-11-21 12:53:10,188 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 7.892e+01 8.555e+01 9.256e+01 1.223e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 12:53:15,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.98 vs. limit=15.0
2023-11-21 12:53:33,290 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10050, loss[loss=0.08278, simple_loss=0.09718, pruned_loss=0.02355, audio_tagging_loss=0.01064, over 14668.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09705, pruned_loss=0.01677, audio_tagging_loss=0.009409, over 3036759.53 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:53:36,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1509820.0, ans=0.0
2023-11-21 12:53:40,065 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.84 vs. limit=15.0
2023-11-21 12:54:01,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1509953.3333333333, ans=0.125
2023-11-21 12:54:03,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1509953.3333333333, ans=0.1
2023-11-21 12:54:05,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226500
2023-11-21 12:54:26,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1510086.6666666667, ans=0.125
2023-11-21 12:54:35,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1510153.3333333333, ans=0.2
2023-11-21 12:54:36,254 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.17 vs. limit=15.0
2023-11-21 12:54:36,872 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10100, loss[loss=0.07383, simple_loss=0.0812, pruned_loss=0.02054, audio_tagging_loss=0.0127, over 14435.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09666, pruned_loss=0.01675, audio_tagging_loss=0.009486, over 3047291.73 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:54:38,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1510153.3333333333, ans=0.2
2023-11-21 12:54:38,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1510153.3333333333, ans=0.1
2023-11-21 12:54:40,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1510153.3333333333, ans=0.0
2023-11-21 12:55:03,195 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.43 vs. limit=22.5
2023-11-21 12:55:08,772 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226550
2023-11-21 12:55:09,269 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.28 vs. limit=15.0
2023-11-21 12:55:10,197 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=9.416e-02
2023-11-21 12:55:17,037 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.133e+01 8.640e+01 9.422e+01 1.310e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 12:55:24,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1510353.3333333333, ans=0.2
2023-11-21 12:55:27,648 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:55:40,537 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10150, loss[loss=0.06126, simple_loss=0.08142, pruned_loss=0.01277, audio_tagging_loss=0.007778, over 15903.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09563, pruned_loss=0.0165, audio_tagging_loss=0.009618, over 3044730.18 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:56:02,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1510553.3333333333, ans=0.125
2023-11-21 12:56:09,686 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:56:12,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226600
2023-11-21 12:56:23,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1510686.6666666667, ans=0.125
2023-11-21 12:56:27,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1510686.6666666667, ans=0.0
2023-11-21 12:56:30,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1510686.6666666667, ans=0.125
2023-11-21 12:56:41,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1510753.3333333333, ans=0.1
2023-11-21 12:56:44,938 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10200, loss[loss=0.06622, simple_loss=0.07964, pruned_loss=0.01478, audio_tagging_loss=0.01162, over 14601.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.0951, pruned_loss=0.01652, audio_tagging_loss=0.0097, over 3049610.55 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:56:59,882 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:57:07,529 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:57:16,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226650
2023-11-21 12:57:22,438 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=15.69 vs. limit=15.0
2023-11-21 12:57:25,997 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.532e+01 7.915e+01 8.561e+01 9.293e+01 1.189e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 12:57:37,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1511086.6666666667, ans=0.0
2023-11-21 12:57:43,932 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.75 vs. limit=15.0
2023-11-21 12:57:48,045 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10250, loss[loss=0.06235, simple_loss=0.08132, pruned_loss=0.009267, audio_tagging_loss=0.01242, over 16282.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09593, pruned_loss=0.01663, audio_tagging_loss=0.009673, over 3043271.92 frames. ], batch size: 63, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:57:50,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1511153.3333333333, ans=0.0
2023-11-21 12:58:19,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1511286.6666666667, ans=0.125
2023-11-21 12:58:20,816 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226700
2023-11-21 12:58:35,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1511353.3333333333, ans=0.125
2023-11-21 12:58:47,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1511420.0, ans=0.0
2023-11-21 12:58:51,845 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10300, loss[loss=0.09096, simple_loss=0.1238, pruned_loss=0.02064, audio_tagging_loss=0.008402, over 15192.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09612, pruned_loss=0.01676, audio_tagging_loss=0.009813, over 3044670.03 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 12:59:05,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1511553.3333333333, ans=0.1
2023-11-21 12:59:15,813 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.28 vs. limit=10.0
2023-11-21 12:59:18,406 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.05 vs. limit=15.0
2023-11-21 12:59:24,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226750
2023-11-21 12:59:25,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1511620.0, ans=0.0
2023-11-21 12:59:28,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.33 vs. limit=22.5
2023-11-21 12:59:32,485 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.325e+01 8.857e+01 9.782e+01 1.240e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 12:59:41,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1511686.6666666667, ans=0.125
2023-11-21 12:59:56,456 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10350, loss[loss=0.08592, simple_loss=0.1193, pruned_loss=0.01885, audio_tagging_loss=0.007442, over 16564.00 frames. ], tot_loss[loss=0.07474, simple_loss=0.09632, pruned_loss=0.01674, audio_tagging_loss=0.009842, over 3046020.27 frames. ], batch size: 59, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:00:01,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1511820.0, ans=0.2
2023-11-21 13:00:27,942 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226800
2023-11-21 13:00:32,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1511953.3333333333, ans=0.09899494936611666
2023-11-21 13:00:35,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1512020.0, ans=0.0
2023-11-21 13:00:51,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1512086.6666666667, ans=0.125
2023-11-21 13:01:00,010 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10400, loss[loss=0.07083, simple_loss=0.0952, pruned_loss=0.01441, audio_tagging_loss=0.008824, over 15194.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09624, pruned_loss=0.0167, audio_tagging_loss=0.009832, over 3046799.40 frames. ], batch size: 57, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:01:24,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1512286.6666666667, ans=0.125
2023-11-21 13:01:32,591 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226850
2023-11-21 13:01:34,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.28 vs. limit=15.0
2023-11-21 13:01:42,510 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.081e+01 8.812e+01 9.594e+01 2.017e+02, threshold=1.762e+02, percent-clipped=1.0
2023-11-21 13:01:50,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1512420.0, ans=0.125
2023-11-21 13:01:58,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1512420.0, ans=0.0
2023-11-21 13:02:03,561 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10450, loss[loss=0.0808, simple_loss=0.1072, pruned_loss=0.01978, audio_tagging_loss=0.007434, over 14631.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09593, pruned_loss=0.01664, audio_tagging_loss=0.009751, over 3049092.29 frames. ], batch size: 57, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:02:05,775 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:02:30,425 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.54 vs. limit=15.0
2023-11-21 13:02:36,213 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226900
2023-11-21 13:02:49,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1512686.6666666667, ans=0.125
2023-11-21 13:02:57,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1512753.3333333333, ans=0.0
2023-11-21 13:03:08,418 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10500, loss[loss=0.07258, simple_loss=0.0926, pruned_loss=0.01791, audio_tagging_loss=0.008369, over 15180.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09613, pruned_loss=0.01658, audio_tagging_loss=0.009612, over 3049877.24 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:03:13,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1512820.0, ans=0.125
2023-11-21 13:03:19,228 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.53 vs. limit=8.0
2023-11-21 13:03:32,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1512953.3333333333, ans=0.125
2023-11-21 13:03:34,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1512953.3333333333, ans=0.125
2023-11-21 13:03:36,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.91 vs. limit=10.0
2023-11-21 13:03:38,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 226950
2023-11-21 13:03:43,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1512953.3333333333, ans=0.0
2023-11-21 13:03:44,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1513020.0, ans=0.125
2023-11-21 13:03:45,251 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0
2023-11-21 13:03:49,552 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.276e+01 8.777e+01 9.449e+01 1.286e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 13:03:54,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1513020.0, ans=0.125
2023-11-21 13:03:56,814 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.53 vs. limit=22.5
2023-11-21 13:04:11,142 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10550, loss[loss=0.05416, simple_loss=0.06067, pruned_loss=0.009719, audio_tagging_loss=0.0141, over 15238.00 frames. ], tot_loss[loss=0.074, simple_loss=0.0956, pruned_loss=0.01662, audio_tagging_loss=0.009579, over 3042922.53 frames. ], batch size: 59, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:04:27,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1513220.0, ans=0.125
2023-11-21 13:04:40,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1513286.6666666667, ans=0.125
2023-11-21 13:04:43,729 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227000
2023-11-21 13:04:49,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1513353.3333333333, ans=0.1
2023-11-21 13:05:14,605 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10600, loss[loss=0.08827, simple_loss=0.1162, pruned_loss=0.02152, audio_tagging_loss=0.00863, over 14503.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.09516, pruned_loss=0.01661, audio_tagging_loss=0.009611, over 3042024.65 frames. ], batch size: 54, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:05:30,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_na.min_abs, batch_count=1513553.3333333333, ans=0.02
2023-11-21 13:05:31,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1513553.3333333333, ans=0.0
2023-11-21 13:05:47,393 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227050
2023-11-21 13:05:49,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1513620.0, ans=0.04949747468305833
2023-11-21 13:05:57,032 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.166e+01 8.856e+01 9.761e+01 1.795e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-21 13:06:12,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1513753.3333333333, ans=0.125
2023-11-21 13:06:19,704 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10650, loss[loss=0.07944, simple_loss=0.107, pruned_loss=0.01863, audio_tagging_loss=0.007304, over 15090.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09573, pruned_loss=0.01665, audio_tagging_loss=0.009562, over 3044346.88 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:06:30,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0
2023-11-21 13:06:43,869 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.47 vs. limit=6.0
2023-11-21 13:06:50,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227100
2023-11-21 13:07:00,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1514020.0, ans=0.125
2023-11-21 13:07:10,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1514086.6666666667, ans=0.0
2023-11-21 13:07:19,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1514086.6666666667, ans=0.07
2023-11-21 13:07:20,029 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:07:22,124 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10700, loss[loss=0.06709, simple_loss=0.08971, pruned_loss=0.01481, audio_tagging_loss=0.007434, over 15667.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09615, pruned_loss=0.01658, audio_tagging_loss=0.009595, over 3047718.47 frames. ], batch size: 59, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:07:28,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1514153.3333333333, ans=0.07
2023-11-21 13:07:29,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1514153.3333333333, ans=0.125
2023-11-21 13:07:41,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1514220.0, ans=0.0
2023-11-21 13:07:43,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1514220.0, ans=0.1
2023-11-21 13:07:43,816 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.30 vs. limit=15.0
2023-11-21 13:07:47,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1514286.6666666667, ans=0.125
2023-11-21 13:07:51,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1514286.6666666667, ans=0.0
2023-11-21 13:07:55,029 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227150
2023-11-21 13:08:04,776 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.492e+01 8.011e+01 8.687e+01 9.462e+01 1.243e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 13:08:14,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0
2023-11-21 13:08:18,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1514420.0, ans=0.125
2023-11-21 13:08:25,526 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10750, loss[loss=0.06991, simple_loss=0.09335, pruned_loss=0.01439, audio_tagging_loss=0.008841, over 14868.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09644, pruned_loss=0.01651, audio_tagging_loss=0.009534, over 3043111.99 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:08:25,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1514486.6666666667, ans=0.1
2023-11-21 13:08:33,959 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:08:42,161 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.10 vs. limit=15.0
2023-11-21 13:08:46,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.94 vs. limit=12.0
2023-11-21 13:08:48,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1514553.3333333333, ans=0.95
2023-11-21 13:08:50,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0
2023-11-21 13:08:54,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1514620.0, ans=0.07
2023-11-21 13:08:57,930 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227200
2023-11-21 13:09:07,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1514686.6666666667, ans=0.125
2023-11-21 13:09:17,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1514753.3333333333, ans=0.0
2023-11-21 13:09:30,346 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10800, loss[loss=0.07653, simple_loss=0.1042, pruned_loss=0.01725, audio_tagging_loss=0.007174, over 14518.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09639, pruned_loss=0.01638, audio_tagging_loss=0.009429, over 3047240.38 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:09:31,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1514820.0, ans=0.07
2023-11-21 13:09:55,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1514953.3333333333, ans=0.125
2023-11-21 13:10:01,804 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227250
2023-11-21 13:10:13,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 7.864e+01 8.499e+01 9.519e+01 1.176e+02, threshold=1.700e+02, percent-clipped=0.0
2023-11-21 13:10:32,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1515086.6666666667, ans=0.125
2023-11-21 13:10:33,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1515153.3333333333, ans=0.125
2023-11-21 13:10:34,454 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10850, loss[loss=0.07536, simple_loss=0.1017, pruned_loss=0.01486, audio_tagging_loss=0.009644, over 14405.00 frames. ], tot_loss[loss=0.07363, simple_loss=0.0957, pruned_loss=0.01631, audio_tagging_loss=0.009466, over 3048040.60 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:10:37,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1515153.3333333333, ans=0.125
2023-11-21 13:10:44,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.90 vs. limit=22.5
2023-11-21 13:11:06,262 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227300
2023-11-21 13:11:08,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1515286.6666666667, ans=0.5
2023-11-21 13:11:33,368 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 13:11:38,247 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10900, loss[loss=0.1121, simple_loss=0.1399, pruned_loss=0.03587, audio_tagging_loss=0.006275, over 16637.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.09598, pruned_loss=0.01646, audio_tagging_loss=0.009484, over 3053933.50 frames. ], batch size: 60, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:11:50,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1515553.3333333333, ans=0.1
2023-11-21 13:11:55,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1515553.3333333333, ans=0.0
2023-11-21 13:11:58,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1515553.3333333333, ans=0.2
2023-11-21 13:12:11,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227350
2023-11-21 13:12:17,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1515686.6666666667, ans=0.125
2023-11-21 13:12:22,177 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.236e+01 8.108e+01 8.684e+01 9.152e+01 1.151e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 13:12:40,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1515753.3333333333, ans=0.125
2023-11-21 13:12:41,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1515820.0, ans=0.2
2023-11-21 13:12:42,776 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 10950, loss[loss=0.07323, simple_loss=0.1009, pruned_loss=0.0133, audio_tagging_loss=0.009476, over 15603.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.0968, pruned_loss=0.01661, audio_tagging_loss=0.009503, over 3058100.99 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:12:53,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1515820.0, ans=0.0
2023-11-21 13:13:03,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1515886.6666666667, ans=0.95
2023-11-21 13:13:13,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1515953.3333333333, ans=0.125
2023-11-21 13:13:14,798 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227400
2023-11-21 13:13:21,691 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.67 vs. limit=6.0
2023-11-21 13:13:26,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1516020.0, ans=0.125
2023-11-21 13:13:30,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.79 vs. limit=15.0
2023-11-21 13:13:34,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1516086.6666666667, ans=0.0
2023-11-21 13:13:40,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1516086.6666666667, ans=0.125
2023-11-21 13:13:48,218 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11000, loss[loss=0.08355, simple_loss=0.1115, pruned_loss=0.01964, audio_tagging_loss=0.008164, over 15408.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09666, pruned_loss=0.01679, audio_tagging_loss=0.009585, over 3048711.49 frames. ], batch size: 54, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:13:58,147 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 13:14:20,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227450
2023-11-21 13:14:32,297 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.986e+01 8.066e+01 8.741e+01 9.512e+01 1.178e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 13:14:45,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1516420.0, ans=0.0
2023-11-21 13:14:52,777 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11050, loss[loss=0.06134, simple_loss=0.08523, pruned_loss=0.009122, audio_tagging_loss=0.009603, over 14710.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09719, pruned_loss=0.01675, audio_tagging_loss=0.00965, over 3053724.09 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:14:54,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1516486.6666666667, ans=0.125
2023-11-21 13:14:55,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1516486.6666666667, ans=0.125
2023-11-21 13:15:12,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1516553.3333333333, ans=0.0
2023-11-21 13:15:25,290 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227500
2023-11-21 13:15:42,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1516686.6666666667, ans=0.1
2023-11-21 13:15:57,816 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11100, loss[loss=0.07061, simple_loss=0.09917, pruned_loss=0.01233, audio_tagging_loss=0.008696, over 15231.00 frames. ], tot_loss[loss=0.07561, simple_loss=0.09814, pruned_loss=0.01689, audio_tagging_loss=0.009655, over 3062652.29 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:16:10,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1516886.6666666667, ans=0.125
2023-11-21 13:16:23,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1516953.3333333333, ans=0.125
2023-11-21 13:16:30,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227550
2023-11-21 13:16:33,899 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.03 vs. limit=15.0
2023-11-21 13:16:36,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1517020.0, ans=0.0
2023-11-21 13:16:41,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1517020.0, ans=0.125
2023-11-21 13:16:41,848 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.109e+01 8.821e+01 9.588e+01 2.410e+02, threshold=1.764e+02, percent-clipped=1.0
2023-11-21 13:16:49,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1517086.6666666667, ans=0.125
2023-11-21 13:17:03,330 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11150, loss[loss=0.07761, simple_loss=0.1076, pruned_loss=0.01484, audio_tagging_loss=0.008967, over 15042.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09775, pruned_loss=0.0168, audio_tagging_loss=0.009805, over 3066448.58 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:17:08,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1517153.3333333333, ans=0.125
2023-11-21 13:17:14,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=1517153.3333333333, ans=15.0
2023-11-21 13:17:35,384 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227600
2023-11-21 13:17:35,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1517286.6666666667, ans=0.125
2023-11-21 13:17:44,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1517353.3333333333, ans=0.0
2023-11-21 13:17:44,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1517353.3333333333, ans=0.0
2023-11-21 13:18:00,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1517420.0, ans=0.0
2023-11-21 13:18:01,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1517420.0, ans=0.125
2023-11-21 13:18:07,667 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11200, loss[loss=0.06956, simple_loss=0.08843, pruned_loss=0.01561, audio_tagging_loss=0.009732, over 15881.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09685, pruned_loss=0.01675, audio_tagging_loss=0.009978, over 3065002.06 frames. ], batch size: 62, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:18:19,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1517553.3333333333, ans=0.0
2023-11-21 13:18:25,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1517553.3333333333, ans=10.0
2023-11-21 13:18:31,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1517553.3333333333, ans=0.2
2023-11-21 13:18:40,683 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227650
2023-11-21 13:18:50,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1517686.6666666667, ans=0.125
2023-11-21 13:18:51,700 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.451e+01 8.180e+01 8.891e+01 9.538e+01 1.267e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 13:18:52,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1517686.6666666667, ans=0.2
2023-11-21 13:18:58,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1517753.3333333333, ans=0.125
2023-11-21 13:19:07,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1517753.3333333333, ans=0.125
2023-11-21 13:19:12,944 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11250, loss[loss=0.08232, simple_loss=0.1066, pruned_loss=0.02185, audio_tagging_loss=0.007158, over 15173.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09479, pruned_loss=0.01645, audio_tagging_loss=0.01003, over 3064954.43 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:19:34,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1517886.6666666667, ans=0.07
2023-11-21 13:19:41,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1517953.3333333333, ans=0.125
2023-11-21 13:19:45,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227700
2023-11-21 13:19:57,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.26 vs. limit=22.5
2023-11-21 13:20:14,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1518086.6666666667, ans=0.1
2023-11-21 13:20:18,097 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11300, loss[loss=0.05477, simple_loss=0.06611, pruned_loss=0.009736, audio_tagging_loss=0.01198, over 15124.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09443, pruned_loss=0.01622, audio_tagging_loss=0.009922, over 3057881.01 frames. ], batch size: 57, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:20:44,906 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.16 vs. limit=10.0
2023-11-21 13:20:48,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227750
2023-11-21 13:20:50,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0
2023-11-21 13:21:01,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.039e+01 8.792e+01 9.406e+01 1.983e+02, threshold=1.758e+02, percent-clipped=1.0
2023-11-21 13:21:05,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1518353.3333333333, ans=0.125
2023-11-21 13:21:07,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1518420.0, ans=0.1
2023-11-21 13:21:19,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.60 vs. limit=10.0
2023-11-21 13:21:21,390 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11350, loss[loss=0.08269, simple_loss=0.116, pruned_loss=0.01867, audio_tagging_loss=0.005992, over 14282.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09591, pruned_loss=0.01667, audio_tagging_loss=0.009655, over 3055772.27 frames. ], batch size: 53, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:21:31,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1518486.6666666667, ans=0.1
2023-11-21 13:21:34,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1518553.3333333333, ans=0.125
2023-11-21 13:21:51,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1518620.0, ans=0.125
2023-11-21 13:21:54,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227800
2023-11-21 13:22:00,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.43 vs. limit=5.0
2023-11-21 13:22:02,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1518686.6666666667, ans=0.035
2023-11-21 13:22:11,408 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-21 13:22:20,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1518753.3333333333, ans=0.0
2023-11-21 13:22:21,654 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.57 vs. limit=15.0
2023-11-21 13:22:26,006 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11400, loss[loss=0.05702, simple_loss=0.07719, pruned_loss=0.009783, audio_tagging_loss=0.008635, over 14420.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09602, pruned_loss=0.01668, audio_tagging_loss=0.009559, over 3057860.02 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:22:26,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1518820.0, ans=0.2
2023-11-21 13:22:26,604 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.07 vs. limit=15.0
2023-11-21 13:22:58,275 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227850
2023-11-21 13:23:02,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1518953.3333333333, ans=0.125
2023-11-21 13:23:07,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1519020.0, ans=0.1
2023-11-21 13:23:09,540 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.020e+01 8.128e+01 8.790e+01 9.462e+01 1.169e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 13:23:26,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.81 vs. limit=15.0
2023-11-21 13:23:30,974 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11450, loss[loss=0.08488, simple_loss=0.1179, pruned_loss=0.01718, audio_tagging_loss=0.008768, over 15612.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09673, pruned_loss=0.01685, audio_tagging_loss=0.009504, over 3052023.09 frames. ], batch size: 54, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:23:32,928 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0
2023-11-21 13:24:02,300 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227900
2023-11-21 13:24:34,782 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11500, loss[loss=0.07777, simple_loss=0.09714, pruned_loss=0.01526, audio_tagging_loss=0.01393, over 15473.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09632, pruned_loss=0.01681, audio_tagging_loss=0.009614, over 3050602.90 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:24:46,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1519553.3333333333, ans=0.125
2023-11-21 13:24:52,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1519553.3333333333, ans=0.125
2023-11-21 13:25:03,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1519620.0, ans=0.2
2023-11-21 13:25:07,788 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 227950
2023-11-21 13:25:15,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.whiten.whitening_limit, batch_count=1519686.6666666667, ans=15.0
2023-11-21 13:25:18,646 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.513e+01 8.149e+01 8.832e+01 9.671e+01 1.256e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 13:25:37,542 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:25:38,549 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11550, loss[loss=0.07191, simple_loss=0.0947, pruned_loss=0.01527, audio_tagging_loss=0.009297, over 15194.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09594, pruned_loss=0.0167, audio_tagging_loss=0.009697, over 3046437.93 frames. ], batch size: 55, lr: 3.56e-03, grad_scale: 16.0
2023-11-21 13:26:07,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1519953.3333333333, ans=0.0
2023-11-21 13:26:08,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1519953.3333333333, ans=0.125
2023-11-21 13:26:10,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228000
2023-11-21 13:26:20,436 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 13:26:32,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1520086.6666666667, ans=0.0
2023-11-21 13:26:43,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1520086.6666666667, ans=0.125
2023-11-21 13:26:47,332 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11600, loss[loss=0.06347, simple_loss=0.07905, pruned_loss=0.01403, audio_tagging_loss=0.009916, over 16019.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09686, pruned_loss=0.01692, audio_tagging_loss=0.009675, over 3040362.11 frames. ], batch size: 59, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:27:14,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1520286.6666666667, ans=0.125
2023-11-21 13:27:18,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228050
2023-11-21 13:27:24,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1520353.3333333333, ans=0.125
2023-11-21 13:27:24,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1520353.3333333333, ans=0.125
2023-11-21 13:27:32,533 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.674e+01 8.168e+01 8.737e+01 9.633e+01 1.247e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 13:27:51,789 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11650, loss[loss=0.0834, simple_loss=0.107, pruned_loss=0.01925, audio_tagging_loss=0.01064, over 16494.00 frames. ], tot_loss[loss=0.07478, simple_loss=0.0964, pruned_loss=0.01683, audio_tagging_loss=0.009747, over 3043811.38 frames. ], batch size: 61, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:27:54,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1520486.6666666667, ans=0.125
2023-11-21 13:27:59,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1520486.6666666667, ans=0.125
2023-11-21 13:28:18,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1520620.0, ans=0.125
2023-11-21 13:28:24,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228100
2023-11-21 13:28:46,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1520753.3333333333, ans=0.125
2023-11-21 13:28:54,841 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11700, loss[loss=0.06883, simple_loss=0.08598, pruned_loss=0.01557, audio_tagging_loss=0.01026, over 15198.00 frames. ], tot_loss[loss=0.07513, simple_loss=0.09703, pruned_loss=0.01693, audio_tagging_loss=0.009688, over 3044907.82 frames. ], batch size: 60, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:29:00,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1520820.0, ans=10.0
2023-11-21 13:29:26,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1520953.3333333333, ans=0.1
2023-11-21 13:29:27,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228150
2023-11-21 13:29:39,517 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.045e+01 8.764e+01 9.289e+01 1.171e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 13:29:55,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1521086.6666666667, ans=0.125
2023-11-21 13:29:59,676 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11750, loss[loss=0.07084, simple_loss=0.08664, pruned_loss=0.01624, audio_tagging_loss=0.01127, over 15081.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.0963, pruned_loss=0.01684, audio_tagging_loss=0.009757, over 3046309.79 frames. ], batch size: 57, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:30:02,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.84 vs. limit=22.5
2023-11-21 13:30:02,872 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:30:04,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1521153.3333333333, ans=0.0
2023-11-21 13:30:06,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1521153.3333333333, ans=0.125
2023-11-21 13:30:18,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1521220.0, ans=0.0
2023-11-21 13:30:30,721 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228200
2023-11-21 13:30:30,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1521286.6666666667, ans=0.125
2023-11-21 13:30:39,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1521353.3333333333, ans=0.09899494936611666
2023-11-21 13:30:43,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1521353.3333333333, ans=0.0
2023-11-21 13:31:03,786 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11800, loss[loss=0.07, simple_loss=0.09156, pruned_loss=0.01644, audio_tagging_loss=0.007779, over 15005.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09633, pruned_loss=0.01688, audio_tagging_loss=0.00976, over 3044366.06 frames. ], batch size: 55, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:31:10,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1521486.6666666667, ans=0.1
2023-11-21 13:31:14,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.69 vs. limit=15.0
2023-11-21 13:31:15,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1521553.3333333333, ans=0.0
2023-11-21 13:31:15,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1521553.3333333333, ans=0.125
2023-11-21 13:31:35,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228250
2023-11-21 13:31:48,731 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.292e+01 8.361e+01 9.006e+01 1.004e+02 1.264e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-21 13:31:50,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1521686.6666666667, ans=0.125
2023-11-21 13:32:07,059 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11850, loss[loss=0.05949, simple_loss=0.07345, pruned_loss=0.01125, audio_tagging_loss=0.01152, over 15412.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09633, pruned_loss=0.01688, audio_tagging_loss=0.009845, over 3041370.93 frames. ], batch size: 59, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:32:16,280 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.06 vs. limit=15.0
2023-11-21 13:32:32,061 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.91 vs. limit=15.0
2023-11-21 13:32:39,637 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.26 vs. limit=10.0
2023-11-21 13:32:40,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228300
2023-11-21 13:32:45,833 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.78 vs. limit=10.0
2023-11-21 13:33:11,965 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11900, loss[loss=0.0826, simple_loss=0.1068, pruned_loss=0.02017, audio_tagging_loss=0.009012, over 16272.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09709, pruned_loss=0.01692, audio_tagging_loss=0.009918, over 3037370.25 frames. ], batch size: 58, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:33:21,174 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-21 13:33:33,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1522220.0, ans=0.125
2023-11-21 13:33:43,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228350
2023-11-21 13:33:51,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1522353.3333333333, ans=0.0
2023-11-21 13:33:56,604 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.221e+01 8.725e+01 9.501e+01 1.211e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 13:34:12,420 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.75 vs. limit=15.0
2023-11-21 13:34:16,701 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 11950, loss[loss=0.04213, simple_loss=0.04634, pruned_loss=0.00571, audio_tagging_loss=0.01325, over 15652.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09605, pruned_loss=0.01688, audio_tagging_loss=0.01003, over 3032453.05 frames. ], batch size: 60, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:34:20,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1522486.6666666667, ans=0.0
2023-11-21 13:34:28,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1522553.3333333333, ans=0.0
2023-11-21 13:34:31,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1522553.3333333333, ans=0.04949747468305833
2023-11-21 13:34:38,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1522553.3333333333, ans=0.0
2023-11-21 13:34:47,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228400
2023-11-21 13:35:10,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1522753.3333333333, ans=0.0
2023-11-21 13:35:18,701 INFO [train_asr.py:1221] (2/4) Epoch 19, batch 12000, loss[loss=0.1147, simple_loss=0.1528, pruned_loss=0.02833, audio_tagging_loss=0.01, over 15127.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09708, pruned_loss=0.01707, audio_tagging_loss=0.009921, over 3045252.75 frames. ], batch size: 52, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:35:18,701 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 13:35:54,474 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9158, 3.5728, 4.8503, 4.5163], device='cuda:2')
2023-11-21 13:36:02,009 INFO [train_asr.py:1253] (2/4) Epoch 19, validation: loss=0.06018, simple_loss=0.05227, pruned_loss=0.005307, audio_tagging_loss=0.02874, over 4681554.00 frames. 
2023-11-21 13:36:02,010 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 13:36:02,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1522820.0, ans=0.2
2023-11-21 13:36:19,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1522886.6666666667, ans=0.125
2023-11-21 13:37:05,049 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 0, loss[loss=0.0737, simple_loss=0.08273, pruned_loss=0.0112, audio_tagging_loss=0.02114, over 14382.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.08273, pruned_loss=0.0112, audio_tagging_loss=0.02114, over 14382.00 frames. ], batch size: 55, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:37:05,050 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 13:37:31,697 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7856, 5.8716, 5.8937, 5.9219], device='cuda:2')
2023-11-21 13:37:41,808 INFO [train_asr.py:1253] (2/4) Epoch 20, validation: loss=0.05938, simple_loss=0.0523, pruned_loss=0.005287, audio_tagging_loss=0.02794, over 4681554.00 frames. 
2023-11-21 13:37:41,808 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 13:37:43,129 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228450
2023-11-21 13:37:55,094 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.213e+01 9.034e+01 9.702e+01 1.198e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 13:37:58,280 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.40 vs. limit=15.0
2023-11-21 13:38:08,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1523113.3333333333, ans=0.125
2023-11-21 13:38:14,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1523113.3333333333, ans=0.125
2023-11-21 13:38:20,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1523180.0, ans=0.07
2023-11-21 13:38:23,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1523180.0, ans=0.1
2023-11-21 13:38:44,845 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 50, loss[loss=0.08264, simple_loss=0.1033, pruned_loss=0.01626, audio_tagging_loss=0.01473, over 14607.00 frames. ], tot_loss[loss=0.084, simple_loss=0.09733, pruned_loss=0.01702, audio_tagging_loss=0.01831, over 690093.63 frames. ], batch size: 55, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:38:45,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1523313.3333333333, ans=0.0
2023-11-21 13:38:46,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228500
2023-11-21 13:38:48,670 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:39:05,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1523380.0, ans=0.0
2023-11-21 13:39:44,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.21 vs. limit=10.0
2023-11-21 13:39:49,070 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 100, loss[loss=0.07544, simple_loss=0.08917, pruned_loss=0.01475, audio_tagging_loss=0.0161, over 14927.00 frames. ], tot_loss[loss=0.08182, simple_loss=0.0958, pruned_loss=0.01629, audio_tagging_loss=0.01763, over 1202500.96 frames. ], batch size: 57, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:39:50,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228550
2023-11-21 13:39:50,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1523646.6666666667, ans=0.1
2023-11-21 13:39:58,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1523646.6666666667, ans=0.125
2023-11-21 13:40:04,264 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.140e+01 8.678e+01 9.460e+01 1.029e+02 1.398e+02, threshold=1.892e+02, percent-clipped=0.0
2023-11-21 13:40:06,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1523713.3333333333, ans=0.035
2023-11-21 13:40:30,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1523846.6666666667, ans=0.125
2023-11-21 13:40:45,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1523913.3333333333, ans=0.0
2023-11-21 13:40:54,001 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 150, loss[loss=0.08141, simple_loss=0.1072, pruned_loss=0.01724, audio_tagging_loss=0.01059, over 16351.00 frames. ], tot_loss[loss=0.08041, simple_loss=0.09626, pruned_loss=0.01637, audio_tagging_loss=0.01592, over 1610814.67 frames. ], batch size: 59, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:40:55,994 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228600
2023-11-21 13:41:58,553 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 200, loss[loss=0.07338, simple_loss=0.1072, pruned_loss=0.01167, audio_tagging_loss=0.008116, over 15343.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.09599, pruned_loss=0.01597, audio_tagging_loss=0.01393, over 1928887.46 frames. ], batch size: 56, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:41:59,826 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228650
2023-11-21 13:42:00,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1524313.3333333333, ans=0.125
2023-11-21 13:42:12,012 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.304e+01 8.770e+01 9.399e+01 1.399e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 13:42:13,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.03 vs. limit=15.0
2023-11-21 13:42:27,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1524446.6666666667, ans=0.2
2023-11-21 13:42:40,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1524513.3333333333, ans=0.125
2023-11-21 13:42:42,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1524513.3333333333, ans=0.125
2023-11-21 13:42:54,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1524580.0, ans=0.0
2023-11-21 13:43:02,074 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 250, loss[loss=0.0886, simple_loss=0.1134, pruned_loss=0.02154, audio_tagging_loss=0.01038, over 15508.00 frames. ], tot_loss[loss=0.07753, simple_loss=0.09671, pruned_loss=0.01648, audio_tagging_loss=0.0127, over 2174287.68 frames. ], batch size: 58, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:43:03,338 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228700
2023-11-21 13:43:20,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.21 vs. limit=10.0
2023-11-21 13:43:23,476 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:43:28,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1524780.0, ans=0.0
2023-11-21 13:43:32,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1524780.0, ans=0.125
2023-11-21 13:43:34,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1524780.0, ans=10.0
2023-11-21 13:43:54,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1524913.3333333333, ans=0.2
2023-11-21 13:44:07,306 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 300, loss[loss=0.06684, simple_loss=0.09411, pruned_loss=0.01366, audio_tagging_loss=0.006129, over 16107.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09661, pruned_loss=0.01642, audio_tagging_loss=0.01184, over 2366255.07 frames. ], batch size: 58, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:44:08,658 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228750
2023-11-21 13:44:19,498 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.63 vs. limit=15.0
2023-11-21 13:44:21,333 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.166e+01 8.999e+01 9.797e+01 1.140e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 13:44:44,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1525180.0, ans=0.0
2023-11-21 13:45:08,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1525246.6666666667, ans=0.125
2023-11-21 13:45:11,683 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 350, loss[loss=0.09235, simple_loss=0.1162, pruned_loss=0.02531, audio_tagging_loss=0.008911, over 14387.00 frames. ], tot_loss[loss=0.07547, simple_loss=0.09575, pruned_loss=0.01633, audio_tagging_loss=0.01126, over 2520333.98 frames. ], batch size: 54, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:45:12,946 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228800
2023-11-21 13:45:33,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1525380.0, ans=0.1
2023-11-21 13:45:39,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1525446.6666666667, ans=0.125
2023-11-21 13:45:54,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1525513.3333333333, ans=0.025
2023-11-21 13:46:07,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.90 vs. limit=12.0
2023-11-21 13:46:14,941 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:46:15,861 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 400, loss[loss=0.08367, simple_loss=0.1064, pruned_loss=0.02118, audio_tagging_loss=0.009303, over 15604.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09652, pruned_loss=0.01664, audio_tagging_loss=0.01076, over 2639139.80 frames. ], batch size: 59, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:46:17,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228850
2023-11-21 13:46:21,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1525646.6666666667, ans=0.125
2023-11-21 13:46:22,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1525646.6666666667, ans=0.09899494936611666
2023-11-21 13:46:30,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.562e+01 8.083e+01 8.649e+01 9.202e+01 1.163e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 13:46:36,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1525713.3333333333, ans=0.125
2023-11-21 13:46:50,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=15.24 vs. limit=15.0
2023-11-21 13:47:20,827 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 450, loss[loss=0.07273, simple_loss=0.09486, pruned_loss=0.01535, audio_tagging_loss=0.009951, over 14885.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.09641, pruned_loss=0.01657, audio_tagging_loss=0.01053, over 2722905.67 frames. ], batch size: 55, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:47:22,698 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228900
2023-11-21 13:47:24,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1525980.0, ans=0.0
2023-11-21 13:47:30,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1525980.0, ans=0.07
2023-11-21 13:48:05,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1526180.0, ans=0.125
2023-11-21 13:48:24,767 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 500, loss[loss=0.0699, simple_loss=0.08287, pruned_loss=0.01745, audio_tagging_loss=0.01103, over 14349.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09609, pruned_loss=0.01666, audio_tagging_loss=0.01022, over 2787154.45 frames. ], batch size: 53, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:48:25,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.78 vs. limit=22.5
2023-11-21 13:48:26,119 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 228950
2023-11-21 13:48:28,653 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:48:32,001 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.50 vs. limit=12.0
2023-11-21 13:48:38,767 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.882e+01 7.917e+01 8.636e+01 9.467e+01 1.294e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-21 13:48:43,332 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.45 vs. limit=15.0
2023-11-21 13:48:56,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0
2023-11-21 13:49:28,430 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.89 vs. limit=15.0
2023-11-21 13:49:28,889 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 550, loss[loss=0.07691, simple_loss=0.1133, pruned_loss=0.01514, audio_tagging_loss=0.005091, over 15857.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09651, pruned_loss=0.01667, audio_tagging_loss=0.01, over 2839915.13 frames. ], batch size: 59, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:49:30,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229000
2023-11-21 13:49:34,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1526646.6666666667, ans=0.05
2023-11-21 13:50:07,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1526846.6666666667, ans=0.0
2023-11-21 13:50:31,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1526913.3333333333, ans=0.2
2023-11-21 13:50:33,710 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 600, loss[loss=0.07746, simple_loss=0.09923, pruned_loss=0.01848, audio_tagging_loss=0.009365, over 15958.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09496, pruned_loss=0.01636, audio_tagging_loss=0.009969, over 2885306.81 frames. ], batch size: 60, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:50:35,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229050
2023-11-21 13:50:47,736 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.121e+01 7.930e+01 8.613e+01 9.327e+01 1.199e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 13:51:07,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1527113.3333333333, ans=0.1
2023-11-21 13:51:19,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1527180.0, ans=0.1
2023-11-21 13:51:38,029 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 650, loss[loss=0.05987, simple_loss=0.07139, pruned_loss=0.0129, audio_tagging_loss=0.01127, over 14243.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09582, pruned_loss=0.01668, audio_tagging_loss=0.009969, over 2920464.68 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:51:39,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229100
2023-11-21 13:52:02,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.22 vs. limit=8.0
2023-11-21 13:52:05,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1527446.6666666667, ans=0.125
2023-11-21 13:52:09,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1527446.6666666667, ans=0.125
2023-11-21 13:52:16,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1527513.3333333333, ans=0.125
2023-11-21 13:52:30,086 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.12 vs. limit=15.0
2023-11-21 13:52:40,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1527646.6666666667, ans=0.0
2023-11-21 13:52:41,376 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 700, loss[loss=0.08069, simple_loss=0.1038, pruned_loss=0.01713, audio_tagging_loss=0.01167, over 15355.00 frames. ], tot_loss[loss=0.07508, simple_loss=0.0968, pruned_loss=0.01679, audio_tagging_loss=0.009892, over 2950883.97 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:52:41,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0
2023-11-21 13:52:43,385 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229150
2023-11-21 13:52:55,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1527713.3333333333, ans=0.125
2023-11-21 13:52:55,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1527713.3333333333, ans=0.125
2023-11-21 13:52:56,221 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.031e+01 8.760e+01 9.624e+01 1.233e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 13:53:15,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1527780.0, ans=0.2
2023-11-21 13:53:16,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.07 vs. limit=15.0
2023-11-21 13:53:17,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1527780.0, ans=0.125
2023-11-21 13:53:31,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1527846.6666666667, ans=0.015
2023-11-21 13:53:45,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1527913.3333333333, ans=0.2
2023-11-21 13:53:47,345 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 750, loss[loss=0.06586, simple_loss=0.08514, pruned_loss=0.01424, audio_tagging_loss=0.009046, over 15715.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09626, pruned_loss=0.01661, audio_tagging_loss=0.009823, over 2967258.91 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:53:48,713 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229200
2023-11-21 13:53:50,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1527980.0, ans=0.0
2023-11-21 13:53:53,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1527980.0, ans=0.2
2023-11-21 13:54:06,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1528046.6666666667, ans=0.125
2023-11-21 13:54:13,088 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.05 vs. limit=15.0
2023-11-21 13:54:34,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1528180.0, ans=0.1
2023-11-21 13:54:48,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1528246.6666666667, ans=0.2
2023-11-21 13:54:48,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1528246.6666666667, ans=0.0
2023-11-21 13:54:52,702 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 800, loss[loss=0.0725, simple_loss=0.09537, pruned_loss=0.0155, audio_tagging_loss=0.009313, over 14755.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09613, pruned_loss=0.01656, audio_tagging_loss=0.009763, over 2992353.00 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:54:54,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229250
2023-11-21 13:54:55,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1528313.3333333333, ans=0.0
2023-11-21 13:55:06,303 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.111e+01 8.814e+01 9.591e+01 1.302e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 13:55:19,044 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.36 vs. limit=15.0
2023-11-21 13:55:21,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1528446.6666666667, ans=0.07
2023-11-21 13:55:44,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1528580.0, ans=0.0
2023-11-21 13:55:46,877 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.18 vs. limit=22.5
2023-11-21 13:55:52,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1528580.0, ans=0.2
2023-11-21 13:55:56,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.14 vs. limit=22.5
2023-11-21 13:55:57,279 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 850, loss[loss=0.07689, simple_loss=0.1018, pruned_loss=0.01649, audio_tagging_loss=0.009469, over 17419.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09529, pruned_loss=0.01639, audio_tagging_loss=0.009971, over 3006817.01 frames. ], batch size: 64, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:55:58,583 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229300
2023-11-21 13:56:10,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1528713.3333333333, ans=0.1
2023-11-21 13:56:15,179 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=15.0
2023-11-21 13:56:32,603 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.75 vs. limit=12.0
2023-11-21 13:57:01,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1528980.0, ans=0.2
2023-11-21 13:57:02,852 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 900, loss[loss=0.08829, simple_loss=0.109, pruned_loss=0.02273, audio_tagging_loss=0.01105, over 14618.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09453, pruned_loss=0.0162, audio_tagging_loss=0.01011, over 3009124.61 frames. ], batch size: 53, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 13:57:04,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229350
2023-11-21 13:57:19,061 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.780e+01 8.022e+01 8.539e+01 9.514e+01 1.359e+02, threshold=1.708e+02, percent-clipped=0.0
2023-11-21 13:58:08,509 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 950, loss[loss=0.06141, simple_loss=0.07122, pruned_loss=0.0117, audio_tagging_loss=0.0141, over 15055.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.09492, pruned_loss=0.01635, audio_tagging_loss=0.01014, over 3015737.56 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 13:58:09,955 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229400
2023-11-21 13:58:18,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1529313.3333333333, ans=0.04949747468305833
2023-11-21 13:58:28,137 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5
2023-11-21 13:58:29,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1529380.0, ans=0.125
2023-11-21 13:58:32,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1529446.6666666667, ans=0.2
2023-11-21 13:58:34,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1529446.6666666667, ans=0.0
2023-11-21 13:58:37,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0
2023-11-21 13:58:46,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1529513.3333333333, ans=0.0
2023-11-21 13:58:53,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1529513.3333333333, ans=0.0
2023-11-21 13:58:55,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1529513.3333333333, ans=0.1
2023-11-21 13:58:59,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1529580.0, ans=0.125
2023-11-21 13:59:12,403 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1000, loss[loss=0.06764, simple_loss=0.08747, pruned_loss=0.01247, audio_tagging_loss=0.01143, over 15555.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09474, pruned_loss=0.01626, audio_tagging_loss=0.009968, over 3017819.35 frames. ], batch size: 61, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 13:59:13,741 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229450
2023-11-21 13:59:20,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1529646.6666666667, ans=0.0
2023-11-21 13:59:27,766 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.761e+01 8.099e+01 8.790e+01 9.899e+01 1.209e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 13:59:41,901 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:00:10,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1529913.3333333333, ans=0.1
2023-11-21 14:00:16,818 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1050, loss[loss=0.08551, simple_loss=0.1215, pruned_loss=0.01683, audio_tagging_loss=0.007941, over 16445.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09492, pruned_loss=0.01623, audio_tagging_loss=0.009806, over 3026352.42 frames. ], batch size: 62, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:00:18,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229500
2023-11-21 14:00:19,853 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.06 vs. limit=15.0
2023-11-21 14:00:36,408 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.79 vs. limit=15.0
2023-11-21 14:00:48,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1530113.3333333333, ans=0.125
2023-11-21 14:01:10,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1530246.6666666667, ans=0.125
2023-11-21 14:01:22,719 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.08 vs. limit=15.0
2023-11-21 14:01:23,283 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1100, loss[loss=0.0583, simple_loss=0.06813, pruned_loss=0.01424, audio_tagging_loss=0.00999, over 14533.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09431, pruned_loss=0.01617, audio_tagging_loss=0.009721, over 3028432.34 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:01:24,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229550
2023-11-21 14:01:27,043 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:01:35,873 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:01:38,198 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.405e+01 8.211e+01 9.088e+01 9.888e+01 2.167e+02, threshold=1.818e+02, percent-clipped=1.0
2023-11-21 14:01:45,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.64 vs. limit=15.0
2023-11-21 14:02:09,971 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:02:17,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.65 vs. limit=15.0
2023-11-21 14:02:23,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1530580.0, ans=0.125
2023-11-21 14:02:28,043 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1150, loss[loss=0.06497, simple_loss=0.08838, pruned_loss=0.01061, audio_tagging_loss=0.01018, over 15891.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09464, pruned_loss=0.01629, audio_tagging_loss=0.009585, over 3027319.46 frames. ], batch size: 58, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:02:29,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229600
2023-11-21 14:03:16,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1530846.6666666667, ans=0.125
2023-11-21 14:03:26,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1530913.3333333333, ans=0.125
2023-11-21 14:03:31,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1530980.0, ans=0.125
2023-11-21 14:03:32,056 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1200, loss[loss=0.07967, simple_loss=0.108, pruned_loss=0.01797, audio_tagging_loss=0.007703, over 15531.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09458, pruned_loss=0.01623, audio_tagging_loss=0.009417, over 3032547.02 frames. ], batch size: 58, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:03:33,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229650
2023-11-21 14:03:37,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1530980.0, ans=0.0
2023-11-21 14:03:48,984 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.148e+01 8.693e+01 9.555e+01 1.704e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 14:04:06,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1531113.3333333333, ans=0.09899494936611666
2023-11-21 14:04:15,696 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.71 vs. limit=15.0
2023-11-21 14:04:38,329 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.20 vs. limit=15.0
2023-11-21 14:04:38,684 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1250, loss[loss=0.07259, simple_loss=0.08882, pruned_loss=0.0169, audio_tagging_loss=0.01129, over 15756.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09462, pruned_loss=0.01632, audio_tagging_loss=0.009511, over 3029003.41 frames. ], batch size: 60, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:04:40,059 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229700
2023-11-21 14:04:44,395 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.45 vs. limit=15.0
2023-11-21 14:04:48,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1531313.3333333333, ans=0.1
2023-11-21 14:04:57,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1531380.0, ans=0.125
2023-11-21 14:04:59,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1531380.0, ans=0.125
2023-11-21 14:05:03,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1531446.6666666667, ans=0.1
2023-11-21 14:05:06,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.91 vs. limit=22.5
2023-11-21 14:05:43,397 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1300, loss[loss=0.07459, simple_loss=0.09424, pruned_loss=0.02039, audio_tagging_loss=0.007074, over 14821.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09535, pruned_loss=0.01641, audio_tagging_loss=0.009405, over 3028953.98 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:05:44,686 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229750
2023-11-21 14:05:48,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1531646.6666666667, ans=0.125
2023-11-21 14:05:58,196 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.180e+01 8.668e+01 9.567e+01 1.145e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 14:06:01,376 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.69 vs. limit=15.0
2023-11-21 14:06:14,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1531780.0, ans=0.0
2023-11-21 14:06:20,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=1531780.0, ans=0.02
2023-11-21 14:06:20,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1531780.0, ans=0.2
2023-11-21 14:06:26,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1531846.6666666667, ans=0.0
2023-11-21 14:06:48,142 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1350, loss[loss=0.063, simple_loss=0.08139, pruned_loss=0.01473, audio_tagging_loss=0.007578, over 15302.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09523, pruned_loss=0.01635, audio_tagging_loss=0.00938, over 3027378.70 frames. ], batch size: 58, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:06:49,462 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229800
2023-11-21 14:06:55,965 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.55 vs. limit=15.0
2023-11-21 14:07:10,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1532046.6666666667, ans=0.0
2023-11-21 14:07:35,716 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:07:37,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1532180.0, ans=0.125
2023-11-21 14:07:40,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.62 vs. limit=22.5
2023-11-21 14:07:44,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.55 vs. limit=15.0
2023-11-21 14:07:55,128 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1400, loss[loss=0.07881, simple_loss=0.1044, pruned_loss=0.01632, audio_tagging_loss=0.01032, over 14994.00 frames. ], tot_loss[loss=0.07311, simple_loss=0.0949, pruned_loss=0.01618, audio_tagging_loss=0.009484, over 3041392.80 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:07:56,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229850
2023-11-21 14:08:00,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1532313.3333333333, ans=0.0
2023-11-21 14:08:01,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1532313.3333333333, ans=0.0
2023-11-21 14:08:07,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1532380.0, ans=0.1
2023-11-21 14:08:10,532 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.288e+01 8.939e+01 9.468e+01 2.052e+02, threshold=1.788e+02, percent-clipped=1.0
2023-11-21 14:08:25,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1532446.6666666667, ans=0.0
2023-11-21 14:08:39,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1532513.3333333333, ans=0.04949747468305833
2023-11-21 14:08:58,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1532646.6666666667, ans=0.125
2023-11-21 14:08:59,463 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1450, loss[loss=0.07215, simple_loss=0.09437, pruned_loss=0.01691, audio_tagging_loss=0.008052, over 14694.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09524, pruned_loss=0.01644, audio_tagging_loss=0.009557, over 3044791.50 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:09:00,919 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229900
2023-11-21 14:09:35,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1532780.0, ans=0.125
2023-11-21 14:09:41,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1532846.6666666667, ans=0.125
2023-11-21 14:09:51,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1532913.3333333333, ans=0.125
2023-11-21 14:09:51,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1532913.3333333333, ans=0.0
2023-11-21 14:10:01,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1532913.3333333333, ans=0.125
2023-11-21 14:10:03,589 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1500, loss[loss=0.08413, simple_loss=0.11, pruned_loss=0.01973, audio_tagging_loss=0.009378, over 15004.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09612, pruned_loss=0.01651, audio_tagging_loss=0.009593, over 3053990.44 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:10:04,918 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 229950
2023-11-21 14:10:18,868 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.143e+01 8.696e+01 9.361e+01 1.676e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 14:10:25,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1533046.6666666667, ans=0.0
2023-11-21 14:10:40,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1533113.3333333333, ans=0.125
2023-11-21 14:10:56,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1533246.6666666667, ans=0.05
2023-11-21 14:10:56,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0
2023-11-21 14:11:08,134 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1550, loss[loss=0.07222, simple_loss=0.08933, pruned_loss=0.0162, audio_tagging_loss=0.01135, over 15377.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09793, pruned_loss=0.01712, audio_tagging_loss=0.009548, over 3054627.07 frames. ], batch size: 60, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:11:10,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230000
2023-11-21 14:11:39,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1533446.6666666667, ans=0.0
2023-11-21 14:11:39,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.24 vs. limit=15.0
2023-11-21 14:11:42,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1533446.6666666667, ans=0.125
2023-11-21 14:11:52,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1533513.3333333333, ans=0.125
2023-11-21 14:12:14,370 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1600, loss[loss=0.06107, simple_loss=0.07969, pruned_loss=0.01054, audio_tagging_loss=0.01069, over 14541.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09686, pruned_loss=0.01671, audio_tagging_loss=0.009717, over 3052142.82 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:12:16,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230050
2023-11-21 14:12:30,020 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.321e+01 8.893e+01 9.688e+01 1.461e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 14:12:43,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1533780.0, ans=0.0
2023-11-21 14:12:52,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1533780.0, ans=0.2
2023-11-21 14:12:59,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.59 vs. limit=15.0
2023-11-21 14:13:01,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.27 vs. limit=15.0
2023-11-21 14:13:02,657 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5
2023-11-21 14:13:06,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1533913.3333333333, ans=0.125
2023-11-21 14:13:19,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.23 vs. limit=15.0
2023-11-21 14:13:19,720 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1650, loss[loss=0.07297, simple_loss=0.09064, pruned_loss=0.01695, audio_tagging_loss=0.0107, over 14149.00 frames. ], tot_loss[loss=0.07508, simple_loss=0.09702, pruned_loss=0.01682, audio_tagging_loss=0.009755, over 3045478.81 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:13:21,122 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230100
2023-11-21 14:13:27,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1533980.0, ans=0.125
2023-11-21 14:13:28,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1533980.0, ans=0.2
2023-11-21 14:13:37,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1534046.6666666667, ans=0.1
2023-11-21 14:13:47,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1534113.3333333333, ans=0.1
2023-11-21 14:14:20,699 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.24 vs. limit=10.0
2023-11-21 14:14:24,744 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1700, loss[loss=0.057, simple_loss=0.07034, pruned_loss=0.01253, audio_tagging_loss=0.009306, over 14463.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09686, pruned_loss=0.01694, audio_tagging_loss=0.009747, over 3045050.64 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:14:26,038 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230150
2023-11-21 14:14:39,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1534380.0, ans=0.2
2023-11-21 14:14:40,785 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.297e+01 8.260e+01 8.837e+01 9.595e+01 1.453e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 14:14:46,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1534380.0, ans=0.2
2023-11-21 14:14:47,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1534380.0, ans=0.125
2023-11-21 14:14:53,246 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.88 vs. limit=15.0
2023-11-21 14:15:03,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.30 vs. limit=15.0
2023-11-21 14:15:06,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.05 vs. limit=12.0
2023-11-21 14:15:08,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1534513.3333333333, ans=0.125
2023-11-21 14:15:08,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1534513.3333333333, ans=0.125
2023-11-21 14:15:10,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0
2023-11-21 14:15:20,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1534580.0, ans=0.0
2023-11-21 14:15:25,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1534580.0, ans=0.125
2023-11-21 14:15:30,399 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1750, loss[loss=0.0621, simple_loss=0.0768, pruned_loss=0.01474, audio_tagging_loss=0.008964, over 15405.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09619, pruned_loss=0.01665, audio_tagging_loss=0.009731, over 3049412.21 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:15:31,729 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230200
2023-11-21 14:15:31,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1534646.6666666667, ans=0.125
2023-11-21 14:15:37,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1534646.6666666667, ans=0.125
2023-11-21 14:15:40,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1534646.6666666667, ans=0.0
2023-11-21 14:15:45,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1534713.3333333333, ans=0.125
2023-11-21 14:15:47,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1534713.3333333333, ans=0.125
2023-11-21 14:16:11,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1534846.6666666667, ans=0.1
2023-11-21 14:16:17,875 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.56 vs. limit=15.0
2023-11-21 14:16:32,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1534913.3333333333, ans=0.1
2023-11-21 14:16:32,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1534913.3333333333, ans=0.125
2023-11-21 14:16:34,739 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1800, loss[loss=0.06938, simple_loss=0.08646, pruned_loss=0.01588, audio_tagging_loss=0.01027, over 14568.00 frames. ], tot_loss[loss=0.07411, simple_loss=0.0957, pruned_loss=0.01648, audio_tagging_loss=0.00978, over 3052582.30 frames. ], batch size: 53, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:16:36,063 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230250
2023-11-21 14:16:44,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1534980.0, ans=0.125
2023-11-21 14:16:51,331 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.097e+01 8.882e+01 9.608e+01 1.325e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 14:16:57,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1535046.6666666667, ans=0.1
2023-11-21 14:16:57,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1535046.6666666667, ans=0.125
2023-11-21 14:17:01,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1535113.3333333333, ans=0.0
2023-11-21 14:17:02,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1535113.3333333333, ans=0.1
2023-11-21 14:17:16,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1535180.0, ans=0.125
2023-11-21 14:17:39,530 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1850, loss[loss=0.06532, simple_loss=0.07573, pruned_loss=0.01445, audio_tagging_loss=0.013, over 14829.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09568, pruned_loss=0.0164, audio_tagging_loss=0.009723, over 3047466.44 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:17:40,812 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230300
2023-11-21 14:17:56,138 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.93 vs. limit=15.0
2023-11-21 14:18:02,249 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:18:08,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1535446.6666666667, ans=0.09899494936611666
2023-11-21 14:18:09,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1535446.6666666667, ans=0.0
2023-11-21 14:18:21,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1535513.3333333333, ans=0.1
2023-11-21 14:18:45,260 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1900, loss[loss=0.06948, simple_loss=0.09311, pruned_loss=0.01534, audio_tagging_loss=0.007589, over 15830.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09545, pruned_loss=0.01636, audio_tagging_loss=0.009623, over 3050660.76 frames. ], batch size: 58, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:18:46,581 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230350
2023-11-21 14:18:48,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1535646.6666666667, ans=0.1
2023-11-21 14:19:00,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1535713.3333333333, ans=0.1
2023-11-21 14:19:01,196 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 7.941e+01 8.654e+01 9.429e+01 1.278e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 14:19:16,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1535780.0, ans=0.5
2023-11-21 14:19:26,525 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-21 14:19:35,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1535913.3333333333, ans=0.125
2023-11-21 14:19:49,121 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 1950, loss[loss=0.07508, simple_loss=0.09786, pruned_loss=0.01851, audio_tagging_loss=0.007632, over 15360.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09577, pruned_loss=0.01649, audio_tagging_loss=0.00957, over 3053730.09 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:19:50,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230400
2023-11-21 14:20:09,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.84 vs. limit=15.0
2023-11-21 14:20:10,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=1536046.6666666667, ans=0.2
2023-11-21 14:20:20,700 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.29 vs. limit=15.0
2023-11-21 14:20:26,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.15 vs. limit=15.0
2023-11-21 14:20:37,760 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.01 vs. limit=15.0
2023-11-21 14:20:54,234 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2000, loss[loss=0.06847, simple_loss=0.08, pruned_loss=0.02016, audio_tagging_loss=0.008306, over 15162.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.09637, pruned_loss=0.01675, audio_tagging_loss=0.009582, over 3053713.02 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:20:55,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230450
2023-11-21 14:20:58,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1536313.3333333333, ans=0.0
2023-11-21 14:21:04,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1536313.3333333333, ans=0.125
2023-11-21 14:21:10,598 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 7.989e+01 8.739e+01 9.353e+01 1.086e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 14:21:13,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1536380.0, ans=0.125
2023-11-21 14:21:15,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1536380.0, ans=0.0
2023-11-21 14:21:18,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1536446.6666666667, ans=0.07
2023-11-21 14:21:19,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1536446.6666666667, ans=0.0
2023-11-21 14:21:58,517 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2050, loss[loss=0.05743, simple_loss=0.07269, pruned_loss=0.01039, audio_tagging_loss=0.01069, over 16593.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.0961, pruned_loss=0.01675, audio_tagging_loss=0.009514, over 3057097.58 frames. ], batch size: 63, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:21:59,786 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230500
2023-11-21 14:22:10,136 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:22:21,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1536713.3333333333, ans=0.0
2023-11-21 14:23:01,846 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2100, loss[loss=0.1043, simple_loss=0.1262, pruned_loss=0.02954, audio_tagging_loss=0.01161, over 14413.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09574, pruned_loss=0.01678, audio_tagging_loss=0.009521, over 3049349.17 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:23:03,198 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230550
2023-11-21 14:23:20,279 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.887e+01 8.079e+01 8.846e+01 9.402e+01 1.190e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 14:23:32,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1537113.3333333333, ans=0.95
2023-11-21 14:24:02,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1537246.6666666667, ans=0.125
2023-11-21 14:24:05,873 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2150, loss[loss=0.0653, simple_loss=0.07694, pruned_loss=0.01437, audio_tagging_loss=0.01246, over 14276.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09681, pruned_loss=0.0169, audio_tagging_loss=0.009529, over 3046110.24 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:24:06,829 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.47 vs. limit=8.0
2023-11-21 14:24:07,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230600
2023-11-21 14:24:14,321 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.42 vs. limit=15.0
2023-11-21 14:24:19,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1537380.0, ans=0.0
2023-11-21 14:24:30,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1537380.0, ans=0.125
2023-11-21 14:24:43,566 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.72 vs. limit=6.0
2023-11-21 14:24:45,361 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:24:54,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1537513.3333333333, ans=0.05
2023-11-21 14:25:09,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.53 vs. limit=15.0
2023-11-21 14:25:11,564 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2200, loss[loss=0.09727, simple_loss=0.1325, pruned_loss=0.0216, audio_tagging_loss=0.009431, over 15706.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09523, pruned_loss=0.01669, audio_tagging_loss=0.009596, over 3038891.16 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:25:12,929 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230650
2023-11-21 14:25:14,890 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.47 vs. limit=15.0
2023-11-21 14:25:18,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1537646.6666666667, ans=0.1
2023-11-21 14:25:28,620 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.037e+01 8.654e+01 9.441e+01 1.350e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 14:25:42,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1537780.0, ans=22.5
2023-11-21 14:25:46,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1537780.0, ans=0.125
2023-11-21 14:25:46,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1537780.0, ans=0.125
2023-11-21 14:25:47,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1537846.6666666667, ans=0.0
2023-11-21 14:25:47,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1537846.6666666667, ans=0.0
2023-11-21 14:25:53,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1537846.6666666667, ans=0.07
2023-11-21 14:26:12,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1537913.3333333333, ans=0.125
2023-11-21 14:26:14,309 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2250, loss[loss=0.07984, simple_loss=0.1027, pruned_loss=0.02122, audio_tagging_loss=0.007294, over 15184.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09616, pruned_loss=0.0169, audio_tagging_loss=0.009609, over 3039397.72 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:26:15,619 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230700
2023-11-21 14:27:00,844 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.32 vs. limit=8.0
2023-11-21 14:27:17,398 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2300, loss[loss=0.06933, simple_loss=0.07974, pruned_loss=0.01757, audio_tagging_loss=0.01189, over 14511.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09617, pruned_loss=0.01673, audio_tagging_loss=0.009576, over 3039153.70 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:27:18,761 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230750
2023-11-21 14:27:23,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1538313.3333333333, ans=0.125
2023-11-21 14:27:23,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1538313.3333333333, ans=15.0
2023-11-21 14:27:36,134 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.774e+01 7.885e+01 8.573e+01 9.171e+01 1.137e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 14:27:42,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1538446.6666666667, ans=0.1
2023-11-21 14:27:47,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1538446.6666666667, ans=0.07
2023-11-21 14:27:52,691 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0
2023-11-21 14:28:13,485 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:28:17,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1538580.0, ans=0.2
2023-11-21 14:28:22,097 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2350, loss[loss=0.1006, simple_loss=0.1302, pruned_loss=0.02715, audio_tagging_loss=0.00834, over 15853.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.0959, pruned_loss=0.01671, audio_tagging_loss=0.009675, over 3045071.96 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:28:23,421 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230800
2023-11-21 14:28:31,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1538646.6666666667, ans=0.125
2023-11-21 14:28:40,407 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.97 vs. limit=15.0
2023-11-21 14:28:46,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1538780.0, ans=0.125
2023-11-21 14:28:51,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1538780.0, ans=0.125
2023-11-21 14:29:00,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1538846.6666666667, ans=0.0
2023-11-21 14:29:26,656 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2400, loss[loss=0.06051, simple_loss=0.07588, pruned_loss=0.01295, audio_tagging_loss=0.009618, over 14934.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09659, pruned_loss=0.01674, audio_tagging_loss=0.009677, over 3048911.14 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:29:28,059 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230850
2023-11-21 14:29:35,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1538980.0, ans=0.5
2023-11-21 14:29:43,570 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.345e+01 8.985e+01 9.638e+01 1.227e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-21 14:29:53,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1539113.3333333333, ans=0.0
2023-11-21 14:30:01,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1539113.3333333333, ans=0.125
2023-11-21 14:30:06,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1539180.0, ans=15.0
2023-11-21 14:30:24,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1539246.6666666667, ans=0.2
2023-11-21 14:30:25,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1539246.6666666667, ans=0.2
2023-11-21 14:30:26,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1539246.6666666667, ans=0.125
2023-11-21 14:30:30,122 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2450, loss[loss=0.08874, simple_loss=0.1193, pruned_loss=0.02241, audio_tagging_loss=0.006664, over 15327.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09592, pruned_loss=0.01655, audio_tagging_loss=0.009773, over 3043759.09 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:30:31,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230900
2023-11-21 14:30:39,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1539313.3333333333, ans=0.125
2023-11-21 14:30:43,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1539380.0, ans=0.2
2023-11-21 14:30:53,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1539380.0, ans=0.0
2023-11-21 14:30:57,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0
2023-11-21 14:31:33,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1539646.6666666667, ans=0.125
2023-11-21 14:31:35,094 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2500, loss[loss=0.08289, simple_loss=0.1068, pruned_loss=0.01727, audio_tagging_loss=0.01223, over 14753.00 frames. ], tot_loss[loss=0.07411, simple_loss=0.09557, pruned_loss=0.01639, audio_tagging_loss=0.009937, over 3047193.41 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:31:37,012 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 230950
2023-11-21 14:31:38,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1539646.6666666667, ans=0.0
2023-11-21 14:31:40,802 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:31:42,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1539646.6666666667, ans=0.125
2023-11-21 14:31:53,384 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 7.942e+01 8.529e+01 9.362e+01 1.301e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 14:32:26,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1539913.3333333333, ans=0.125
2023-11-21 14:32:40,140 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2550, loss[loss=0.07286, simple_loss=0.09267, pruned_loss=0.01849, audio_tagging_loss=0.008031, over 14823.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09466, pruned_loss=0.0162, audio_tagging_loss=0.009875, over 3037768.43 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:32:41,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231000
2023-11-21 14:32:45,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1539980.0, ans=0.2
2023-11-21 14:32:52,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1540046.6666666667, ans=0.2
2023-11-21 14:32:52,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1540046.6666666667, ans=0.1
2023-11-21 14:32:56,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1540046.6666666667, ans=0.125
2023-11-21 14:32:57,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1540046.6666666667, ans=0.125
2023-11-21 14:33:14,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1540113.3333333333, ans=0.1
2023-11-21 14:33:17,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1540180.0, ans=10.0
2023-11-21 14:33:27,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.58 vs. limit=15.0
2023-11-21 14:33:29,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1540180.0, ans=0.0
2023-11-21 14:33:31,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1540246.6666666667, ans=0.125
2023-11-21 14:33:32,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1540246.6666666667, ans=0.1
2023-11-21 14:33:32,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1540246.6666666667, ans=0.125
2023-11-21 14:33:44,011 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2600, loss[loss=0.07399, simple_loss=0.1019, pruned_loss=0.01235, audio_tagging_loss=0.01069, over 14621.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09485, pruned_loss=0.01629, audio_tagging_loss=0.009787, over 3039952.32 frames. ], batch size: 53, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:33:45,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231050
2023-11-21 14:33:46,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.80 vs. limit=15.0
2023-11-21 14:33:58,595 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.58 vs. limit=10.0
2023-11-21 14:34:03,318 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.933e+01 8.665e+01 9.563e+01 1.438e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 14:34:18,124 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0
2023-11-21 14:34:45,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.50 vs. limit=10.0
2023-11-21 14:34:46,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1540646.6666666667, ans=0.0
2023-11-21 14:34:47,250 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2650, loss[loss=0.09107, simple_loss=0.1204, pruned_loss=0.02302, audio_tagging_loss=0.00787, over 14517.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09456, pruned_loss=0.01632, audio_tagging_loss=0.009692, over 3038441.33 frames. ], batch size: 54, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:34:48,569 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231100
2023-11-21 14:35:16,158 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:35:22,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1540780.0, ans=0.05
2023-11-21 14:35:28,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1540846.6666666667, ans=0.2
2023-11-21 14:35:31,632 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.44 vs. limit=22.5
2023-11-21 14:35:37,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1540913.3333333333, ans=0.1
2023-11-21 14:35:37,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1540913.3333333333, ans=0.125
2023-11-21 14:35:46,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.84 vs. limit=15.0
2023-11-21 14:35:51,737 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2700, loss[loss=0.05175, simple_loss=0.06444, pruned_loss=0.008582, audio_tagging_loss=0.01095, over 15312.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09322, pruned_loss=0.01595, audio_tagging_loss=0.009661, over 3046007.46 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:35:53,041 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231150
2023-11-21 14:35:53,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1540980.0, ans=0.125
2023-11-21 14:36:03,575 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:36:10,604 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 8.073e+01 8.815e+01 9.662e+01 1.358e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 14:36:22,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1541113.3333333333, ans=0.125
2023-11-21 14:36:24,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1541113.3333333333, ans=0.0
2023-11-21 14:36:34,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1541180.0, ans=0.0
2023-11-21 14:36:38,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1541180.0, ans=0.0
2023-11-21 14:36:49,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1541246.6666666667, ans=0.0
2023-11-21 14:36:50,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1541246.6666666667, ans=0.1
2023-11-21 14:36:55,383 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2750, loss[loss=0.06673, simple_loss=0.07466, pruned_loss=0.02127, audio_tagging_loss=0.008139, over 14669.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09342, pruned_loss=0.01597, audio_tagging_loss=0.009668, over 3042674.51 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:36:56,721 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231200
2023-11-21 14:37:09,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1541380.0, ans=0.2
2023-11-21 14:37:10,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1541380.0, ans=0.125
2023-11-21 14:37:10,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1541380.0, ans=0.125
2023-11-21 14:37:11,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1541380.0, ans=0.0
2023-11-21 14:37:24,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1541446.6666666667, ans=0.125
2023-11-21 14:37:26,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1541446.6666666667, ans=0.125
2023-11-21 14:37:32,236 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0
2023-11-21 14:37:42,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1541513.3333333333, ans=0.0
2023-11-21 14:37:43,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1541513.3333333333, ans=0.1
2023-11-21 14:37:50,371 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:37:59,547 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2800, loss[loss=0.0841, simple_loss=0.08964, pruned_loss=0.02743, audio_tagging_loss=0.01185, over 14249.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09336, pruned_loss=0.01608, audio_tagging_loss=0.00965, over 3042388.45 frames. ], batch size: 54, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:38:00,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231250
2023-11-21 14:38:03,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1541646.6666666667, ans=0.125
2023-11-21 14:38:12,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1541713.3333333333, ans=0.09899494936611666
2023-11-21 14:38:15,214 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.54 vs. limit=15.0
2023-11-21 14:38:17,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1541713.3333333333, ans=0.0
2023-11-21 14:38:19,381 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.133e+01 8.565e+01 9.239e+01 1.235e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 14:38:21,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.16 vs. limit=22.5
2023-11-21 14:38:42,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1541846.6666666667, ans=0.0
2023-11-21 14:38:46,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1541846.6666666667, ans=0.0
2023-11-21 14:38:46,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1541846.6666666667, ans=0.125
2023-11-21 14:38:56,916 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.65 vs. limit=10.0
2023-11-21 14:39:04,752 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2850, loss[loss=0.07984, simple_loss=0.09909, pruned_loss=0.021, audio_tagging_loss=0.009302, over 13694.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.0941, pruned_loss=0.01623, audio_tagging_loss=0.009593, over 3048503.21 frames. ], batch size: 52, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:39:06,094 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231300
2023-11-21 14:40:08,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.10 vs. limit=15.0
2023-11-21 14:40:09,148 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2900, loss[loss=0.07487, simple_loss=0.09324, pruned_loss=0.01791, audio_tagging_loss=0.01034, over 15316.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09484, pruned_loss=0.01633, audio_tagging_loss=0.009506, over 3052713.98 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:40:10,489 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231350
2023-11-21 14:40:15,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1542313.3333333333, ans=0.0
2023-11-21 14:40:17,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1542313.3333333333, ans=0.1
2023-11-21 14:40:21,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1542380.0, ans=0.2
2023-11-21 14:40:29,512 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.134e+01 8.745e+01 9.473e+01 1.198e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 14:40:42,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1542446.6666666667, ans=0.125
2023-11-21 14:40:45,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1542446.6666666667, ans=0.125
2023-11-21 14:40:45,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1542446.6666666667, ans=0.0
2023-11-21 14:40:56,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1542513.3333333333, ans=0.0
2023-11-21 14:41:09,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1542580.0, ans=6.0
2023-11-21 14:41:12,919 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 2950, loss[loss=0.06799, simple_loss=0.08614, pruned_loss=0.01589, audio_tagging_loss=0.009031, over 15099.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09545, pruned_loss=0.01651, audio_tagging_loss=0.009505, over 3056240.74 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:41:14,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231400
2023-11-21 14:41:27,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1542713.3333333333, ans=0.125
2023-11-21 14:41:55,080 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.65 vs. limit=22.5
2023-11-21 14:42:18,040 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3000, loss[loss=0.06676, simple_loss=0.07924, pruned_loss=0.01955, audio_tagging_loss=0.007587, over 14959.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09634, pruned_loss=0.0168, audio_tagging_loss=0.009415, over 3055620.30 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:42:18,040 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 14:42:56,769 INFO [train_asr.py:1253] (2/4) Epoch 20, validation: loss=0.05942, simple_loss=0.05225, pruned_loss=0.00524, audio_tagging_loss=0.02805, over 4681554.00 frames. 
2023-11-21 14:42:56,770 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 14:42:58,165 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231450
2023-11-21 14:43:05,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1542980.0, ans=0.1
2023-11-21 14:43:10,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1543046.6666666667, ans=0.125
2023-11-21 14:43:12,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543046.6666666667, ans=0.1
2023-11-21 14:43:17,881 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.247e+01 8.896e+01 9.640e+01 1.197e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 14:43:22,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543113.3333333333, ans=0.1
2023-11-21 14:43:29,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1543113.3333333333, ans=0.05
2023-11-21 14:43:36,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.whiten.whitening_limit, batch_count=1543180.0, ans=12.0
2023-11-21 14:43:46,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1543246.6666666667, ans=22.5
2023-11-21 14:43:52,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.56 vs. limit=15.0
2023-11-21 14:43:52,614 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.01 vs. limit=10.0
2023-11-21 14:43:55,565 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.54 vs. limit=15.0
2023-11-21 14:44:00,899 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3050, loss[loss=0.08293, simple_loss=0.1009, pruned_loss=0.02338, audio_tagging_loss=0.009116, over 14227.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.097, pruned_loss=0.01675, audio_tagging_loss=0.009566, over 3057641.37 frames. ], batch size: 54, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:44:02,284 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231500
2023-11-21 14:44:12,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543313.3333333333, ans=0.1
2023-11-21 14:44:13,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1543380.0, ans=0.125
2023-11-21 14:44:17,635 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.95 vs. limit=15.0
2023-11-21 14:44:17,753 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.53 vs. limit=15.0
2023-11-21 14:44:28,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1543446.6666666667, ans=0.1
2023-11-21 14:44:31,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1543446.6666666667, ans=0.0
2023-11-21 14:44:35,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1543446.6666666667, ans=0.0
2023-11-21 14:44:36,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1543446.6666666667, ans=0.125
2023-11-21 14:44:37,575 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:44:51,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1543580.0, ans=0.1
2023-11-21 14:45:05,766 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3100, loss[loss=0.08267, simple_loss=0.1174, pruned_loss=0.01518, audio_tagging_loss=0.008791, over 15277.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09838, pruned_loss=0.01697, audio_tagging_loss=0.009546, over 3060588.20 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:45:07,045 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231550
2023-11-21 14:45:17,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1543713.3333333333, ans=0.0
2023-11-21 14:45:17,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1543713.3333333333, ans=0.2
2023-11-21 14:45:19,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1543713.3333333333, ans=0.1
2023-11-21 14:45:20,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1543713.3333333333, ans=0.125
2023-11-21 14:45:25,358 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.074e+01 8.626e+01 9.315e+01 1.250e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 14:45:37,558 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:45:50,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1543846.6666666667, ans=0.0
2023-11-21 14:46:08,550 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3150, loss[loss=0.08261, simple_loss=0.1149, pruned_loss=0.0182, audio_tagging_loss=0.006974, over 15741.00 frames. ], tot_loss[loss=0.07582, simple_loss=0.09836, pruned_loss=0.01702, audio_tagging_loss=0.009628, over 3062506.95 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:46:09,952 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231600
2023-11-21 14:46:21,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1544046.6666666667, ans=0.07
2023-11-21 14:46:28,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1544046.6666666667, ans=0.0
2023-11-21 14:46:40,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1544113.3333333333, ans=0.0
2023-11-21 14:46:42,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1544113.3333333333, ans=0.0
2023-11-21 14:46:48,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1544180.0, ans=0.1
2023-11-21 14:47:02,352 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.0
2023-11-21 14:47:13,969 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3200, loss[loss=0.07835, simple_loss=0.09878, pruned_loss=0.01545, audio_tagging_loss=0.01351, over 13804.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09868, pruned_loss=0.01712, audio_tagging_loss=0.009656, over 3054209.77 frames. ], batch size: 52, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:47:15,313 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231650
2023-11-21 14:47:19,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1544313.3333333333, ans=0.5
2023-11-21 14:47:22,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1544313.3333333333, ans=0.0
2023-11-21 14:47:34,900 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.766e+01 7.989e+01 8.860e+01 9.564e+01 1.510e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-21 14:48:05,228 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.67 vs. limit=22.5
2023-11-21 14:48:19,243 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3250, loss[loss=0.06883, simple_loss=0.08099, pruned_loss=0.01546, audio_tagging_loss=0.01287, over 15598.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09749, pruned_loss=0.01678, audio_tagging_loss=0.009792, over 3060495.55 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:48:20,587 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231700
2023-11-21 14:48:48,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.39 vs. limit=6.0
2023-11-21 14:48:51,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0
2023-11-21 14:49:06,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1544846.6666666667, ans=0.2
2023-11-21 14:49:10,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1544913.3333333333, ans=0.1
2023-11-21 14:49:14,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1544913.3333333333, ans=0.125
2023-11-21 14:49:19,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1544913.3333333333, ans=0.2
2023-11-21 14:49:21,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1544980.0, ans=0.125
2023-11-21 14:49:22,565 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3300, loss[loss=0.07702, simple_loss=0.09537, pruned_loss=0.02065, audio_tagging_loss=0.008688, over 13906.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.09656, pruned_loss=0.01677, audio_tagging_loss=0.009875, over 3055150.03 frames. ], batch size: 54, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:49:23,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231750
2023-11-21 14:49:30,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1544980.0, ans=0.125
2023-11-21 14:49:30,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1544980.0, ans=0.125
2023-11-21 14:49:31,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1544980.0, ans=0.1
2023-11-21 14:49:34,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1545046.6666666667, ans=0.125
2023-11-21 14:49:40,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1545046.6666666667, ans=0.2
2023-11-21 14:49:42,194 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.36 vs. limit=15.0
2023-11-21 14:49:43,229 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.162e+01 8.724e+01 9.354e+01 1.405e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 14:49:51,815 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:50:00,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.89 vs. limit=15.0
2023-11-21 14:50:26,009 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3350, loss[loss=0.08578, simple_loss=0.1079, pruned_loss=0.02061, audio_tagging_loss=0.01121, over 15864.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09752, pruned_loss=0.0169, audio_tagging_loss=0.009695, over 3064395.32 frames. ], batch size: 62, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:50:27,342 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231800
2023-11-21 14:50:31,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1545313.3333333333, ans=0.0
2023-11-21 14:50:42,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1545380.0, ans=0.1
2023-11-21 14:51:12,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1545513.3333333333, ans=0.125
2023-11-21 14:51:30,456 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3400, loss[loss=0.09081, simple_loss=0.1207, pruned_loss=0.02301, audio_tagging_loss=0.007451, over 14549.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.0973, pruned_loss=0.01693, audio_tagging_loss=0.009566, over 3056848.72 frames. ], batch size: 53, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:51:30,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1545646.6666666667, ans=0.1
2023-11-21 14:51:31,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231850
2023-11-21 14:51:33,406 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=22.5
2023-11-21 14:51:48,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1545713.3333333333, ans=0.1
2023-11-21 14:51:49,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.184e+01 8.749e+01 9.223e+01 1.235e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 14:51:57,829 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.64 vs. limit=22.5
2023-11-21 14:52:10,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1545846.6666666667, ans=0.125
2023-11-21 14:52:19,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1545846.6666666667, ans=0.125
2023-11-21 14:52:22,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1545913.3333333333, ans=0.125
2023-11-21 14:52:30,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1545913.3333333333, ans=0.125
2023-11-21 14:52:33,412 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3450, loss[loss=0.06751, simple_loss=0.08007, pruned_loss=0.01696, audio_tagging_loss=0.01051, over 15439.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09705, pruned_loss=0.01686, audio_tagging_loss=0.009485, over 3059229.39 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:52:34,790 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231900
2023-11-21 14:53:00,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1546113.3333333333, ans=0.125
2023-11-21 14:53:03,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1546113.3333333333, ans=0.125
2023-11-21 14:53:21,624 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.33 vs. limit=5.0
2023-11-21 14:53:36,600 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3500, loss[loss=0.05876, simple_loss=0.07757, pruned_loss=0.01033, audio_tagging_loss=0.009644, over 14557.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09722, pruned_loss=0.01672, audio_tagging_loss=0.009403, over 3052229.83 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 14:53:38,525 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 231950
2023-11-21 14:53:55,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1546380.0, ans=0.125
2023-11-21 14:53:59,559 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.228e+01 8.776e+01 9.674e+01 1.230e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 14:54:08,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1546446.6666666667, ans=0.125
2023-11-21 14:54:10,763 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:54:23,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1546513.3333333333, ans=0.0
2023-11-21 14:54:37,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1546580.0, ans=0.1
2023-11-21 14:54:41,976 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3550, loss[loss=0.09403, simple_loss=0.1405, pruned_loss=0.01704, audio_tagging_loss=0.006725, over 16288.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09736, pruned_loss=0.01682, audio_tagging_loss=0.009418, over 3052654.51 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 14:54:43,377 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232000
2023-11-21 14:54:50,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1546646.6666666667, ans=0.2
2023-11-21 14:54:59,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1546713.3333333333, ans=0.1
2023-11-21 14:55:24,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1546846.6666666667, ans=0.125
2023-11-21 14:55:43,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1546913.3333333333, ans=0.1
2023-11-21 14:55:49,455 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3600, loss[loss=0.06667, simple_loss=0.0884, pruned_loss=0.01263, audio_tagging_loss=0.009841, over 15034.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09581, pruned_loss=0.01663, audio_tagging_loss=0.00936, over 3052612.63 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:55:50,723 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232050
2023-11-21 14:55:54,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1546980.0, ans=0.125
2023-11-21 14:56:10,953 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.980e+01 8.692e+01 9.366e+01 1.171e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 14:56:50,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.23 vs. limit=15.0
2023-11-21 14:56:50,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1547246.6666666667, ans=0.1
2023-11-21 14:56:53,216 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3650, loss[loss=0.06978, simple_loss=0.09179, pruned_loss=0.01456, audio_tagging_loss=0.009322, over 14972.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09634, pruned_loss=0.0167, audio_tagging_loss=0.009378, over 3054418.62 frames. ], batch size: 55, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:56:54,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232100
2023-11-21 14:57:30,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1547446.6666666667, ans=0.0
2023-11-21 14:57:33,193 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.67 vs. limit=22.5
2023-11-21 14:57:35,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1547513.3333333333, ans=0.0
2023-11-21 14:57:35,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=23.05 vs. limit=22.5
2023-11-21 14:57:42,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1547513.3333333333, ans=0.0
2023-11-21 14:57:58,552 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3700, loss[loss=0.06209, simple_loss=0.08601, pruned_loss=0.01132, audio_tagging_loss=0.007766, over 16589.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09572, pruned_loss=0.01646, audio_tagging_loss=0.009409, over 3051399.25 frames. ], batch size: 62, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:57:59,865 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232150
2023-11-21 14:58:12,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1547713.3333333333, ans=0.0
2023-11-21 14:58:20,224 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.897e+01 8.102e+01 8.765e+01 9.489e+01 1.317e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 14:58:29,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1547780.0, ans=0.0
2023-11-21 14:58:37,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1547846.6666666667, ans=0.2
2023-11-21 14:58:38,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1547846.6666666667, ans=0.0
2023-11-21 14:58:38,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.45 vs. limit=10.0
2023-11-21 14:58:42,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1547846.6666666667, ans=0.0
2023-11-21 14:59:02,965 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3750, loss[loss=0.09551, simple_loss=0.1291, pruned_loss=0.02374, audio_tagging_loss=0.007229, over 16157.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09643, pruned_loss=0.01666, audio_tagging_loss=0.009385, over 3064185.50 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:59:04,280 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232200
2023-11-21 14:59:05,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1547980.0, ans=0.125
2023-11-21 14:59:08,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1547980.0, ans=0.125
2023-11-21 14:59:17,530 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.40 vs. limit=15.0
2023-11-21 14:59:18,693 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.00 vs. limit=15.0
2023-11-21 14:59:27,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1548113.3333333333, ans=0.125
2023-11-21 14:59:41,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.26 vs. limit=22.5
2023-11-21 14:59:46,989 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:59:59,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1548246.6666666667, ans=0.0
2023-11-21 15:00:06,918 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3800, loss[loss=0.09358, simple_loss=0.1139, pruned_loss=0.02564, audio_tagging_loss=0.011, over 15498.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09681, pruned_loss=0.01667, audio_tagging_loss=0.009319, over 3068352.08 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:00:07,612 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.57 vs. limit=10.0
2023-11-21 15:00:08,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232250
2023-11-21 15:00:29,252 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.346e+01 8.333e+01 9.130e+01 9.759e+01 1.207e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-21 15:00:34,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1548446.6666666667, ans=0.09899494936611666
2023-11-21 15:00:49,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1548513.3333333333, ans=0.125
2023-11-21 15:01:11,471 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3850, loss[loss=0.06819, simple_loss=0.08824, pruned_loss=0.01359, audio_tagging_loss=0.01048, over 16649.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09649, pruned_loss=0.01647, audio_tagging_loss=0.009414, over 3066473.17 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:01:12,792 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232300
2023-11-21 15:02:15,385 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3900, loss[loss=0.07149, simple_loss=0.095, pruned_loss=0.013, audio_tagging_loss=0.01099, over 16117.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09701, pruned_loss=0.01641, audio_tagging_loss=0.009474, over 3056742.37 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:02:16,683 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232350
2023-11-21 15:02:30,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1549046.6666666667, ans=0.2
2023-11-21 15:02:35,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1549046.6666666667, ans=0.0
2023-11-21 15:02:36,405 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.138e+01 8.613e+01 9.443e+01 1.181e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 15:02:45,006 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.21 vs. limit=6.0
2023-11-21 15:02:48,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1549113.3333333333, ans=0.0
2023-11-21 15:02:53,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1549180.0, ans=0.125
2023-11-21 15:02:57,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1549180.0, ans=0.1
2023-11-21 15:03:02,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1549180.0, ans=0.2
2023-11-21 15:03:19,032 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 3950, loss[loss=0.07476, simple_loss=0.1005, pruned_loss=0.01588, audio_tagging_loss=0.008607, over 15142.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09634, pruned_loss=0.01648, audio_tagging_loss=0.009612, over 3051784.10 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:03:20,365 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232400
2023-11-21 15:04:06,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1549513.3333333333, ans=0.1
2023-11-21 15:04:09,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1549580.0, ans=0.2
2023-11-21 15:04:21,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1549580.0, ans=0.07
2023-11-21 15:04:23,704 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4000, loss[loss=0.08062, simple_loss=0.1138, pruned_loss=0.01638, audio_tagging_loss=0.007354, over 14600.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09633, pruned_loss=0.01662, audio_tagging_loss=0.009684, over 3045451.08 frames. ], batch size: 54, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:04:24,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1549646.6666666667, ans=6.0
2023-11-21 15:04:25,040 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232450
2023-11-21 15:04:39,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1549713.3333333333, ans=0.125
2023-11-21 15:04:45,498 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.188e+01 8.912e+01 9.535e+01 1.152e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 15:04:47,292 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.97 vs. limit=15.0
2023-11-21 15:05:28,173 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4050, loss[loss=0.08166, simple_loss=0.1118, pruned_loss=0.01735, audio_tagging_loss=0.008406, over 14818.00 frames. ], tot_loss[loss=0.07461, simple_loss=0.09661, pruned_loss=0.01665, audio_tagging_loss=0.009655, over 3048157.94 frames. ], batch size: 55, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:05:29,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232500
2023-11-21 15:05:30,622 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:05:34,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1549980.0, ans=0.125
2023-11-21 15:05:41,630 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.48 vs. limit=22.5
2023-11-21 15:06:02,908 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.14 vs. limit=10.0
2023-11-21 15:06:32,480 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4100, loss[loss=0.0728, simple_loss=0.08914, pruned_loss=0.01837, audio_tagging_loss=0.009866, over 14367.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09687, pruned_loss=0.01655, audio_tagging_loss=0.009677, over 3042658.29 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:06:33,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232550
2023-11-21 15:06:46,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1550380.0, ans=0.1
2023-11-21 15:06:55,661 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.215e+01 8.893e+01 9.542e+01 1.222e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 15:06:57,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1550446.6666666667, ans=0.1
2023-11-21 15:07:04,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1550446.6666666667, ans=0.0
2023-11-21 15:07:20,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.88 vs. limit=15.0
2023-11-21 15:07:25,416 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0
2023-11-21 15:07:36,363 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4150, loss[loss=0.08177, simple_loss=0.1099, pruned_loss=0.01852, audio_tagging_loss=0.008297, over 16148.00 frames. ], tot_loss[loss=0.07433, simple_loss=0.09633, pruned_loss=0.01651, audio_tagging_loss=0.00966, over 3048897.35 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:07:37,679 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232600
2023-11-21 15:07:41,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1550646.6666666667, ans=0.125
2023-11-21 15:07:58,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1550713.3333333333, ans=0.125
2023-11-21 15:08:09,286 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.27 vs. limit=6.0
2023-11-21 15:08:11,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1550780.0, ans=0.1
2023-11-21 15:08:22,760 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:08:25,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1550846.6666666667, ans=0.0
2023-11-21 15:08:30,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1550913.3333333333, ans=0.125
2023-11-21 15:08:37,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1550913.3333333333, ans=0.125
2023-11-21 15:08:41,867 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4200, loss[loss=0.07269, simple_loss=0.09372, pruned_loss=0.01618, audio_tagging_loss=0.009651, over 15808.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09588, pruned_loss=0.01643, audio_tagging_loss=0.009582, over 3052201.78 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:08:43,208 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232650
2023-11-21 15:08:54,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.85 vs. limit=15.0
2023-11-21 15:08:56,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1551046.6666666667, ans=0.125
2023-11-21 15:08:59,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1551046.6666666667, ans=0.0
2023-11-21 15:09:03,667 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.544e+01 8.218e+01 8.944e+01 9.900e+01 1.172e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 15:09:24,728 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.90 vs. limit=6.0
2023-11-21 15:09:37,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1551246.6666666667, ans=0.1
2023-11-21 15:09:40,026 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.35 vs. limit=15.0
2023-11-21 15:09:45,352 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4250, loss[loss=0.0808, simple_loss=0.1109, pruned_loss=0.01749, audio_tagging_loss=0.007877, over 15289.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09655, pruned_loss=0.01656, audio_tagging_loss=0.009503, over 3049056.00 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:09:46,706 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232700
2023-11-21 15:10:11,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1551446.6666666667, ans=0.05
2023-11-21 15:10:18,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1551446.6666666667, ans=0.125
2023-11-21 15:10:28,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1551513.3333333333, ans=0.125
2023-11-21 15:10:49,917 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4300, loss[loss=0.05753, simple_loss=0.07556, pruned_loss=0.009553, audio_tagging_loss=0.0102, over 14495.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09736, pruned_loss=0.01669, audio_tagging_loss=0.009353, over 3046021.58 frames. ], batch size: 54, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:10:50,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1551646.6666666667, ans=0.0
2023-11-21 15:10:51,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232750
2023-11-21 15:11:13,428 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.861e+01 8.351e+01 9.107e+01 1.007e+02 1.335e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-21 15:11:18,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1551780.0, ans=0.125
2023-11-21 15:11:45,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1551913.3333333333, ans=0.125
2023-11-21 15:11:54,992 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4350, loss[loss=0.06839, simple_loss=0.08404, pruned_loss=0.01634, audio_tagging_loss=0.01003, over 14825.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.0966, pruned_loss=0.01636, audio_tagging_loss=0.009385, over 3040984.79 frames. ], batch size: 53, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:11:56,301 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232800
2023-11-21 15:11:57,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1551980.0, ans=0.125
2023-11-21 15:12:05,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1551980.0, ans=0.2
2023-11-21 15:12:05,834 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.88 vs. limit=15.0
2023-11-21 15:12:19,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1552113.3333333333, ans=0.125
2023-11-21 15:12:22,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1552113.3333333333, ans=0.125
2023-11-21 15:12:30,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1552113.3333333333, ans=0.125
2023-11-21 15:12:33,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1552180.0, ans=0.125
2023-11-21 15:12:50,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1552246.6666666667, ans=0.0
2023-11-21 15:12:55,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1552246.6666666667, ans=0.1
2023-11-21 15:12:58,353 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4400, loss[loss=0.05814, simple_loss=0.07717, pruned_loss=0.01118, audio_tagging_loss=0.008383, over 13676.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09708, pruned_loss=0.01651, audio_tagging_loss=0.009358, over 3038937.82 frames. ], batch size: 52, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:12:59,772 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232850
2023-11-21 15:13:03,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1552313.3333333333, ans=0.125
2023-11-21 15:13:04,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1552313.3333333333, ans=0.0
2023-11-21 15:13:06,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1552313.3333333333, ans=0.2
2023-11-21 15:13:12,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1552380.0, ans=0.07
2023-11-21 15:13:20,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1552380.0, ans=0.125
2023-11-21 15:13:20,843 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.302e+01 8.149e+01 8.654e+01 9.425e+01 1.207e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 15:13:36,034 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.59 vs. limit=6.0
2023-11-21 15:14:01,269 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4450, loss[loss=0.08464, simple_loss=0.122, pruned_loss=0.01532, audio_tagging_loss=0.008341, over 16171.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09669, pruned_loss=0.01658, audio_tagging_loss=0.009335, over 3045911.46 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:14:01,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1552646.6666666667, ans=0.1
2023-11-21 15:14:02,511 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232900
2023-11-21 15:14:08,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1552646.6666666667, ans=0.0
2023-11-21 15:14:45,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.36 vs. limit=10.0
2023-11-21 15:14:55,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1552913.3333333333, ans=0.125
2023-11-21 15:15:00,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.53 vs. limit=22.5
2023-11-21 15:15:04,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1552913.3333333333, ans=0.2
2023-11-21 15:15:06,410 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4500, loss[loss=0.08201, simple_loss=0.1045, pruned_loss=0.01954, audio_tagging_loss=0.01019, over 13545.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09652, pruned_loss=0.01643, audio_tagging_loss=0.009328, over 3047007.81 frames. ], batch size: 53, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:15:07,746 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 232950
2023-11-21 15:15:24,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1553046.6666666667, ans=0.125
2023-11-21 15:15:30,003 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.048e+01 8.795e+01 9.628e+01 1.621e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 15:15:30,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1553113.3333333333, ans=0.125
2023-11-21 15:15:34,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1553113.3333333333, ans=0.125
2023-11-21 15:15:37,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1553113.3333333333, ans=0.1
2023-11-21 15:15:44,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1553180.0, ans=0.2
2023-11-21 15:15:49,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1553180.0, ans=0.125
2023-11-21 15:15:59,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1553246.6666666667, ans=0.125
2023-11-21 15:15:59,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1553246.6666666667, ans=0.2
2023-11-21 15:16:02,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1553246.6666666667, ans=0.125
2023-11-21 15:16:04,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1553246.6666666667, ans=0.0
2023-11-21 15:16:10,797 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4550, loss[loss=0.06543, simple_loss=0.08566, pruned_loss=0.01367, audio_tagging_loss=0.008932, over 15281.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09659, pruned_loss=0.01651, audio_tagging_loss=0.00931, over 3038389.16 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:16:12,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233000
2023-11-21 15:16:19,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1553313.3333333333, ans=0.125
2023-11-21 15:16:30,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1553380.0, ans=0.125
2023-11-21 15:16:46,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1553446.6666666667, ans=0.125
2023-11-21 15:16:56,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1553513.3333333333, ans=0.125
2023-11-21 15:17:00,258 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:17:00,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1553513.3333333333, ans=10.0
2023-11-21 15:17:01,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1553580.0, ans=0.1
2023-11-21 15:17:15,283 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4600, loss[loss=0.04598, simple_loss=0.05688, pruned_loss=0.006234, audio_tagging_loss=0.01131, over 14256.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09588, pruned_loss=0.01633, audio_tagging_loss=0.009396, over 3036971.89 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:17:16,658 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233050
2023-11-21 15:17:40,765 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.142e+01 8.759e+01 9.508e+01 1.262e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 15:17:47,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1553780.0, ans=0.125
2023-11-21 15:18:04,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1553846.6666666667, ans=0.125
2023-11-21 15:18:14,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1553913.3333333333, ans=0.125
2023-11-21 15:18:21,087 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4650, loss[loss=0.0801, simple_loss=0.09374, pruned_loss=0.02306, audio_tagging_loss=0.01017, over 15392.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09591, pruned_loss=0.01641, audio_tagging_loss=0.009512, over 3033867.00 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:18:22,991 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233100
2023-11-21 15:18:25,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.43 vs. limit=5.0
2023-11-21 15:18:32,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1553980.0, ans=0.1
2023-11-21 15:18:50,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1554113.3333333333, ans=0.0
2023-11-21 15:19:13,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1554246.6666666667, ans=0.0
2023-11-21 15:19:26,205 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4700, loss[loss=0.0638, simple_loss=0.0783, pruned_loss=0.01483, audio_tagging_loss=0.009826, over 14967.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09553, pruned_loss=0.01632, audio_tagging_loss=0.00963, over 3031770.50 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:19:27,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233150
2023-11-21 15:19:49,363 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.239e+01 8.978e+01 9.572e+01 1.151e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 15:19:54,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1554446.6666666667, ans=0.1
2023-11-21 15:19:58,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1554446.6666666667, ans=0.0
2023-11-21 15:19:59,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1554446.6666666667, ans=0.125
2023-11-21 15:20:24,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1554580.0, ans=0.0
2023-11-21 15:20:26,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1554580.0, ans=0.0
2023-11-21 15:20:29,867 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4750, loss[loss=0.07676, simple_loss=0.1004, pruned_loss=0.01626, audio_tagging_loss=0.01028, over 15076.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09515, pruned_loss=0.01617, audio_tagging_loss=0.009697, over 3030046.14 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:20:31,184 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233200
2023-11-21 15:20:44,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1554713.3333333333, ans=0.2
2023-11-21 15:20:45,803 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:20:59,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=15.0
2023-11-21 15:21:10,488 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.51 vs. limit=15.0
2023-11-21 15:21:11,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1554846.6666666667, ans=0.2
2023-11-21 15:21:34,022 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4800, loss[loss=0.07972, simple_loss=0.1011, pruned_loss=0.01752, audio_tagging_loss=0.01164, over 15445.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.0958, pruned_loss=0.0163, audio_tagging_loss=0.009803, over 3033085.08 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:21:35,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233250
2023-11-21 15:21:59,668 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.234e+01 8.749e+01 9.624e+01 1.145e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 15:22:02,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1555113.3333333333, ans=0.1
2023-11-21 15:22:26,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1555246.6666666667, ans=0.125
2023-11-21 15:22:31,274 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.49 vs. limit=15.0
2023-11-21 15:22:40,336 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4850, loss[loss=0.06978, simple_loss=0.09812, pruned_loss=0.01467, audio_tagging_loss=0.006053, over 15293.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09609, pruned_loss=0.01639, audio_tagging_loss=0.009879, over 3041264.50 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:22:41,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233300
2023-11-21 15:22:53,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1555380.0, ans=0.0
2023-11-21 15:23:37,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1555580.0, ans=0.125
2023-11-21 15:23:43,894 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4900, loss[loss=0.08623, simple_loss=0.1143, pruned_loss=0.01915, audio_tagging_loss=0.009932, over 15700.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09706, pruned_loss=0.01663, audio_tagging_loss=0.009819, over 3042145.19 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:23:45,297 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233350
2023-11-21 15:23:49,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1555646.6666666667, ans=0.125
2023-11-21 15:23:55,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1555713.3333333333, ans=0.125
2023-11-21 15:24:03,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1555713.3333333333, ans=0.125
2023-11-21 15:24:08,601 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.805e+01 8.037e+01 8.614e+01 9.424e+01 1.262e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 15:24:10,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.97 vs. limit=15.0
2023-11-21 15:24:16,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1555780.0, ans=0.2
2023-11-21 15:24:31,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1555846.6666666667, ans=0.1
2023-11-21 15:24:37,327 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.53 vs. limit=12.0
2023-11-21 15:24:42,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.40 vs. limit=15.0
2023-11-21 15:24:47,983 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 4950, loss[loss=0.09247, simple_loss=0.1162, pruned_loss=0.02471, audio_tagging_loss=0.009674, over 16059.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.09674, pruned_loss=0.01653, audio_tagging_loss=0.009612, over 3037818.36 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:24:48,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1555980.0, ans=0.125
2023-11-21 15:24:48,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.85 vs. limit=15.0
2023-11-21 15:24:49,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233400
2023-11-21 15:25:04,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1556046.6666666667, ans=0.125
2023-11-21 15:25:07,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.12 vs. limit=6.0
2023-11-21 15:25:11,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1556046.6666666667, ans=0.125
2023-11-21 15:25:13,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1556113.3333333333, ans=0.0
2023-11-21 15:25:28,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1556180.0, ans=0.0
2023-11-21 15:25:43,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.96 vs. limit=15.0
2023-11-21 15:25:43,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.52 vs. limit=22.5
2023-11-21 15:25:53,388 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5000, loss[loss=0.07916, simple_loss=0.09808, pruned_loss=0.01912, audio_tagging_loss=0.011, over 15087.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09576, pruned_loss=0.01637, audio_tagging_loss=0.009448, over 3032760.70 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:25:54,613 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233450
2023-11-21 15:25:58,770 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.70 vs. limit=15.0
2023-11-21 15:26:03,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-21 15:26:17,059 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.408e+01 8.018e+01 8.729e+01 9.400e+01 1.250e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 15:26:44,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1556580.0, ans=0.125
2023-11-21 15:26:57,634 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5050, loss[loss=0.08148, simple_loss=0.1069, pruned_loss=0.01682, audio_tagging_loss=0.01123, over 15848.00 frames. ], tot_loss[loss=0.07411, simple_loss=0.09659, pruned_loss=0.01651, audio_tagging_loss=0.009304, over 3038844.33 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:26:58,932 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233500
2023-11-21 15:27:04,363 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.69 vs. limit=22.5
2023-11-21 15:27:10,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1556713.3333333333, ans=0.1
2023-11-21 15:27:21,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.74 vs. limit=15.0
2023-11-21 15:27:22,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1556780.0, ans=0.125
2023-11-21 15:27:34,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1556780.0, ans=0.125
2023-11-21 15:27:36,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1556846.6666666667, ans=0.125
2023-11-21 15:27:48,376 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:27:58,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.81 vs. limit=12.0
2023-11-21 15:28:01,659 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5100, loss[loss=0.08237, simple_loss=0.103, pruned_loss=0.01867, audio_tagging_loss=0.0122, over 16132.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09649, pruned_loss=0.0164, audio_tagging_loss=0.009306, over 3050493.01 frames. ], batch size: 61, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:28:02,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233550
2023-11-21 15:28:26,725 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.860e+01 8.610e+01 9.200e+01 1.229e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 15:28:28,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1557113.3333333333, ans=0.05
2023-11-21 15:28:31,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.13 vs. limit=15.0
2023-11-21 15:28:33,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1557113.3333333333, ans=0.0
2023-11-21 15:28:36,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1557113.3333333333, ans=0.09899494936611666
2023-11-21 15:28:37,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1557113.3333333333, ans=0.125
2023-11-21 15:28:40,230 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.20 vs. limit=12.0
2023-11-21 15:28:44,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1557180.0, ans=0.1
2023-11-21 15:28:52,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1557246.6666666667, ans=0.125
2023-11-21 15:28:59,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1557246.6666666667, ans=0.125
2023-11-21 15:29:00,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1557246.6666666667, ans=0.125
2023-11-21 15:29:01,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1557246.6666666667, ans=0.125
2023-11-21 15:29:06,992 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5150, loss[loss=0.07512, simple_loss=0.1024, pruned_loss=0.01482, audio_tagging_loss=0.009107, over 16170.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09576, pruned_loss=0.01623, audio_tagging_loss=0.009342, over 3042228.68 frames. ], batch size: 59, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:29:08,272 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233600
2023-11-21 15:29:34,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1557446.6666666667, ans=0.125
2023-11-21 15:29:46,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1557513.3333333333, ans=0.125
2023-11-21 15:29:48,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1557513.3333333333, ans=0.125
2023-11-21 15:29:51,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1557513.3333333333, ans=0.07
2023-11-21 15:30:11,325 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5200, loss[loss=0.05756, simple_loss=0.07439, pruned_loss=0.01176, audio_tagging_loss=0.00861, over 14270.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09608, pruned_loss=0.01629, audio_tagging_loss=0.009349, over 3034388.28 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:30:12,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233650
2023-11-21 15:30:35,766 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.165e+01 8.844e+01 9.636e+01 1.346e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 15:30:44,876 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:30:55,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1557846.6666666667, ans=0.125
2023-11-21 15:31:15,861 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5250, loss[loss=0.08555, simple_loss=0.1116, pruned_loss=0.02166, audio_tagging_loss=0.008104, over 15637.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09651, pruned_loss=0.01634, audio_tagging_loss=0.00925, over 3043825.40 frames. ], batch size: 60, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:31:17,265 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233700
2023-11-21 15:31:19,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1557980.0, ans=0.1
2023-11-21 15:31:29,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1558046.6666666667, ans=0.0
2023-11-21 15:31:30,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0
2023-11-21 15:31:34,045 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.848e-02
2023-11-21 15:31:46,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.93 vs. limit=15.0
2023-11-21 15:31:53,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1558180.0, ans=0.125
2023-11-21 15:32:03,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1558180.0, ans=0.035
2023-11-21 15:32:04,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1558180.0, ans=0.04949747468305833
2023-11-21 15:32:08,321 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-21 15:32:10,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1558246.6666666667, ans=0.0
2023-11-21 15:32:21,266 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5300, loss[loss=0.06595, simple_loss=0.08416, pruned_loss=0.01605, audio_tagging_loss=0.007824, over 14878.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09731, pruned_loss=0.01675, audio_tagging_loss=0.009263, over 3039924.95 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:32:22,554 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233750
2023-11-21 15:32:45,203 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.789e+01 8.165e+01 8.716e+01 9.457e+01 1.108e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 15:32:58,827 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.08 vs. limit=10.0
2023-11-21 15:33:25,896 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5350, loss[loss=0.05845, simple_loss=0.07202, pruned_loss=0.01186, audio_tagging_loss=0.01058, over 15254.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09713, pruned_loss=0.01662, audio_tagging_loss=0.009265, over 3045101.51 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:33:27,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233800
2023-11-21 15:33:37,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1558713.3333333333, ans=0.0
2023-11-21 15:33:37,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1558713.3333333333, ans=0.125
2023-11-21 15:33:44,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.96 vs. limit=6.0
2023-11-21 15:34:03,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1558780.0, ans=0.0
2023-11-21 15:34:10,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1558846.6666666667, ans=0.0
2023-11-21 15:34:16,170 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.50 vs. limit=15.0
2023-11-21 15:34:30,877 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5400, loss[loss=0.08677, simple_loss=0.12, pruned_loss=0.01953, audio_tagging_loss=0.007237, over 15343.00 frames. ], tot_loss[loss=0.07468, simple_loss=0.09728, pruned_loss=0.01667, audio_tagging_loss=0.009364, over 3041836.99 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:34:32,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233850
2023-11-21 15:34:38,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1558980.0, ans=0.1
2023-11-21 15:34:47,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1559046.6666666667, ans=0.0
2023-11-21 15:34:55,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1559046.6666666667, ans=0.125
2023-11-21 15:34:57,301 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.221e+01 8.112e+01 8.686e+01 9.399e+01 1.125e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 15:34:58,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1559113.3333333333, ans=0.1
2023-11-21 15:35:02,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1559113.3333333333, ans=0.125
2023-11-21 15:35:10,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1559180.0, ans=0.125
2023-11-21 15:35:17,586 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:35:24,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1559246.6666666667, ans=0.0
2023-11-21 15:35:32,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.60 vs. limit=15.0
2023-11-21 15:35:36,483 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5450, loss[loss=0.1025, simple_loss=0.1326, pruned_loss=0.0264, audio_tagging_loss=0.009795, over 15350.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09834, pruned_loss=0.01709, audio_tagging_loss=0.009375, over 3050592.72 frames. ], batch size: 54, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:35:37,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233900
2023-11-21 15:35:42,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1559313.3333333333, ans=0.0
2023-11-21 15:35:50,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1559380.0, ans=0.0
2023-11-21 15:36:40,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.10 vs. limit=15.0
2023-11-21 15:36:40,854 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5500, loss[loss=0.08214, simple_loss=0.1051, pruned_loss=0.01721, audio_tagging_loss=0.01238, over 14753.00 frames. ], tot_loss[loss=0.07537, simple_loss=0.09822, pruned_loss=0.01683, audio_tagging_loss=0.009428, over 3055968.72 frames. ], batch size: 53, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:36:42,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 233950
2023-11-21 15:37:03,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1559713.3333333333, ans=0.125
2023-11-21 15:37:05,851 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.151e+01 8.405e+01 9.190e+01 9.926e+01 1.565e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-21 15:37:32,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1559913.3333333333, ans=0.04949747468305833
2023-11-21 15:37:42,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1559913.3333333333, ans=0.0
2023-11-21 15:37:44,555 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5550, loss[loss=0.06053, simple_loss=0.08005, pruned_loss=0.01015, audio_tagging_loss=0.01036, over 15574.00 frames. ], tot_loss[loss=0.07564, simple_loss=0.09845, pruned_loss=0.01695, audio_tagging_loss=0.009464, over 3051299.36 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:37:45,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234000
2023-11-21 15:37:51,843 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.30 vs. limit=22.5
2023-11-21 15:37:57,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1560046.6666666667, ans=0.1
2023-11-21 15:37:58,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1560046.6666666667, ans=0.0
2023-11-21 15:38:11,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1560113.3333333333, ans=0.125
2023-11-21 15:38:15,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1560113.3333333333, ans=0.125
2023-11-21 15:38:21,837 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.36 vs. limit=15.0
2023-11-21 15:38:34,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1560180.0, ans=0.125
2023-11-21 15:38:35,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=15.0
2023-11-21 15:38:51,268 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5600, loss[loss=0.08131, simple_loss=0.1074, pruned_loss=0.01727, audio_tagging_loss=0.01033, over 15144.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09835, pruned_loss=0.01683, audio_tagging_loss=0.009591, over 3048465.76 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:38:51,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1560313.3333333333, ans=0.125
2023-11-21 15:38:52,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234050
2023-11-21 15:38:56,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1560313.3333333333, ans=0.0
2023-11-21 15:39:04,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1560380.0, ans=0.125
2023-11-21 15:39:16,696 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 7.943e+01 8.570e+01 9.599e+01 1.261e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 15:39:23,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1560446.6666666667, ans=0.125
2023-11-21 15:39:29,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1560513.3333333333, ans=0.125
2023-11-21 15:39:30,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1560513.3333333333, ans=0.2
2023-11-21 15:39:37,673 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:39:48,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1560580.0, ans=0.125
2023-11-21 15:39:53,492 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.33 vs. limit=15.0
2023-11-21 15:39:56,436 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5650, loss[loss=0.06117, simple_loss=0.07378, pruned_loss=0.01431, audio_tagging_loss=0.009966, over 14942.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09758, pruned_loss=0.01677, audio_tagging_loss=0.009783, over 3044393.56 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:39:57,800 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234100
2023-11-21 15:40:09,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1560713.3333333333, ans=0.07
2023-11-21 15:40:19,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1560713.3333333333, ans=0.1
2023-11-21 15:40:22,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.79 vs. limit=15.0
2023-11-21 15:40:24,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff2.min_abs, batch_count=1560780.0, ans=0.1
2023-11-21 15:40:27,875 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.99 vs. limit=15.0
2023-11-21 15:40:30,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1560780.0, ans=0.1
2023-11-21 15:40:38,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1560846.6666666667, ans=0.2
2023-11-21 15:40:56,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1560913.3333333333, ans=0.1
2023-11-21 15:41:00,849 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5700, loss[loss=0.08157, simple_loss=0.1096, pruned_loss=0.01919, audio_tagging_loss=0.007561, over 15090.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09673, pruned_loss=0.01669, audio_tagging_loss=0.009823, over 3034838.82 frames. ], batch size: 54, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:41:02,206 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234150
2023-11-21 15:41:08,910 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.22 vs. limit=22.5
2023-11-21 15:41:11,438 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.31 vs. limit=15.0
2023-11-21 15:41:20,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1561046.6666666667, ans=0.125
2023-11-21 15:41:21,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1561046.6666666667, ans=0.2
2023-11-21 15:41:23,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1561046.6666666667, ans=0.0
2023-11-21 15:41:27,209 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.266e+01 8.051e+01 8.513e+01 9.127e+01 1.242e+02, threshold=1.703e+02, percent-clipped=0.0
2023-11-21 15:41:29,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1561113.3333333333, ans=0.0
2023-11-21 15:41:32,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.13 vs. limit=15.0
2023-11-21 15:41:49,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1561180.0, ans=0.0
2023-11-21 15:41:55,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1561246.6666666667, ans=0.0
2023-11-21 15:42:03,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1561246.6666666667, ans=0.1
2023-11-21 15:42:05,386 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5750, loss[loss=0.08584, simple_loss=0.1118, pruned_loss=0.02286, audio_tagging_loss=0.007081, over 15241.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09758, pruned_loss=0.01685, audio_tagging_loss=0.009655, over 3040880.74 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:42:06,715 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234200
2023-11-21 15:42:36,111 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:42:55,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1561513.3333333333, ans=0.0
2023-11-21 15:42:56,770 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.05 vs. limit=22.5
2023-11-21 15:43:07,589 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.70 vs. limit=15.0
2023-11-21 15:43:11,824 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5800, loss[loss=0.06643, simple_loss=0.08615, pruned_loss=0.01095, audio_tagging_loss=0.0124, over 14899.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.09672, pruned_loss=0.01653, audio_tagging_loss=0.009627, over 3039818.15 frames. ], batch size: 54, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:43:13,155 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234250
2023-11-21 15:43:32,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1561713.3333333333, ans=0.125
2023-11-21 15:43:36,360 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.121e+01 8.819e+01 9.541e+01 1.299e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 15:43:38,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1561780.0, ans=0.125
2023-11-21 15:44:09,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1561913.3333333333, ans=0.0
2023-11-21 15:44:16,107 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5850, loss[loss=0.06634, simple_loss=0.07646, pruned_loss=0.01886, audio_tagging_loss=0.009247, over 14822.00 frames. ], tot_loss[loss=0.07398, simple_loss=0.09581, pruned_loss=0.0164, audio_tagging_loss=0.009673, over 3035536.50 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:44:17,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234300
2023-11-21 15:44:35,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1562046.6666666667, ans=0.125
2023-11-21 15:44:38,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1562046.6666666667, ans=0.2
2023-11-21 15:44:46,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1562113.3333333333, ans=0.07
2023-11-21 15:44:49,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=1562113.3333333333, ans=0.025
2023-11-21 15:44:49,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1562113.3333333333, ans=0.125
2023-11-21 15:44:53,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1562113.3333333333, ans=0.125
2023-11-21 15:44:56,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1562180.0, ans=0.125
2023-11-21 15:45:00,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1562180.0, ans=0.125
2023-11-21 15:45:06,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.69 vs. limit=15.0
2023-11-21 15:45:07,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1562246.6666666667, ans=0.125
2023-11-21 15:45:12,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1562246.6666666667, ans=0.125
2023-11-21 15:45:15,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1562246.6666666667, ans=0.125
2023-11-21 15:45:20,801 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5900, loss[loss=0.076, simple_loss=0.1088, pruned_loss=0.01606, audio_tagging_loss=0.005552, over 15173.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09565, pruned_loss=0.01639, audio_tagging_loss=0.00961, over 3036320.32 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:45:22,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234350
2023-11-21 15:45:33,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1562380.0, ans=0.125
2023-11-21 15:45:35,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.76 vs. limit=22.5
2023-11-21 15:45:41,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1562380.0, ans=0.125
2023-11-21 15:45:47,510 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.953e+01 8.033e+01 8.577e+01 9.444e+01 1.216e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 15:46:07,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1562513.3333333333, ans=0.015
2023-11-21 15:46:25,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1562646.6666666667, ans=0.0
2023-11-21 15:46:26,927 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 5950, loss[loss=0.09007, simple_loss=0.1294, pruned_loss=0.0154, audio_tagging_loss=0.009972, over 15678.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.0969, pruned_loss=0.01668, audio_tagging_loss=0.009573, over 3042445.26 frames. ], batch size: 54, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:46:28,269 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234400
2023-11-21 15:46:31,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1562646.6666666667, ans=0.0
2023-11-21 15:46:32,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1562646.6666666667, ans=0.0
2023-11-21 15:47:01,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1562780.0, ans=0.015
2023-11-21 15:47:29,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.40 vs. limit=15.0
2023-11-21 15:47:31,483 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6000, loss[loss=0.07101, simple_loss=0.09447, pruned_loss=0.01514, audio_tagging_loss=0.008635, over 14984.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09687, pruned_loss=0.01675, audio_tagging_loss=0.009545, over 3040157.36 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:47:31,485 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 15:47:54,719 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8247, 4.9108, 5.0622, 4.9143], device='cuda:2')
2023-11-21 15:48:12,906 INFO [train_asr.py:1253] (2/4) Epoch 20, validation: loss=0.06068, simple_loss=0.0522, pruned_loss=0.005214, audio_tagging_loss=0.02937, over 4681554.00 frames. 
2023-11-21 15:48:12,907 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 15:48:14,188 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234450
2023-11-21 15:48:16,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1562980.0, ans=0.0
2023-11-21 15:48:18,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1562980.0, ans=0.0
2023-11-21 15:48:19,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1562980.0, ans=0.125
2023-11-21 15:48:24,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1563046.6666666667, ans=0.5
2023-11-21 15:48:28,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.27 vs. limit=22.5
2023-11-21 15:48:30,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1563046.6666666667, ans=0.1
2023-11-21 15:48:38,429 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 7.928e+01 8.696e+01 9.354e+01 1.094e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 15:48:45,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1563113.3333333333, ans=0.0
2023-11-21 15:48:49,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1563180.0, ans=0.07
2023-11-21 15:48:58,818 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:48:58,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1563180.0, ans=0.125
2023-11-21 15:49:17,911 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6050, loss[loss=0.06091, simple_loss=0.08348, pruned_loss=0.01103, audio_tagging_loss=0.008134, over 15671.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.09645, pruned_loss=0.01646, audio_tagging_loss=0.0095, over 3044176.18 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:49:19,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234500
2023-11-21 15:49:21,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1563313.3333333333, ans=0.1
2023-11-21 15:49:24,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.32 vs. limit=15.0
2023-11-21 15:49:31,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1563380.0, ans=0.5
2023-11-21 15:49:37,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1563380.0, ans=0.09899494936611666
2023-11-21 15:49:43,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1563446.6666666667, ans=0.0
2023-11-21 15:49:44,380 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.32 vs. limit=12.0
2023-11-21 15:49:56,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1563513.3333333333, ans=0.125
2023-11-21 15:50:21,456 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6100, loss[loss=0.07823, simple_loss=0.1106, pruned_loss=0.01536, audio_tagging_loss=0.007575, over 15564.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09674, pruned_loss=0.01658, audio_tagging_loss=0.009435, over 3051956.09 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:50:22,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.91 vs. limit=15.0
2023-11-21 15:50:22,762 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234550
2023-11-21 15:50:22,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1563646.6666666667, ans=0.0
2023-11-21 15:50:47,223 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.060e+01 8.562e+01 9.373e+01 1.321e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 15:50:50,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1563780.0, ans=0.0
2023-11-21 15:50:51,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1563780.0, ans=0.0
2023-11-21 15:51:06,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.67 vs. limit=15.0
2023-11-21 15:51:19,707 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.35 vs. limit=15.0
2023-11-21 15:51:25,607 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6150, loss[loss=0.06724, simple_loss=0.08335, pruned_loss=0.01478, audio_tagging_loss=0.01079, over 14820.00 frames. ], tot_loss[loss=0.07363, simple_loss=0.09568, pruned_loss=0.01629, audio_tagging_loss=0.00951, over 3049946.02 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:51:27,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234600
2023-11-21 15:51:42,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1564046.6666666667, ans=0.125
2023-11-21 15:51:43,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1564046.6666666667, ans=0.125
2023-11-21 15:51:54,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1564113.3333333333, ans=0.125
2023-11-21 15:52:31,547 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6200, loss[loss=0.08282, simple_loss=0.1101, pruned_loss=0.01916, audio_tagging_loss=0.008587, over 15065.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09522, pruned_loss=0.01645, audio_tagging_loss=0.009589, over 3043275.44 frames. ], batch size: 54, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:52:32,803 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234650
2023-11-21 15:52:45,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1564380.0, ans=0.125
2023-11-21 15:52:56,129 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 7.947e+01 8.423e+01 9.173e+01 1.216e+02, threshold=1.685e+02, percent-clipped=0.0
2023-11-21 15:53:01,549 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.25 vs. limit=15.0
2023-11-21 15:53:04,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1564446.6666666667, ans=0.0
2023-11-21 15:53:16,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1564513.3333333333, ans=0.125
2023-11-21 15:53:21,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.62 vs. limit=15.0
2023-11-21 15:53:33,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1564580.0, ans=0.125
2023-11-21 15:53:35,420 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6250, loss[loss=0.07741, simple_loss=0.09977, pruned_loss=0.01974, audio_tagging_loss=0.007782, over 15747.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.09528, pruned_loss=0.01648, audio_tagging_loss=0.009688, over 3043049.26 frames. ], batch size: 61, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:53:36,781 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234700
2023-11-21 15:53:58,151 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.19 vs. limit=6.0
2023-11-21 15:54:05,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1564780.0, ans=0.125
2023-11-21 15:54:19,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1564846.6666666667, ans=0.125
2023-11-21 15:54:28,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1564913.3333333333, ans=0.1
2023-11-21 15:54:39,039 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6300, loss[loss=0.08422, simple_loss=0.1055, pruned_loss=0.02064, audio_tagging_loss=0.01086, over 14980.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09572, pruned_loss=0.01659, audio_tagging_loss=0.009685, over 3038180.50 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:54:40,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234750
2023-11-21 15:54:46,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1564980.0, ans=0.0
2023-11-21 15:54:54,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1565046.6666666667, ans=0.0
2023-11-21 15:55:05,695 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.628e+01 8.110e+01 8.746e+01 9.579e+01 1.135e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 15:55:17,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1565180.0, ans=0.1
2023-11-21 15:55:35,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1565246.6666666667, ans=0.025
2023-11-21 15:55:45,087 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6350, loss[loss=0.06569, simple_loss=0.08624, pruned_loss=0.01322, audio_tagging_loss=0.009348, over 13831.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09544, pruned_loss=0.0166, audio_tagging_loss=0.009742, over 3033124.42 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:55:46,392 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234800
2023-11-21 15:55:47,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1565313.3333333333, ans=0.09899494936611666
2023-11-21 15:55:51,833 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:56:16,372 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.62 vs. limit=15.0
2023-11-21 15:56:17,072 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:56:49,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1565646.6666666667, ans=0.125
2023-11-21 15:56:49,996 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6400, loss[loss=0.07859, simple_loss=0.101, pruned_loss=0.01605, audio_tagging_loss=0.01202, over 15864.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09539, pruned_loss=0.01654, audio_tagging_loss=0.009893, over 3035185.32 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:56:51,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234850
2023-11-21 15:57:04,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1565713.3333333333, ans=0.125
2023-11-21 15:57:06,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1565713.3333333333, ans=0.125
2023-11-21 15:57:16,951 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.183e+01 8.791e+01 9.502e+01 1.187e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 15:57:17,300 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:57:24,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1565780.0, ans=0.125
2023-11-21 15:57:43,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1565913.3333333333, ans=0.125
2023-11-21 15:57:51,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1565913.3333333333, ans=0.125
2023-11-21 15:57:51,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1565913.3333333333, ans=0.2
2023-11-21 15:57:54,340 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:57:55,256 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6450, loss[loss=0.04802, simple_loss=0.0514, pruned_loss=0.008278, audio_tagging_loss=0.01404, over 15031.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09491, pruned_loss=0.01652, audio_tagging_loss=0.01007, over 3034603.25 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:57:55,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1565980.0, ans=0.2
2023-11-21 15:57:56,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234900
2023-11-21 15:58:09,389 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=3.062e-02
2023-11-21 15:58:32,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1566113.3333333333, ans=0.0
2023-11-21 15:58:33,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1566180.0, ans=0.125
2023-11-21 15:58:39,095 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:58:56,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1566246.6666666667, ans=0.05
2023-11-21 15:59:01,490 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6500, loss[loss=0.06892, simple_loss=0.1032, pruned_loss=0.009371, audio_tagging_loss=0.007937, over 15690.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09497, pruned_loss=0.01631, audio_tagging_loss=0.00999, over 3041364.88 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:59:01,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1566313.3333333333, ans=0.0
2023-11-21 15:59:02,858 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 234950
2023-11-21 15:59:08,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1566313.3333333333, ans=0.0
2023-11-21 15:59:12,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1566313.3333333333, ans=0.2
2023-11-21 15:59:16,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1566380.0, ans=0.2
2023-11-21 15:59:26,835 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.005e+01 8.613e+01 9.254e+01 1.234e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 16:00:04,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1566580.0, ans=0.2
2023-11-21 16:00:05,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1566646.6666666667, ans=0.2
2023-11-21 16:00:06,227 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6550, loss[loss=0.06807, simple_loss=0.09268, pruned_loss=0.01265, audio_tagging_loss=0.00908, over 15324.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09592, pruned_loss=0.01637, audio_tagging_loss=0.009761, over 3047569.57 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:00:07,641 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235000
2023-11-21 16:00:07,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1566646.6666666667, ans=0.0
2023-11-21 16:00:16,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1566646.6666666667, ans=0.2
2023-11-21 16:00:54,210 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:01:01,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1566913.3333333333, ans=0.125
2023-11-21 16:01:01,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.33 vs. limit=15.0
2023-11-21 16:01:08,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1566913.3333333333, ans=0.125
2023-11-21 16:01:11,128 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6600, loss[loss=0.08275, simple_loss=0.1038, pruned_loss=0.01766, audio_tagging_loss=0.01318, over 13666.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09645, pruned_loss=0.01656, audio_tagging_loss=0.00958, over 3042960.63 frames. ], batch size: 53, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:01:11,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1566980.0, ans=0.1
2023-11-21 16:01:13,052 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235050
2023-11-21 16:01:26,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1567046.6666666667, ans=0.125
2023-11-21 16:01:39,702 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.039e+01 8.543e+01 9.389e+01 1.376e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-21 16:01:44,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.31 vs. limit=10.0
2023-11-21 16:02:15,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1567246.6666666667, ans=0.1
2023-11-21 16:02:17,542 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6650, loss[loss=0.08559, simple_loss=0.1066, pruned_loss=0.02463, audio_tagging_loss=0.007641, over 14250.00 frames. ], tot_loss[loss=0.07423, simple_loss=0.09624, pruned_loss=0.01654, audio_tagging_loss=0.009567, over 3046382.26 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:02:18,845 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235100
2023-11-21 16:02:19,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1567313.3333333333, ans=0.125
2023-11-21 16:02:33,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1567380.0, ans=0.07
2023-11-21 16:02:36,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1567380.0, ans=0.1
2023-11-21 16:02:41,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1567446.6666666667, ans=0.125
2023-11-21 16:03:05,741 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.70 vs. limit=15.0
2023-11-21 16:03:22,503 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6700, loss[loss=0.09943, simple_loss=0.1342, pruned_loss=0.02057, audio_tagging_loss=0.01175, over 15775.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.09622, pruned_loss=0.01656, audio_tagging_loss=0.009551, over 3043373.35 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:03:23,897 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235150
2023-11-21 16:03:31,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1567646.6666666667, ans=0.1
2023-11-21 16:03:49,608 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.083e+01 8.674e+01 9.269e+01 1.242e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 16:04:24,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1567913.3333333333, ans=0.125
2023-11-21 16:04:26,627 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6750, loss[loss=0.06604, simple_loss=0.08707, pruned_loss=0.01373, audio_tagging_loss=0.008773, over 15104.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09596, pruned_loss=0.01656, audio_tagging_loss=0.009513, over 3037034.88 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:04:27,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235200
2023-11-21 16:04:33,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.46 vs. limit=10.0
2023-11-21 16:04:38,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1568046.6666666667, ans=0.1
2023-11-21 16:05:09,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1568180.0, ans=0.125
2023-11-21 16:05:19,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1568246.6666666667, ans=0.2
2023-11-21 16:05:30,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1568313.3333333333, ans=0.125
2023-11-21 16:05:31,740 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6800, loss[loss=0.07048, simple_loss=0.09023, pruned_loss=0.01664, audio_tagging_loss=0.008732, over 14781.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09549, pruned_loss=0.01651, audio_tagging_loss=0.009476, over 3043981.69 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:05:33,149 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235250
2023-11-21 16:05:35,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1568313.3333333333, ans=0.1
2023-11-21 16:05:44,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.70 vs. limit=15.0
2023-11-21 16:05:46,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1568380.0, ans=0.2
2023-11-21 16:05:49,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.24 vs. limit=22.5
2023-11-21 16:05:57,890 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.054e+01 8.686e+01 9.528e+01 1.643e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 16:06:11,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1568513.3333333333, ans=0.125
2023-11-21 16:06:35,984 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6850, loss[loss=0.07051, simple_loss=0.09386, pruned_loss=0.01223, audio_tagging_loss=0.01135, over 16105.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09595, pruned_loss=0.01674, audio_tagging_loss=0.009444, over 3041371.94 frames. ], batch size: 63, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:06:37,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235300
2023-11-21 16:06:55,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1568713.3333333333, ans=0.1
2023-11-21 16:07:08,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1568780.0, ans=0.125
2023-11-21 16:07:39,673 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6900, loss[loss=0.07976, simple_loss=0.1044, pruned_loss=0.0172, audio_tagging_loss=0.01035, over 14153.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09696, pruned_loss=0.01667, audio_tagging_loss=0.009305, over 3044294.32 frames. ], batch size: 54, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:07:41,054 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235350
2023-11-21 16:08:06,947 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 7.983e+01 8.656e+01 9.439e+01 1.131e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 16:08:08,385 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.78 vs. limit=6.0
2023-11-21 16:08:09,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1569113.3333333333, ans=0.125
2023-11-21 16:08:26,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1569180.0, ans=0.2
2023-11-21 16:08:30,043 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:08:30,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1569246.6666666667, ans=0.0
2023-11-21 16:08:44,332 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 6950, loss[loss=0.08092, simple_loss=0.1196, pruned_loss=0.01362, audio_tagging_loss=0.007503, over 15622.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.09608, pruned_loss=0.01657, audio_tagging_loss=0.009416, over 3044207.60 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:08:45,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235400
2023-11-21 16:08:55,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1569313.3333333333, ans=10.0
2023-11-21 16:09:24,837 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0
2023-11-21 16:09:32,510 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:09:50,346 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7000, loss[loss=0.07813, simple_loss=0.09837, pruned_loss=0.0185, audio_tagging_loss=0.01044, over 15394.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09585, pruned_loss=0.01655, audio_tagging_loss=0.009522, over 3044279.29 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:09:51,702 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235450
2023-11-21 16:09:54,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1569646.6666666667, ans=0.125
2023-11-21 16:10:02,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1569713.3333333333, ans=0.125
2023-11-21 16:10:15,855 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.127e+01 8.792e+01 9.421e+01 1.149e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 16:10:28,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1569846.6666666667, ans=0.2
2023-11-21 16:10:28,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1569846.6666666667, ans=0.125
2023-11-21 16:10:35,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1569846.6666666667, ans=0.125
2023-11-21 16:10:53,818 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7050, loss[loss=0.07202, simple_loss=0.09074, pruned_loss=0.0156, audio_tagging_loss=0.01106, over 15306.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09554, pruned_loss=0.01643, audio_tagging_loss=0.00953, over 3037885.21 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:10:55,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235500
2023-11-21 16:10:55,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1569980.0, ans=0.0
2023-11-21 16:11:56,999 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7100, loss[loss=0.07848, simple_loss=0.09457, pruned_loss=0.02215, audio_tagging_loss=0.009049, over 14314.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09526, pruned_loss=0.0164, audio_tagging_loss=0.009592, over 3043577.29 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:11:58,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235550
2023-11-21 16:12:09,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.91 vs. limit=22.5
2023-11-21 16:12:13,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1570380.0, ans=0.125
2023-11-21 16:12:15,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1570380.0, ans=0.125
2023-11-21 16:12:20,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1570380.0, ans=0.0
2023-11-21 16:12:25,217 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.128e+01 8.670e+01 9.348e+01 1.083e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 16:12:25,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1570446.6666666667, ans=0.125
2023-11-21 16:12:42,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.09 vs. limit=12.0
2023-11-21 16:13:01,323 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7150, loss[loss=0.06968, simple_loss=0.08227, pruned_loss=0.01651, audio_tagging_loss=0.01203, over 15440.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09472, pruned_loss=0.01631, audio_tagging_loss=0.00965, over 3041438.37 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:13:03,301 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235600
2023-11-21 16:13:12,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=15.0
2023-11-21 16:13:20,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1570713.3333333333, ans=0.125
2023-11-21 16:13:36,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1570780.0, ans=0.05
2023-11-21 16:13:37,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1570780.0, ans=0.2
2023-11-21 16:14:05,657 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7200, loss[loss=0.07413, simple_loss=0.1037, pruned_loss=0.01228, audio_tagging_loss=0.01001, over 14867.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09553, pruned_loss=0.01638, audio_tagging_loss=0.009716, over 3046187.10 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:14:06,995 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235650
2023-11-21 16:14:32,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1571113.3333333333, ans=10.0
2023-11-21 16:14:33,174 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.636e+01 8.075e+01 8.939e+01 9.765e+01 1.346e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 16:14:54,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1571180.0, ans=0.0
2023-11-21 16:15:06,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1571246.6666666667, ans=0.2
2023-11-21 16:15:08,516 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7250, loss[loss=0.1091, simple_loss=0.1508, pruned_loss=0.02644, audio_tagging_loss=0.007235, over 16428.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09604, pruned_loss=0.01649, audio_tagging_loss=0.009657, over 3042984.00 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:15:09,865 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235700
2023-11-21 16:15:11,374 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:15:15,589 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.85 vs. limit=10.0
2023-11-21 16:16:09,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1571580.0, ans=0.125
2023-11-21 16:16:13,803 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7300, loss[loss=0.0887, simple_loss=0.1267, pruned_loss=0.01879, audio_tagging_loss=0.006565, over 16241.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09598, pruned_loss=0.01645, audio_tagging_loss=0.009563, over 3036644.24 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:16:15,141 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235750
2023-11-21 16:16:31,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1571713.3333333333, ans=0.0
2023-11-21 16:16:42,202 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.221e+01 8.789e+01 9.369e+01 1.391e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 16:16:56,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1571846.6666666667, ans=0.0
2023-11-21 16:16:57,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.95 vs. limit=22.5
2023-11-21 16:17:05,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1571913.3333333333, ans=0.2
2023-11-21 16:17:14,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1571913.3333333333, ans=0.125
2023-11-21 16:17:19,076 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7350, loss[loss=0.07633, simple_loss=0.1039, pruned_loss=0.01694, audio_tagging_loss=0.007442, over 15677.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09551, pruned_loss=0.01645, audio_tagging_loss=0.009452, over 3044500.76 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:17:20,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235800
2023-11-21 16:17:25,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1571980.0, ans=0.0
2023-11-21 16:17:51,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1572113.3333333333, ans=0.125
2023-11-21 16:18:06,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1572180.0, ans=0.0
2023-11-21 16:18:22,172 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7400, loss[loss=0.04965, simple_loss=0.05893, pruned_loss=0.0103, audio_tagging_loss=0.009887, over 14442.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09585, pruned_loss=0.01633, audio_tagging_loss=0.009425, over 3043331.11 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:18:23,533 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235850
2023-11-21 16:18:46,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1572380.0, ans=0.0
2023-11-21 16:18:50,704 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.093e+01 8.770e+01 9.313e+01 1.126e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 16:18:59,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1572513.3333333333, ans=0.0
2023-11-21 16:19:00,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1572513.3333333333, ans=0.125
2023-11-21 16:19:06,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1572513.3333333333, ans=0.0
2023-11-21 16:19:26,604 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7450, loss[loss=0.0641, simple_loss=0.08808, pruned_loss=0.01167, audio_tagging_loss=0.008396, over 15336.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09585, pruned_loss=0.01634, audio_tagging_loss=0.009397, over 3037028.53 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:19:27,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235900
2023-11-21 16:19:31,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-21 16:19:39,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1572713.3333333333, ans=0.1
2023-11-21 16:19:42,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1572713.3333333333, ans=0.125
2023-11-21 16:20:09,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1572846.6666666667, ans=0.2
2023-11-21 16:20:11,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1572846.6666666667, ans=0.2
2023-11-21 16:20:12,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.14 vs. limit=15.0
2023-11-21 16:20:30,999 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7500, loss[loss=0.05873, simple_loss=0.07465, pruned_loss=0.0124, audio_tagging_loss=0.009002, over 14632.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.09538, pruned_loss=0.01626, audio_tagging_loss=0.009471, over 3038495.47 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:20:31,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1572980.0, ans=0.5
2023-11-21 16:20:32,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 235950
2023-11-21 16:20:55,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1573113.3333333333, ans=0.2
2023-11-21 16:20:59,387 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 8.351e+01 8.857e+01 9.417e+01 1.207e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 16:21:04,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1573113.3333333333, ans=0.125
2023-11-21 16:21:28,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1573246.6666666667, ans=0.2
2023-11-21 16:21:34,634 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7550, loss[loss=0.05814, simple_loss=0.07821, pruned_loss=0.009952, audio_tagging_loss=0.009085, over 15672.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09597, pruned_loss=0.01632, audio_tagging_loss=0.009424, over 3037934.45 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:21:35,906 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236000
2023-11-21 16:21:42,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1573313.3333333333, ans=0.125
2023-11-21 16:21:51,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1573380.0, ans=0.1
2023-11-21 16:21:57,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1573380.0, ans=0.125
2023-11-21 16:22:06,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1573446.6666666667, ans=0.2
2023-11-21 16:22:16,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1573513.3333333333, ans=0.0
2023-11-21 16:22:41,886 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7600, loss[loss=0.09515, simple_loss=0.1258, pruned_loss=0.02193, audio_tagging_loss=0.0103, over 16977.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09526, pruned_loss=0.0163, audio_tagging_loss=0.009522, over 3039504.31 frames. ], batch size: 61, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:22:43,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236050
2023-11-21 16:22:48,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1573646.6666666667, ans=0.025
2023-11-21 16:23:09,948 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.352e+01 8.154e+01 8.768e+01 9.478e+01 1.233e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 16:23:37,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.92 vs. limit=15.0
2023-11-21 16:23:46,824 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7650, loss[loss=0.07097, simple_loss=0.09411, pruned_loss=0.01531, audio_tagging_loss=0.0086, over 15289.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09401, pruned_loss=0.01586, audio_tagging_loss=0.009445, over 3041147.39 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:23:48,122 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236100
2023-11-21 16:24:23,810 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:24:29,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1574180.0, ans=0.0
2023-11-21 16:24:38,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1574246.6666666667, ans=0.125
2023-11-21 16:24:52,296 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7700, loss[loss=0.07373, simple_loss=0.096, pruned_loss=0.01494, audio_tagging_loss=0.01079, over 15517.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09351, pruned_loss=0.01571, audio_tagging_loss=0.009452, over 3034385.37 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:24:53,595 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236150
2023-11-21 16:24:55,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1574313.3333333333, ans=0.125
2023-11-21 16:25:22,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.998e+01 8.716e+01 9.349e+01 1.459e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 16:25:31,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1574513.3333333333, ans=0.125
2023-11-21 16:25:40,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1574513.3333333333, ans=0.125
2023-11-21 16:25:42,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1574513.3333333333, ans=0.1
2023-11-21 16:25:46,088 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.97 vs. limit=10.0
2023-11-21 16:25:57,886 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7750, loss[loss=0.06034, simple_loss=0.07113, pruned_loss=0.01177, audio_tagging_loss=0.013, over 15431.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09325, pruned_loss=0.01573, audio_tagging_loss=0.009528, over 3032826.26 frames. ], batch size: 61, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:25:59,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236200
2023-11-21 16:26:30,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.48 vs. limit=6.0
2023-11-21 16:26:34,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.54 vs. limit=15.0
2023-11-21 16:26:45,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1574846.6666666667, ans=0.1
2023-11-21 16:26:45,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1574846.6666666667, ans=0.2
2023-11-21 16:26:51,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1574913.3333333333, ans=0.1
2023-11-21 16:26:55,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1574913.3333333333, ans=0.2
2023-11-21 16:26:58,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1574913.3333333333, ans=0.2
2023-11-21 16:27:03,058 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7800, loss[loss=0.08403, simple_loss=0.1079, pruned_loss=0.02185, audio_tagging_loss=0.008237, over 16141.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09429, pruned_loss=0.01607, audio_tagging_loss=0.009567, over 3028997.66 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:27:04,371 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236250
2023-11-21 16:27:16,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1575046.6666666667, ans=0.0
2023-11-21 16:27:23,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1575046.6666666667, ans=0.125
2023-11-21 16:27:23,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1575046.6666666667, ans=0.0
2023-11-21 16:27:32,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.496e+01 8.260e+01 8.924e+01 9.487e+01 1.142e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 16:27:51,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1575180.0, ans=0.0
2023-11-21 16:28:00,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1575246.6666666667, ans=0.125
2023-11-21 16:28:06,809 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7850, loss[loss=0.07818, simple_loss=0.1018, pruned_loss=0.01794, audio_tagging_loss=0.009353, over 14652.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09486, pruned_loss=0.01609, audio_tagging_loss=0.009618, over 3034029.69 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:28:08,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236300
2023-11-21 16:28:09,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1575313.3333333333, ans=0.125
2023-11-21 16:28:12,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.40 vs. limit=6.0
2023-11-21 16:28:35,461 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=4.200e-02
2023-11-21 16:28:45,506 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.76 vs. limit=12.0
2023-11-21 16:28:49,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1575513.3333333333, ans=0.125
2023-11-21 16:28:55,444 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.68 vs. limit=15.0
2023-11-21 16:29:12,456 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7900, loss[loss=0.08645, simple_loss=0.1114, pruned_loss=0.02113, audio_tagging_loss=0.009616, over 15501.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09541, pruned_loss=0.01612, audio_tagging_loss=0.009649, over 3038740.81 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:29:13,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236350
2023-11-21 16:29:34,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1575713.3333333333, ans=0.0
2023-11-21 16:29:34,502 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0
2023-11-21 16:29:35,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.39 vs. limit=15.0
2023-11-21 16:29:42,401 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.695e+01 8.223e+01 8.741e+01 9.737e+01 2.619e+02, threshold=1.748e+02, percent-clipped=1.0
2023-11-21 16:29:46,877 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.82 vs. limit=10.0
2023-11-21 16:29:48,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1575846.6666666667, ans=0.1
2023-11-21 16:30:16,197 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.84 vs. limit=12.0
2023-11-21 16:30:16,762 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 7950, loss[loss=0.08148, simple_loss=0.09937, pruned_loss=0.02136, audio_tagging_loss=0.01043, over 14578.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09532, pruned_loss=0.01631, audio_tagging_loss=0.00973, over 3039453.74 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:30:16,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1575980.0, ans=0.1
2023-11-21 16:30:18,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236400
2023-11-21 16:30:22,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1575980.0, ans=0.1
2023-11-21 16:30:31,793 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:30:38,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1576046.6666666667, ans=0.2
2023-11-21 16:30:39,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1576046.6666666667, ans=0.125
2023-11-21 16:31:05,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1576180.0, ans=0.125
2023-11-21 16:31:20,504 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8000, loss[loss=0.09239, simple_loss=0.1281, pruned_loss=0.02026, audio_tagging_loss=0.008084, over 16032.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09522, pruned_loss=0.01637, audio_tagging_loss=0.009858, over 3038551.30 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:31:21,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236450
2023-11-21 16:31:48,751 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.14 vs. limit=22.5
2023-11-21 16:31:52,260 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.291e+01 7.990e+01 8.622e+01 9.652e+01 1.226e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-21 16:31:55,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1576446.6666666667, ans=0.0
2023-11-21 16:32:01,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1576513.3333333333, ans=0.0
2023-11-21 16:32:13,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1576580.0, ans=0.0
2023-11-21 16:32:15,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1576580.0, ans=0.125
2023-11-21 16:32:23,866 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8050, loss[loss=0.09926, simple_loss=0.12, pruned_loss=0.02868, audio_tagging_loss=0.01059, over 16088.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09541, pruned_loss=0.01636, audio_tagging_loss=0.009875, over 3049358.91 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:32:25,820 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236500
2023-11-21 16:32:41,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1576713.3333333333, ans=0.125
2023-11-21 16:32:41,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1576713.3333333333, ans=0.125
2023-11-21 16:32:50,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1576780.0, ans=0.0
2023-11-21 16:33:00,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1576780.0, ans=0.0
2023-11-21 16:33:28,740 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8100, loss[loss=0.06454, simple_loss=0.07658, pruned_loss=0.01276, audio_tagging_loss=0.01349, over 15499.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09588, pruned_loss=0.01656, audio_tagging_loss=0.009759, over 3044568.46 frames. ], batch size: 61, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:33:30,043 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236550
2023-11-21 16:33:31,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1576980.0, ans=0.0
2023-11-21 16:33:32,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1576980.0, ans=0.1
2023-11-21 16:33:35,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1576980.0, ans=0.0
2023-11-21 16:33:42,901 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.38 vs. limit=12.0
2023-11-21 16:33:44,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1577046.6666666667, ans=0.125
2023-11-21 16:33:53,871 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0
2023-11-21 16:33:59,137 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.498e+01 8.221e+01 8.771e+01 9.447e+01 1.298e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 16:34:00,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1577113.3333333333, ans=0.125
2023-11-21 16:34:31,734 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8150, loss[loss=0.05447, simple_loss=0.06743, pruned_loss=0.009842, audio_tagging_loss=0.01091, over 14808.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09614, pruned_loss=0.01639, audio_tagging_loss=0.009588, over 3052263.26 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:34:33,054 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236600
2023-11-21 16:34:42,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.38 vs. limit=15.0
2023-11-21 16:35:01,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=1577446.6666666667, ans=15.0
2023-11-21 16:35:02,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1577446.6666666667, ans=0.0
2023-11-21 16:35:06,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1577446.6666666667, ans=0.0
2023-11-21 16:35:07,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1577446.6666666667, ans=0.0
2023-11-21 16:35:11,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1577513.3333333333, ans=0.125
2023-11-21 16:35:17,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1577513.3333333333, ans=22.5
2023-11-21 16:35:24,878 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.59 vs. limit=10.0
2023-11-21 16:35:34,879 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8200, loss[loss=0.1035, simple_loss=0.1419, pruned_loss=0.0264, audio_tagging_loss=0.006129, over 15108.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.0961, pruned_loss=0.0163, audio_tagging_loss=0.009363, over 3051210.89 frames. ], batch size: 54, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:35:34,945 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:35:36,153 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236650
2023-11-21 16:35:51,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1577713.3333333333, ans=0.125
2023-11-21 16:36:07,539 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.532e+01 8.090e+01 8.732e+01 9.271e+01 1.131e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 16:36:40,254 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8250, loss[loss=0.06717, simple_loss=0.0958, pruned_loss=0.0117, audio_tagging_loss=0.007573, over 15447.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09562, pruned_loss=0.01613, audio_tagging_loss=0.009375, over 3049589.34 frames. ], batch size: 58, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:36:41,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236700
2023-11-21 16:37:08,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1578113.3333333333, ans=0.0
2023-11-21 16:37:12,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1578113.3333333333, ans=0.125
2023-11-21 16:37:15,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1578113.3333333333, ans=0.125
2023-11-21 16:37:29,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1578246.6666666667, ans=0.07
2023-11-21 16:37:43,355 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8300, loss[loss=0.08577, simple_loss=0.1178, pruned_loss=0.01989, audio_tagging_loss=0.006994, over 15877.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09587, pruned_loss=0.01609, audio_tagging_loss=0.009297, over 3047446.89 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:37:44,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236750
2023-11-21 16:37:46,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1578313.3333333333, ans=0.0
2023-11-21 16:37:54,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1578380.0, ans=0.125
2023-11-21 16:37:58,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1578380.0, ans=0.0
2023-11-21 16:38:06,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1578380.0, ans=0.125
2023-11-21 16:38:09,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1578446.6666666667, ans=0.0
2023-11-21 16:38:14,686 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.835e+01 8.193e+01 8.671e+01 9.431e+01 1.215e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 16:38:17,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1578446.6666666667, ans=0.125
2023-11-21 16:38:21,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1578513.3333333333, ans=0.125
2023-11-21 16:38:28,058 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.64 vs. limit=12.0
2023-11-21 16:38:45,470 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8350, loss[loss=0.07818, simple_loss=0.09775, pruned_loss=0.01655, audio_tagging_loss=0.01275, over 15165.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09572, pruned_loss=0.01616, audio_tagging_loss=0.00933, over 3048835.26 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:38:46,745 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236800
2023-11-21 16:38:58,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1578713.3333333333, ans=0.125
2023-11-21 16:39:34,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1578846.6666666667, ans=0.0
2023-11-21 16:39:48,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1578980.0, ans=0.125
2023-11-21 16:39:49,534 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8400, loss[loss=0.06585, simple_loss=0.08909, pruned_loss=0.01418, audio_tagging_loss=0.007122, over 15020.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09504, pruned_loss=0.01582, audio_tagging_loss=0.009402, over 3050514.59 frames. ], batch size: 54, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:39:50,873 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236850
2023-11-21 16:39:54,463 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0
2023-11-21 16:39:56,335 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.95 vs. limit=15.0
2023-11-21 16:40:01,308 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.48 vs. limit=22.5
2023-11-21 16:40:11,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1579046.6666666667, ans=0.125
2023-11-21 16:40:21,097 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 7.886e+01 8.656e+01 9.499e+01 1.187e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 16:40:24,091 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.69 vs. limit=15.0
2023-11-21 16:40:27,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1579180.0, ans=0.04949747468305833
2023-11-21 16:40:37,596 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.40 vs. limit=6.0
2023-11-21 16:40:47,919 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.76 vs. limit=6.0
2023-11-21 16:40:53,311 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8450, loss[loss=0.07897, simple_loss=0.1062, pruned_loss=0.0193, audio_tagging_loss=0.006552, over 15464.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09475, pruned_loss=0.01591, audio_tagging_loss=0.009489, over 3052550.84 frames. ], batch size: 58, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:40:54,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236900
2023-11-21 16:41:02,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1579313.3333333333, ans=0.125
2023-11-21 16:41:07,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1579380.0, ans=0.1
2023-11-21 16:41:07,483 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.88 vs. limit=22.5
2023-11-21 16:41:33,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1579513.3333333333, ans=0.0
2023-11-21 16:41:52,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1579580.0, ans=0.0
2023-11-21 16:41:53,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1579580.0, ans=0.125
2023-11-21 16:41:56,706 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8500, loss[loss=0.05465, simple_loss=0.07038, pruned_loss=0.01028, audio_tagging_loss=0.009182, over 14082.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09545, pruned_loss=0.01608, audio_tagging_loss=0.009419, over 3049972.73 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:41:58,066 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 236950
2023-11-21 16:42:26,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1579780.0, ans=0.125
2023-11-21 16:42:27,760 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.82 vs. limit=22.5
2023-11-21 16:42:29,770 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.566e+01 7.919e+01 8.659e+01 9.250e+01 1.200e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 16:43:01,491 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8550, loss[loss=0.06966, simple_loss=0.09479, pruned_loss=0.01372, audio_tagging_loss=0.008544, over 14621.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09556, pruned_loss=0.01622, audio_tagging_loss=0.009417, over 3046094.66 frames. ], batch size: 54, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:43:02,737 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237000
2023-11-21 16:43:13,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1580046.6666666667, ans=0.09899494936611666
2023-11-21 16:43:27,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1580113.3333333333, ans=0.1
2023-11-21 16:43:36,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1580113.3333333333, ans=0.95
2023-11-21 16:43:44,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1580180.0, ans=0.125
2023-11-21 16:43:56,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1580246.6666666667, ans=0.0
2023-11-21 16:44:06,845 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8600, loss[loss=0.08814, simple_loss=0.1223, pruned_loss=0.02058, audio_tagging_loss=0.006435, over 15169.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09547, pruned_loss=0.01618, audio_tagging_loss=0.009498, over 3051245.81 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:44:08,121 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237050
2023-11-21 16:44:21,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1580380.0, ans=0.125
2023-11-21 16:44:34,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1580446.6666666667, ans=0.125
2023-11-21 16:44:38,169 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 8.198e+01 8.655e+01 9.570e+01 1.230e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 16:44:44,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1580513.3333333333, ans=0.125
2023-11-21 16:44:49,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.84 vs. limit=15.0
2023-11-21 16:44:49,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1580513.3333333333, ans=0.125
2023-11-21 16:44:54,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1580513.3333333333, ans=0.0
2023-11-21 16:44:56,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1580580.0, ans=10.0
2023-11-21 16:45:09,712 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8650, loss[loss=0.05591, simple_loss=0.07503, pruned_loss=0.00834, audio_tagging_loss=0.01005, over 15526.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09663, pruned_loss=0.0164, audio_tagging_loss=0.009454, over 3050929.14 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:45:11,026 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237100
2023-11-21 16:45:32,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1580713.3333333333, ans=0.1
2023-11-21 16:45:43,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1580780.0, ans=0.125
2023-11-21 16:46:14,162 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8700, loss[loss=0.07009, simple_loss=0.08327, pruned_loss=0.01684, audio_tagging_loss=0.01162, over 14498.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09649, pruned_loss=0.0164, audio_tagging_loss=0.009507, over 3058137.57 frames. ], batch size: 53, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:46:15,419 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237150
2023-11-21 16:46:22,312 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.39 vs. limit=12.0
2023-11-21 16:46:45,561 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.271e+01 8.853e+01 9.798e+01 1.350e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 16:47:05,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1581246.6666666667, ans=0.07
2023-11-21 16:47:13,400 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.55 vs. limit=22.5
2023-11-21 16:47:17,469 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8750, loss[loss=0.07893, simple_loss=0.09629, pruned_loss=0.01985, audio_tagging_loss=0.01093, over 16245.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09753, pruned_loss=0.01649, audio_tagging_loss=0.009601, over 3060277.75 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:47:18,786 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237200
2023-11-21 16:47:30,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1581380.0, ans=0.0
2023-11-21 16:47:41,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1581446.6666666667, ans=0.125
2023-11-21 16:48:01,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1581513.3333333333, ans=0.025
2023-11-21 16:48:15,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1581580.0, ans=0.125
2023-11-21 16:48:20,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1581646.6666666667, ans=0.1
2023-11-21 16:48:21,621 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8800, loss[loss=0.09257, simple_loss=0.114, pruned_loss=0.0234, audio_tagging_loss=0.01218, over 14660.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09883, pruned_loss=0.0169, audio_tagging_loss=0.009602, over 3057466.29 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:48:22,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237250
2023-11-21 16:48:26,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1581646.6666666667, ans=0.0
2023-11-21 16:48:29,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1581646.6666666667, ans=0.125
2023-11-21 16:48:47,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1581780.0, ans=0.0
2023-11-21 16:48:50,981 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.56 vs. limit=12.0
2023-11-21 16:48:53,818 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.845e+01 8.270e+01 9.064e+01 1.008e+02 1.291e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-21 16:49:08,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.94 vs. limit=15.0
2023-11-21 16:49:25,645 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8850, loss[loss=0.08271, simple_loss=0.1069, pruned_loss=0.01838, audio_tagging_loss=0.01088, over 15896.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09864, pruned_loss=0.01692, audio_tagging_loss=0.009662, over 3050595.84 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:49:26,911 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237300
2023-11-21 16:49:38,425 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:50:02,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1582113.3333333333, ans=0.125
2023-11-21 16:50:04,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1582180.0, ans=0.125
2023-11-21 16:50:11,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1582180.0, ans=0.04949747468305833
2023-11-21 16:50:17,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1582246.6666666667, ans=0.1
2023-11-21 16:50:30,408 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8900, loss[loss=0.07861, simple_loss=0.1167, pruned_loss=0.01554, audio_tagging_loss=0.004728, over 14817.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09739, pruned_loss=0.01669, audio_tagging_loss=0.009496, over 3045662.84 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:50:30,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1582313.3333333333, ans=0.125
2023-11-21 16:50:31,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237350
2023-11-21 16:50:32,243 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.64 vs. limit=15.0
2023-11-21 16:50:45,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1582380.0, ans=0.0
2023-11-21 16:51:01,804 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 7.931e+01 8.617e+01 9.547e+01 1.329e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 16:51:07,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1582513.3333333333, ans=0.125
2023-11-21 16:51:33,472 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 8950, loss[loss=0.08443, simple_loss=0.1096, pruned_loss=0.02164, audio_tagging_loss=0.008007, over 14960.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.0973, pruned_loss=0.01669, audio_tagging_loss=0.009302, over 3044524.58 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:51:34,824 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237400
2023-11-21 16:51:37,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0
2023-11-21 16:51:47,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1582713.3333333333, ans=0.015
2023-11-21 16:51:55,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1582713.3333333333, ans=0.125
2023-11-21 16:51:56,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1582713.3333333333, ans=0.1
2023-11-21 16:51:56,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1582713.3333333333, ans=0.125
2023-11-21 16:52:14,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1582846.6666666667, ans=0.0
2023-11-21 16:52:19,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1582846.6666666667, ans=0.0
2023-11-21 16:52:23,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1582846.6666666667, ans=0.125
2023-11-21 16:52:23,770 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.83 vs. limit=15.0
2023-11-21 16:52:30,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1582913.3333333333, ans=0.125
2023-11-21 16:52:38,627 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9000, loss[loss=0.05538, simple_loss=0.07795, pruned_loss=0.007558, audio_tagging_loss=0.008844, over 14276.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.09637, pruned_loss=0.01644, audio_tagging_loss=0.009404, over 3044054.00 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:52:38,628 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 16:53:19,764 INFO [train_asr.py:1253] (2/4) Epoch 20, validation: loss=0.06046, simple_loss=0.05217, pruned_loss=0.005278, audio_tagging_loss=0.0291, over 4681554.00 frames. 
2023-11-21 16:53:19,765 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 16:53:21,050 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237450
2023-11-21 16:53:39,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1583046.6666666667, ans=0.0
2023-11-21 16:53:52,349 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.325e+01 8.299e+01 8.982e+01 9.463e+01 1.306e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 16:53:58,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1583180.0, ans=0.1
2023-11-21 16:54:22,605 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9050, loss[loss=0.08889, simple_loss=0.1279, pruned_loss=0.0215, audio_tagging_loss=0.00344, over 15961.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09611, pruned_loss=0.01624, audio_tagging_loss=0.009335, over 3048934.20 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:54:23,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237500
2023-11-21 16:54:31,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1583313.3333333333, ans=0.125
2023-11-21 16:54:35,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1583380.0, ans=0.0
2023-11-21 16:54:42,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1583380.0, ans=0.1
2023-11-21 16:55:27,146 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9100, loss[loss=0.07234, simple_loss=0.09292, pruned_loss=0.01639, audio_tagging_loss=0.009492, over 14446.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.09655, pruned_loss=0.01652, audio_tagging_loss=0.009243, over 3053117.70 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:55:28,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237550
2023-11-21 16:55:28,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1583646.6666666667, ans=0.125
2023-11-21 16:55:35,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1583646.6666666667, ans=0.025
2023-11-21 16:56:00,156 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.100e+01 8.666e+01 9.594e+01 1.245e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 16:56:00,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1583780.0, ans=0.0
2023-11-21 16:56:13,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1583846.6666666667, ans=0.125
2023-11-21 16:56:21,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.66 vs. limit=22.5
2023-11-21 16:56:23,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1583913.3333333333, ans=0.0
2023-11-21 16:56:31,660 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-21 16:56:32,110 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9150, loss[loss=0.06035, simple_loss=0.08029, pruned_loss=0.01058, audio_tagging_loss=0.009631, over 15532.00 frames. ], tot_loss[loss=0.07414, simple_loss=0.0964, pruned_loss=0.01676, audio_tagging_loss=0.009179, over 3050451.85 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:56:33,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237600
2023-11-21 16:56:49,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.99 vs. limit=10.0
2023-11-21 16:56:53,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1584046.6666666667, ans=0.0
2023-11-21 16:56:59,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1584113.3333333333, ans=0.07
2023-11-21 16:57:05,712 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.38 vs. limit=10.0
2023-11-21 16:57:16,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1584180.0, ans=0.0
2023-11-21 16:57:35,486 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9200, loss[loss=0.08188, simple_loss=0.1023, pruned_loss=0.01991, audio_tagging_loss=0.01081, over 15318.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09639, pruned_loss=0.0168, audio_tagging_loss=0.00925, over 3044284.35 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:57:36,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237650
2023-11-21 16:58:08,692 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.617e+01 8.012e+01 8.576e+01 9.231e+01 1.366e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 16:58:13,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.76 vs. limit=15.0
2023-11-21 16:58:22,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1584513.3333333333, ans=0.125
2023-11-21 16:58:30,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1584580.0, ans=0.125
2023-11-21 16:58:37,792 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9250, loss[loss=0.08266, simple_loss=0.11, pruned_loss=0.02058, audio_tagging_loss=0.007057, over 15763.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09641, pruned_loss=0.01667, audio_tagging_loss=0.009222, over 3049017.41 frames. ], batch size: 58, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:58:39,730 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237700
2023-11-21 16:58:53,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1584713.3333333333, ans=0.5
2023-11-21 16:59:09,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1584780.0, ans=0.125
2023-11-21 16:59:20,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1584846.6666666667, ans=0.1
2023-11-21 16:59:20,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1584846.6666666667, ans=0.125
2023-11-21 16:59:40,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.93 vs. limit=22.5
2023-11-21 16:59:42,841 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9300, loss[loss=0.07512, simple_loss=0.09247, pruned_loss=0.01938, audio_tagging_loss=0.009507, over 14561.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09605, pruned_loss=0.01657, audio_tagging_loss=0.009382, over 3047558.06 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:59:44,224 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237750
2023-11-21 16:59:45,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1584980.0, ans=0.125
2023-11-21 16:59:53,438 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.13 vs. limit=15.0
2023-11-21 17:00:08,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1585113.3333333333, ans=0.0
2023-11-21 17:00:14,591 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.079e+01 8.667e+01 9.296e+01 1.099e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 17:00:43,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1585246.6666666667, ans=0.125
2023-11-21 17:00:44,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.30 vs. limit=15.0
2023-11-21 17:00:45,971 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9350, loss[loss=0.07467, simple_loss=0.1018, pruned_loss=0.01315, audio_tagging_loss=0.01064, over 14738.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09607, pruned_loss=0.01647, audio_tagging_loss=0.009326, over 3047316.84 frames. ], batch size: 53, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:00:47,285 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237800
2023-11-21 17:01:01,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=12.0
2023-11-21 17:01:49,448 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9400, loss[loss=0.06828, simple_loss=0.08717, pruned_loss=0.01416, audio_tagging_loss=0.01053, over 15689.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.0955, pruned_loss=0.01636, audio_tagging_loss=0.00951, over 3045716.07 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:01:49,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1585646.6666666667, ans=0.125
2023-11-21 17:01:50,841 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237850
2023-11-21 17:01:52,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1585646.6666666667, ans=0.2
2023-11-21 17:01:56,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1585646.6666666667, ans=0.125
2023-11-21 17:01:59,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1585646.6666666667, ans=0.05
2023-11-21 17:01:59,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1585646.6666666667, ans=0.125
2023-11-21 17:02:10,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1585713.3333333333, ans=0.125
2023-11-21 17:02:24,232 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.317e+01 9.011e+01 9.971e+01 1.419e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-21 17:02:34,636 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.64 vs. limit=22.5
2023-11-21 17:02:41,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1585913.3333333333, ans=0.2
2023-11-21 17:02:42,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1585913.3333333333, ans=0.1
2023-11-21 17:02:42,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1585913.3333333333, ans=0.125
2023-11-21 17:02:53,505 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:02:55,316 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9450, loss[loss=0.09114, simple_loss=0.1214, pruned_loss=0.02183, audio_tagging_loss=0.008618, over 14381.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09598, pruned_loss=0.01637, audio_tagging_loss=0.009562, over 3042280.30 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:02:56,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237900
2023-11-21 17:02:58,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1585980.0, ans=0.0
2023-11-21 17:03:01,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1585980.0, ans=0.125
2023-11-21 17:03:09,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1586046.6666666667, ans=0.1
2023-11-21 17:03:12,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1586046.6666666667, ans=0.125
2023-11-21 17:03:33,335 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.11 vs. limit=6.0
2023-11-21 17:03:38,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1586180.0, ans=0.125
2023-11-21 17:03:41,153 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:03:59,173 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9500, loss[loss=0.07049, simple_loss=0.09748, pruned_loss=0.01344, audio_tagging_loss=0.00831, over 15518.00 frames. ], tot_loss[loss=0.07414, simple_loss=0.09617, pruned_loss=0.01644, audio_tagging_loss=0.009607, over 3047500.36 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:04:00,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 237950
2023-11-21 17:04:12,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.59 vs. limit=22.5
2023-11-21 17:04:17,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1586380.0, ans=0.125
2023-11-21 17:04:31,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.163e+01 8.915e+01 9.675e+01 1.127e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 17:04:32,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1586446.6666666667, ans=0.2
2023-11-21 17:04:43,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1586513.3333333333, ans=0.1
2023-11-21 17:04:44,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0
2023-11-21 17:04:45,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1586513.3333333333, ans=0.125
2023-11-21 17:05:01,684 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9550, loss[loss=0.06559, simple_loss=0.08657, pruned_loss=0.01553, audio_tagging_loss=0.00678, over 15436.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09718, pruned_loss=0.01671, audio_tagging_loss=0.009581, over 3047373.56 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:05:03,035 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238000
2023-11-21 17:05:04,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1586646.6666666667, ans=0.1
2023-11-21 17:05:20,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1586713.3333333333, ans=0.125
2023-11-21 17:05:23,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1586713.3333333333, ans=0.04949747468305833
2023-11-21 17:05:48,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1586846.6666666667, ans=0.125
2023-11-21 17:06:01,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1586913.3333333333, ans=0.0
2023-11-21 17:06:06,217 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9600, loss[loss=0.07834, simple_loss=0.1083, pruned_loss=0.01497, audio_tagging_loss=0.009229, over 14013.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.09721, pruned_loss=0.01674, audio_tagging_loss=0.009649, over 3053214.38 frames. ], batch size: 53, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:06:07,577 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238050
2023-11-21 17:06:38,578 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.410e+01 9.133e+01 9.513e+01 1.325e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-21 17:07:10,206 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9650, loss[loss=0.04814, simple_loss=0.05756, pruned_loss=0.006703, audio_tagging_loss=0.01266, over 15214.00 frames. ], tot_loss[loss=0.07519, simple_loss=0.09744, pruned_loss=0.01692, audio_tagging_loss=0.009554, over 3048704.08 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:07:11,574 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238100
2023-11-21 17:07:22,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1587380.0, ans=0.125
2023-11-21 17:07:23,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.11 vs. limit=15.0
2023-11-21 17:07:27,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1587380.0, ans=0.125
2023-11-21 17:07:27,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1587380.0, ans=0.125
2023-11-21 17:07:32,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1587380.0, ans=0.125
2023-11-21 17:07:36,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1587446.6666666667, ans=0.0
2023-11-21 17:08:12,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1587646.6666666667, ans=0.125
2023-11-21 17:08:13,229 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9700, loss[loss=0.07552, simple_loss=0.103, pruned_loss=0.01633, audio_tagging_loss=0.007708, over 13700.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09731, pruned_loss=0.01679, audio_tagging_loss=0.009352, over 3042091.90 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:08:14,592 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238150
2023-11-21 17:08:17,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1587646.6666666667, ans=0.2
2023-11-21 17:08:27,130 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.76 vs. limit=15.0
2023-11-21 17:08:29,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1587713.3333333333, ans=0.1
2023-11-21 17:08:46,890 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.277e+01 8.905e+01 9.605e+01 1.183e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 17:08:56,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1587846.6666666667, ans=0.1
2023-11-21 17:09:07,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1587913.3333333333, ans=0.0
2023-11-21 17:09:17,304 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9750, loss[loss=0.1027, simple_loss=0.1457, pruned_loss=0.02177, audio_tagging_loss=0.0081, over 16092.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09681, pruned_loss=0.01671, audio_tagging_loss=0.009299, over 3045761.20 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:09:18,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238200
2023-11-21 17:09:21,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1587980.0, ans=0.1
2023-11-21 17:09:29,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1588046.6666666667, ans=0.0
2023-11-21 17:09:45,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1588113.3333333333, ans=0.125
2023-11-21 17:09:48,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1588113.3333333333, ans=0.125
2023-11-21 17:10:14,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1588246.6666666667, ans=0.125
2023-11-21 17:10:16,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1588246.6666666667, ans=0.1
2023-11-21 17:10:21,291 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9800, loss[loss=0.0554, simple_loss=0.07384, pruned_loss=0.0102, audio_tagging_loss=0.008277, over 14144.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09687, pruned_loss=0.01662, audio_tagging_loss=0.009258, over 3045518.87 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:10:22,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238250
2023-11-21 17:10:25,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1588313.3333333333, ans=0.125
2023-11-21 17:10:32,495 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.02 vs. limit=10.0
2023-11-21 17:10:37,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1588380.0, ans=0.125
2023-11-21 17:10:39,558 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=15.0
2023-11-21 17:10:45,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1588446.6666666667, ans=0.125
2023-11-21 17:10:55,714 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.038e+01 8.658e+01 9.399e+01 1.112e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 17:11:02,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1588513.3333333333, ans=0.5
2023-11-21 17:11:04,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1588513.3333333333, ans=0.125
2023-11-21 17:11:17,620 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:11:24,841 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9850, loss[loss=0.07813, simple_loss=0.1101, pruned_loss=0.01691, audio_tagging_loss=0.006154, over 16926.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09742, pruned_loss=0.01676, audio_tagging_loss=0.009186, over 3052289.35 frames. ], batch size: 62, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:11:26,110 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238300
2023-11-21 17:11:45,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1588713.3333333333, ans=0.1
2023-11-21 17:12:02,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1588846.6666666667, ans=0.2
2023-11-21 17:12:16,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1588913.3333333333, ans=0.125
2023-11-21 17:12:29,620 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9900, loss[loss=0.08343, simple_loss=0.1024, pruned_loss=0.02219, audio_tagging_loss=0.01002, over 15289.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09686, pruned_loss=0.01671, audio_tagging_loss=0.009214, over 3045220.18 frames. ], batch size: 58, lr: 3.40e-03, grad_scale: 8.0
2023-11-21 17:12:30,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238350
2023-11-21 17:12:33,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1588980.0, ans=0.0
2023-11-21 17:12:37,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1588980.0, ans=0.09899494936611666
2023-11-21 17:12:38,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1588980.0, ans=0.125
2023-11-21 17:12:50,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1589046.6666666667, ans=0.125
2023-11-21 17:13:04,963 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 7.926e+01 8.564e+01 9.409e+01 1.276e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 17:13:06,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1589180.0, ans=0.0
2023-11-21 17:13:23,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1589246.6666666667, ans=0.1
2023-11-21 17:13:30,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.34 vs. limit=8.0
2023-11-21 17:13:32,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1589313.3333333333, ans=0.0
2023-11-21 17:13:33,228 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 9950, loss[loss=0.06314, simple_loss=0.07921, pruned_loss=0.01064, audio_tagging_loss=0.0129, over 15311.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.09699, pruned_loss=0.01671, audio_tagging_loss=0.00922, over 3042817.70 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 8.0
2023-11-21 17:13:34,525 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238400
2023-11-21 17:13:36,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.57 vs. limit=15.0
2023-11-21 17:13:37,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1589313.3333333333, ans=0.2
2023-11-21 17:13:43,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1589313.3333333333, ans=0.07
2023-11-21 17:14:10,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1589513.3333333333, ans=0.125
2023-11-21 17:14:11,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1589513.3333333333, ans=0.125
2023-11-21 17:14:25,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1589580.0, ans=0.0
2023-11-21 17:14:36,963 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10000, loss[loss=0.08776, simple_loss=0.1085, pruned_loss=0.02307, audio_tagging_loss=0.01042, over 15629.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09659, pruned_loss=0.01657, audio_tagging_loss=0.009265, over 3045374.70 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:14:38,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238450
2023-11-21 17:14:50,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1589713.3333333333, ans=0.2
2023-11-21 17:15:13,228 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.031e+01 7.913e+01 8.716e+01 9.510e+01 1.205e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 17:15:28,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1589913.3333333333, ans=0.125
2023-11-21 17:15:28,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1589913.3333333333, ans=0.0
2023-11-21 17:15:28,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1589913.3333333333, ans=0.07
2023-11-21 17:15:32,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1589913.3333333333, ans=0.2
2023-11-21 17:15:36,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1589913.3333333333, ans=0.125
2023-11-21 17:15:41,472 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10050, loss[loss=0.05832, simple_loss=0.06462, pruned_loss=0.01094, audio_tagging_loss=0.01506, over 14331.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09663, pruned_loss=0.01655, audio_tagging_loss=0.009284, over 3050344.36 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:15:42,760 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238500
2023-11-21 17:15:53,652 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=13.22 vs. limit=15.0
2023-11-21 17:16:12,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1590113.3333333333, ans=0.125
2023-11-21 17:16:12,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1590113.3333333333, ans=0.125
2023-11-21 17:16:20,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1590180.0, ans=0.1
2023-11-21 17:16:45,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.91 vs. limit=10.0
2023-11-21 17:16:46,158 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10100, loss[loss=0.06166, simple_loss=0.07892, pruned_loss=0.01223, audio_tagging_loss=0.009973, over 15108.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.0964, pruned_loss=0.01633, audio_tagging_loss=0.00944, over 3055947.31 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:16:47,419 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238550
2023-11-21 17:17:22,058 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.174e+01 9.110e+01 9.882e+01 1.190e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-21 17:17:24,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1590513.3333333333, ans=0.125
2023-11-21 17:17:31,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1590513.3333333333, ans=0.0
2023-11-21 17:17:37,888 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:17:49,995 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10150, loss[loss=0.09062, simple_loss=0.1058, pruned_loss=0.02625, audio_tagging_loss=0.01149, over 15374.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09637, pruned_loss=0.01651, audio_tagging_loss=0.009544, over 3055320.81 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:17:51,347 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238600
2023-11-21 17:18:09,159 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.49 vs. limit=22.5
2023-11-21 17:18:20,050 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:18:33,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1590846.6666666667, ans=0.0
2023-11-21 17:18:35,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1590846.6666666667, ans=0.125
2023-11-21 17:18:40,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1590913.3333333333, ans=0.0
2023-11-21 17:18:41,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1590913.3333333333, ans=0.125
2023-11-21 17:18:46,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1590913.3333333333, ans=0.0
2023-11-21 17:18:54,013 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10200, loss[loss=0.09929, simple_loss=0.1335, pruned_loss=0.02678, audio_tagging_loss=0.005761, over 15076.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09579, pruned_loss=0.01645, audio_tagging_loss=0.009625, over 3058759.15 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:18:55,302 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238650
2023-11-21 17:18:55,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1590980.0, ans=0.125
2023-11-21 17:18:57,651 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.31 vs. limit=15.0
2023-11-21 17:19:06,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1591046.6666666667, ans=0.125
2023-11-21 17:19:12,448 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.99 vs. limit=15.0
2023-11-21 17:19:17,861 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:19:23,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1591113.3333333333, ans=0.0
2023-11-21 17:19:29,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1591113.3333333333, ans=0.1
2023-11-21 17:19:29,934 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.636e+01 9.230e+01 9.917e+01 1.306e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-21 17:19:41,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=15.0
2023-11-21 17:19:58,395 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10250, loss[loss=0.07212, simple_loss=0.09283, pruned_loss=0.01501, audio_tagging_loss=0.01069, over 14993.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.0959, pruned_loss=0.01635, audio_tagging_loss=0.009708, over 3053466.59 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:20:00,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238700
2023-11-21 17:20:01,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1591313.3333333333, ans=0.125
2023-11-21 17:20:01,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1591313.3333333333, ans=0.125
2023-11-21 17:20:30,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1591446.6666666667, ans=0.125
2023-11-21 17:20:34,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1591446.6666666667, ans=15.0
2023-11-21 17:20:35,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1591513.3333333333, ans=0.1
2023-11-21 17:20:44,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1591513.3333333333, ans=0.125
2023-11-21 17:20:58,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1591580.0, ans=0.2
2023-11-21 17:21:01,963 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10300, loss[loss=0.07417, simple_loss=0.102, pruned_loss=0.0167, audio_tagging_loss=0.006457, over 14553.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09622, pruned_loss=0.01643, audio_tagging_loss=0.009748, over 3055511.16 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:21:03,222 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238750
2023-11-21 17:21:33,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.75 vs. limit=15.0
2023-11-21 17:21:38,272 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 7.986e+01 8.562e+01 9.250e+01 1.387e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 17:21:45,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1591846.6666666667, ans=0.1
2023-11-21 17:21:47,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1591846.6666666667, ans=0.1
2023-11-21 17:21:54,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1591913.3333333333, ans=0.2
2023-11-21 17:21:59,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1591913.3333333333, ans=0.125
2023-11-21 17:22:05,310 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10350, loss[loss=0.07731, simple_loss=0.1034, pruned_loss=0.01659, audio_tagging_loss=0.009029, over 15707.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09583, pruned_loss=0.01634, audio_tagging_loss=0.009831, over 3063102.27 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:22:06,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238800
2023-11-21 17:22:15,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1591980.0, ans=0.125
2023-11-21 17:22:15,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1591980.0, ans=0.0
2023-11-21 17:22:43,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1592180.0, ans=0.0
2023-11-21 17:22:47,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1592180.0, ans=0.1
2023-11-21 17:23:10,376 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10400, loss[loss=0.06979, simple_loss=0.08529, pruned_loss=0.0132, audio_tagging_loss=0.01395, over 14873.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09645, pruned_loss=0.01655, audio_tagging_loss=0.009952, over 3059352.34 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:23:11,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238850
2023-11-21 17:23:11,833 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:23:31,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1592380.0, ans=0.1
2023-11-21 17:23:46,369 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 7.924e+01 8.815e+01 9.464e+01 2.008e+02, threshold=1.763e+02, percent-clipped=1.0
2023-11-21 17:24:14,053 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10450, loss[loss=0.06018, simple_loss=0.07774, pruned_loss=0.01133, audio_tagging_loss=0.009977, over 15483.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09599, pruned_loss=0.01639, audio_tagging_loss=0.009816, over 3058998.83 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:24:15,346 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238900
2023-11-21 17:24:15,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1592646.6666666667, ans=0.0
2023-11-21 17:24:18,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1592646.6666666667, ans=0.125
2023-11-21 17:24:24,435 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.31 vs. limit=15.0
2023-11-21 17:24:55,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1592846.6666666667, ans=0.125
2023-11-21 17:25:10,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1592913.3333333333, ans=0.2
2023-11-21 17:25:14,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1592913.3333333333, ans=0.1
2023-11-21 17:25:16,684 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10500, loss[loss=0.06069, simple_loss=0.07915, pruned_loss=0.01294, audio_tagging_loss=0.008176, over 15635.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09575, pruned_loss=0.01643, audio_tagging_loss=0.009667, over 3057040.58 frames. ], batch size: 61, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:25:18,003 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 238950
2023-11-21 17:25:37,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1593046.6666666667, ans=0.1
2023-11-21 17:25:41,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1593046.6666666667, ans=0.1
2023-11-21 17:25:54,783 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.981e+01 8.666e+01 9.446e+01 1.198e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 17:26:10,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1593246.6666666667, ans=0.125
2023-11-21 17:26:18,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1593246.6666666667, ans=0.125
2023-11-21 17:26:22,632 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10550, loss[loss=0.07166, simple_loss=0.09231, pruned_loss=0.01496, audio_tagging_loss=0.01054, over 14299.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09597, pruned_loss=0.01641, audio_tagging_loss=0.009596, over 3052440.92 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:26:23,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239000
2023-11-21 17:26:25,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1593313.3333333333, ans=0.125
2023-11-21 17:26:30,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.71 vs. limit=15.0
2023-11-21 17:26:45,034 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.08 vs. limit=15.0
2023-11-21 17:26:45,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1593380.0, ans=0.0
2023-11-21 17:26:48,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1593446.6666666667, ans=0.2
2023-11-21 17:26:57,177 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.12 vs. limit=15.0
2023-11-21 17:27:06,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1593513.3333333333, ans=0.0
2023-11-21 17:27:08,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1593513.3333333333, ans=0.035
2023-11-21 17:27:24,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1593580.0, ans=0.0
2023-11-21 17:27:26,928 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10600, loss[loss=0.1236, simple_loss=0.1571, pruned_loss=0.03832, audio_tagging_loss=0.006723, over 15821.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09574, pruned_loss=0.01638, audio_tagging_loss=0.00958, over 3051051.70 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:27:28,361 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239050
2023-11-21 17:27:39,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1593713.3333333333, ans=0.07
2023-11-21 17:28:04,041 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.229e+01 8.607e+01 9.453e+01 1.327e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 17:28:15,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1593846.6666666667, ans=0.0
2023-11-21 17:28:30,410 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10650, loss[loss=0.1004, simple_loss=0.1301, pruned_loss=0.0267, audio_tagging_loss=0.008674, over 15152.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09618, pruned_loss=0.01647, audio_tagging_loss=0.009488, over 3048009.47 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:28:31,741 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239100
2023-11-21 17:28:36,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1593980.0, ans=0.125
2023-11-21 17:28:58,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1594113.3333333333, ans=0.0
2023-11-21 17:28:58,974 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.06 vs. limit=22.5
2023-11-21 17:29:02,598 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.36 vs. limit=15.0
2023-11-21 17:29:05,108 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.28 vs. limit=10.0
2023-11-21 17:29:20,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1594246.6666666667, ans=0.125
2023-11-21 17:29:30,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1594246.6666666667, ans=0.0
2023-11-21 17:29:34,667 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10700, loss[loss=0.04884, simple_loss=0.06153, pruned_loss=0.007732, audio_tagging_loss=0.01034, over 15749.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09545, pruned_loss=0.01634, audio_tagging_loss=0.009601, over 3045653.04 frames. ], batch size: 60, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:29:35,986 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239150
2023-11-21 17:30:01,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1594446.6666666667, ans=0.125
2023-11-21 17:30:11,208 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.705e+01 8.191e+01 8.750e+01 9.714e+01 1.316e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 17:30:38,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1594646.6666666667, ans=0.1
2023-11-21 17:30:39,177 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10750, loss[loss=0.08109, simple_loss=0.1066, pruned_loss=0.02118, audio_tagging_loss=0.006589, over 14803.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09563, pruned_loss=0.01628, audio_tagging_loss=0.009508, over 3046109.40 frames. ], batch size: 53, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:30:40,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239200
2023-11-21 17:30:59,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1594713.3333333333, ans=0.125
2023-11-21 17:31:15,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1594780.0, ans=0.0
2023-11-21 17:31:23,375 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.53 vs. limit=10.0
2023-11-21 17:31:43,360 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10800, loss[loss=0.05771, simple_loss=0.07414, pruned_loss=0.009374, audio_tagging_loss=0.01127, over 16752.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09542, pruned_loss=0.01609, audio_tagging_loss=0.009444, over 3054736.34 frames. ], batch size: 63, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:31:44,696 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239250
2023-11-21 17:31:54,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1595046.6666666667, ans=0.1
2023-11-21 17:32:21,227 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.353e+01 8.126e+01 8.721e+01 9.296e+01 1.155e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 17:32:21,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1595180.0, ans=0.0
2023-11-21 17:32:22,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1595180.0, ans=0.125
2023-11-21 17:32:35,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1595246.6666666667, ans=0.125
2023-11-21 17:32:43,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1595246.6666666667, ans=0.025
2023-11-21 17:32:48,044 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10850, loss[loss=0.06965, simple_loss=0.09387, pruned_loss=0.0149, audio_tagging_loss=0.007811, over 14720.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09485, pruned_loss=0.01601, audio_tagging_loss=0.009473, over 3049227.44 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:32:49,358 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239300
2023-11-21 17:32:52,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.99 vs. limit=15.0
2023-11-21 17:32:58,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1595313.3333333333, ans=0.1
2023-11-21 17:33:07,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1595380.0, ans=0.2
2023-11-21 17:33:11,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1595380.0, ans=0.07
2023-11-21 17:33:43,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1595580.0, ans=0.125
2023-11-21 17:33:46,328 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:33:51,747 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10900, loss[loss=0.09303, simple_loss=0.1201, pruned_loss=0.02688, audio_tagging_loss=0.00612, over 14077.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09544, pruned_loss=0.01613, audio_tagging_loss=0.009424, over 3049146.86 frames. ], batch size: 54, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:33:53,721 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239350
2023-11-21 17:33:55,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1595646.6666666667, ans=0.0
2023-11-21 17:34:13,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1595713.3333333333, ans=0.5
2023-11-21 17:34:28,937 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.501e+01 8.126e+01 8.588e+01 9.275e+01 1.184e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 17:34:40,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1595846.6666666667, ans=0.0
2023-11-21 17:34:42,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1595913.3333333333, ans=0.1
2023-11-21 17:34:56,177 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 10950, loss[loss=0.06767, simple_loss=0.08441, pruned_loss=0.01647, audio_tagging_loss=0.008993, over 15291.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09492, pruned_loss=0.01593, audio_tagging_loss=0.009493, over 3046034.04 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:34:56,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1595980.0, ans=0.0
2023-11-21 17:34:57,575 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239400
2023-11-21 17:34:59,044 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:35:08,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1596046.6666666667, ans=0.1
2023-11-21 17:35:10,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1596046.6666666667, ans=0.0
2023-11-21 17:35:10,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1596046.6666666667, ans=0.025
2023-11-21 17:35:14,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0
2023-11-21 17:35:25,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=1596113.3333333333, ans=0.05
2023-11-21 17:35:30,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1596113.3333333333, ans=0.2
2023-11-21 17:35:32,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.79 vs. limit=15.0
2023-11-21 17:35:48,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1596246.6666666667, ans=0.125
2023-11-21 17:36:00,722 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11000, loss[loss=0.07421, simple_loss=0.09814, pruned_loss=0.01709, audio_tagging_loss=0.008045, over 15284.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09526, pruned_loss=0.01608, audio_tagging_loss=0.00953, over 3045998.46 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:36:02,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239450
2023-11-21 17:36:07,230 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:36:11,853 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:36:40,595 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.031e+01 8.533e+01 9.347e+01 2.206e+02, threshold=1.707e+02, percent-clipped=1.0
2023-11-21 17:36:43,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1596513.3333333333, ans=0.125
2023-11-21 17:36:44,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1596513.3333333333, ans=0.125
2023-11-21 17:36:54,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1596580.0, ans=0.125
2023-11-21 17:37:06,728 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11050, loss[loss=0.08647, simple_loss=0.1173, pruned_loss=0.02026, audio_tagging_loss=0.007579, over 16265.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09487, pruned_loss=0.01621, audio_tagging_loss=0.009615, over 3043837.01 frames. ], batch size: 61, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:37:08,085 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239500
2023-11-21 17:37:21,142 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:37:33,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1596780.0, ans=0.0
2023-11-21 17:37:44,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1596846.6666666667, ans=0.1
2023-11-21 17:38:11,235 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11100, loss[loss=0.07729, simple_loss=0.1032, pruned_loss=0.01594, audio_tagging_loss=0.009746, over 16000.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09604, pruned_loss=0.01631, audio_tagging_loss=0.009704, over 3047427.99 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:38:12,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239550
2023-11-21 17:38:19,129 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.27 vs. limit=10.0
2023-11-21 17:38:27,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1597046.6666666667, ans=0.125
2023-11-21 17:38:32,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1597046.6666666667, ans=0.2
2023-11-21 17:38:45,175 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.87 vs. limit=15.0
2023-11-21 17:38:48,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1597180.0, ans=0.04949747468305833
2023-11-21 17:38:49,490 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.177e+01 8.722e+01 9.312e+01 1.121e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 17:38:53,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1597180.0, ans=0.0
2023-11-21 17:39:14,664 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11150, loss[loss=0.07918, simple_loss=0.1032, pruned_loss=0.01932, audio_tagging_loss=0.008257, over 15424.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09666, pruned_loss=0.01658, audio_tagging_loss=0.009754, over 3050083.19 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:39:15,932 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239600
2023-11-21 17:39:22,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1597313.3333333333, ans=0.125
2023-11-21 17:39:48,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1597446.6666666667, ans=0.125
2023-11-21 17:40:00,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1597513.3333333333, ans=0.1
2023-11-21 17:40:19,210 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11200, loss[loss=0.06884, simple_loss=0.09078, pruned_loss=0.0125, audio_tagging_loss=0.01095, over 15698.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09547, pruned_loss=0.01625, audio_tagging_loss=0.00989, over 3056040.22 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:40:20,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239650
2023-11-21 17:40:23,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1597646.6666666667, ans=0.125
2023-11-21 17:40:34,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1597713.3333333333, ans=0.2
2023-11-21 17:40:48,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1597780.0, ans=0.125
2023-11-21 17:40:58,065 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.528e+01 8.115e+01 8.678e+01 9.817e+01 1.232e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 17:41:23,469 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11250, loss[loss=0.07584, simple_loss=0.0976, pruned_loss=0.01737, audio_tagging_loss=0.009669, over 16246.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.095, pruned_loss=0.0163, audio_tagging_loss=0.009785, over 3056119.05 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:41:24,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239700
2023-11-21 17:41:36,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1598046.6666666667, ans=0.125
2023-11-21 17:42:04,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1598180.0, ans=0.2
2023-11-21 17:42:08,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1598180.0, ans=0.125
2023-11-21 17:42:27,205 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11300, loss[loss=0.06988, simple_loss=0.09204, pruned_loss=0.01463, audio_tagging_loss=0.00923, over 14581.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09519, pruned_loss=0.01625, audio_tagging_loss=0.009635, over 3047123.65 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:42:28,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239750
2023-11-21 17:42:28,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1598313.3333333333, ans=0.125
2023-11-21 17:42:32,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1598313.3333333333, ans=0.125
2023-11-21 17:42:37,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1598313.3333333333, ans=0.125
2023-11-21 17:42:42,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1598380.0, ans=0.125
2023-11-21 17:43:01,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1598446.6666666667, ans=0.1
2023-11-21 17:43:06,116 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.360e+01 8.263e+01 8.887e+01 9.511e+01 1.252e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 17:43:11,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.42 vs. limit=15.0
2023-11-21 17:43:30,680 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11350, loss[loss=0.07446, simple_loss=0.1082, pruned_loss=0.01356, audio_tagging_loss=0.006802, over 15336.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09644, pruned_loss=0.01629, audio_tagging_loss=0.009478, over 3047253.47 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:43:31,982 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239800
2023-11-21 17:43:38,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1598646.6666666667, ans=0.0
2023-11-21 17:43:39,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1598646.6666666667, ans=0.1
2023-11-21 17:44:03,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1598780.0, ans=0.125
2023-11-21 17:44:05,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1598780.0, ans=0.1
2023-11-21 17:44:33,775 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11400, loss[loss=0.076, simple_loss=0.1128, pruned_loss=0.01337, audio_tagging_loss=0.006245, over 15197.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09661, pruned_loss=0.01643, audio_tagging_loss=0.00945, over 3044286.92 frames. ], batch size: 53, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:44:35,075 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239850
2023-11-21 17:44:41,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1598980.0, ans=0.0
2023-11-21 17:45:10,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1599180.0, ans=0.2
2023-11-21 17:45:11,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-21 17:45:13,346 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.120e+01 8.500e+01 9.528e+01 1.320e+02, threshold=1.700e+02, percent-clipped=0.0
2023-11-21 17:45:37,181 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11450, loss[loss=0.0673, simple_loss=0.09278, pruned_loss=0.01421, audio_tagging_loss=0.006694, over 15457.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09549, pruned_loss=0.01628, audio_tagging_loss=0.009435, over 3042819.32 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:45:37,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1599313.3333333333, ans=0.5
2023-11-21 17:45:37,889 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.48 vs. limit=15.0
2023-11-21 17:45:38,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239900
2023-11-21 17:45:51,659 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.08 vs. limit=22.5
2023-11-21 17:46:25,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten.whitening_limit, batch_count=1599513.3333333333, ans=22.5
2023-11-21 17:46:40,484 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11500, loss[loss=0.07694, simple_loss=0.09919, pruned_loss=0.01593, audio_tagging_loss=0.01141, over 15296.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09509, pruned_loss=0.01615, audio_tagging_loss=0.009455, over 3038213.84 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:46:41,773 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 239950
2023-11-21 17:46:45,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1599646.6666666667, ans=0.0
2023-11-21 17:46:46,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1599646.6666666667, ans=0.0
2023-11-21 17:46:58,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1599713.3333333333, ans=0.1
2023-11-21 17:47:18,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.75 vs. limit=15.0
2023-11-21 17:47:19,135 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 7.879e+01 8.671e+01 9.205e+01 1.101e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 17:47:43,798 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11550, loss[loss=0.05302, simple_loss=0.06431, pruned_loss=0.008328, audio_tagging_loss=0.01254, over 15745.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.09551, pruned_loss=0.01621, audio_tagging_loss=0.009478, over 3039902.58 frames. ], batch size: 61, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:47:45,074 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240000
2023-11-21 17:47:46,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1599980.0, ans=0.0
2023-11-21 17:48:15,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1600113.3333333333, ans=0.95
2023-11-21 17:48:19,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1600113.3333333333, ans=0.2
2023-11-21 17:48:24,243 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:48:25,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1600180.0, ans=0.1
2023-11-21 17:48:42,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1600246.6666666667, ans=0.125
2023-11-21 17:48:50,299 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11600, loss[loss=0.06371, simple_loss=0.08779, pruned_loss=0.01183, audio_tagging_loss=0.007986, over 15783.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09562, pruned_loss=0.0162, audio_tagging_loss=0.009504, over 3039964.58 frames. ], batch size: 58, lr: 3.38e-03, grad_scale: 32.0
2023-11-21 17:48:51,593 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240050
2023-11-21 17:48:56,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1600313.3333333333, ans=0.125
2023-11-21 17:49:26,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1600446.6666666667, ans=0.125
2023-11-21 17:49:29,614 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.173e+01 8.910e+01 9.625e+01 1.220e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 17:49:31,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1600513.3333333333, ans=0.125
2023-11-21 17:49:54,066 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11650, loss[loss=0.06065, simple_loss=0.07593, pruned_loss=0.01387, audio_tagging_loss=0.008818, over 15124.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09577, pruned_loss=0.01619, audio_tagging_loss=0.00949, over 3043653.85 frames. ], batch size: 57, lr: 3.38e-03, grad_scale: 32.0
2023-11-21 17:49:55,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240100
2023-11-21 17:49:56,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1600646.6666666667, ans=0.125
2023-11-21 17:50:27,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1600780.0, ans=0.2
2023-11-21 17:50:57,977 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11700, loss[loss=0.07942, simple_loss=0.1002, pruned_loss=0.01896, audio_tagging_loss=0.01035, over 16015.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09554, pruned_loss=0.01607, audio_tagging_loss=0.009573, over 3045631.48 frames. ], batch size: 60, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:50:59,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240150
2023-11-21 17:50:59,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1600980.0, ans=0.125
2023-11-21 17:51:02,415 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.06 vs. limit=15.0
2023-11-21 17:51:06,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1600980.0, ans=0.125
2023-11-21 17:51:09,790 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.82 vs. limit=15.0
2023-11-21 17:51:15,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1601046.6666666667, ans=0.1
2023-11-21 17:51:19,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1601046.6666666667, ans=0.0
2023-11-21 17:51:38,211 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.589e+01 8.116e+01 8.836e+01 9.594e+01 1.189e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 17:51:41,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.09 vs. limit=15.0
2023-11-21 17:51:45,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1601180.0, ans=0.1
2023-11-21 17:51:50,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1601246.6666666667, ans=0.125
2023-11-21 17:51:56,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1601246.6666666667, ans=0.125
2023-11-21 17:52:00,595 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11750, loss[loss=0.08735, simple_loss=0.1111, pruned_loss=0.02063, audio_tagging_loss=0.01115, over 15738.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09563, pruned_loss=0.01621, audio_tagging_loss=0.009568, over 3046595.77 frames. ], batch size: 59, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:52:01,932 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240200
2023-11-21 17:52:07,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1601313.3333333333, ans=0.0
2023-11-21 17:52:10,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1601313.3333333333, ans=0.125
2023-11-21 17:52:12,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1601380.0, ans=0.125
2023-11-21 17:52:38,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1601513.3333333333, ans=0.025
2023-11-21 17:52:45,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1601513.3333333333, ans=0.1
2023-11-21 17:53:03,890 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11800, loss[loss=0.07149, simple_loss=0.08945, pruned_loss=0.01842, audio_tagging_loss=0.008348, over 14389.00 frames. ], tot_loss[loss=0.07372, simple_loss=0.09541, pruned_loss=0.01643, audio_tagging_loss=0.009584, over 3039285.07 frames. ], batch size: 53, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:53:05,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240250
2023-11-21 17:53:19,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1601713.3333333333, ans=0.125
2023-11-21 17:53:19,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1601713.3333333333, ans=0.125
2023-11-21 17:53:31,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1601780.0, ans=0.125
2023-11-21 17:53:36,261 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.20 vs. limit=6.0
2023-11-21 17:53:42,306 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.76 vs. limit=15.0
2023-11-21 17:53:44,071 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.430e+01 8.055e+01 8.902e+01 9.602e+01 1.321e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 17:53:49,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1601846.6666666667, ans=0.125
2023-11-21 17:54:02,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1601913.3333333333, ans=0.125
2023-11-21 17:54:07,654 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11850, loss[loss=0.06002, simple_loss=0.07823, pruned_loss=0.01143, audio_tagging_loss=0.009476, over 14652.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09601, pruned_loss=0.01651, audio_tagging_loss=0.009754, over 3044351.07 frames. ], batch size: 56, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:54:08,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240300
2023-11-21 17:54:12,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1601980.0, ans=0.0
2023-11-21 17:54:29,164 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.64 vs. limit=15.0
2023-11-21 17:54:32,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1602113.3333333333, ans=0.0
2023-11-21 17:54:37,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1602113.3333333333, ans=0.1
2023-11-21 17:54:51,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1602180.0, ans=0.0
2023-11-21 17:55:02,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1602246.6666666667, ans=0.125
2023-11-21 17:55:11,006 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11900, loss[loss=0.07102, simple_loss=0.08881, pruned_loss=0.01326, audio_tagging_loss=0.01335, over 14819.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09513, pruned_loss=0.01618, audio_tagging_loss=0.009781, over 3049170.82 frames. ], batch size: 58, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:55:11,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1602313.3333333333, ans=0.125
2023-11-21 17:55:12,294 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240350
2023-11-21 17:55:12,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1602313.3333333333, ans=0.125
2023-11-21 17:55:17,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1602313.3333333333, ans=0.2
2023-11-21 17:55:19,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1602313.3333333333, ans=0.125
2023-11-21 17:55:19,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1602313.3333333333, ans=0.125
2023-11-21 17:55:34,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1602380.0, ans=0.0
2023-11-21 17:55:35,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1602446.6666666667, ans=0.125
2023-11-21 17:55:52,200 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 7.826e+01 8.536e+01 9.227e+01 1.162e+02, threshold=1.707e+02, percent-clipped=0.0
2023-11-21 17:56:03,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1602580.0, ans=0.125
2023-11-21 17:56:14,810 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 11950, loss[loss=0.06356, simple_loss=0.08471, pruned_loss=0.01254, audio_tagging_loss=0.008668, over 15452.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09492, pruned_loss=0.01599, audio_tagging_loss=0.009828, over 3044468.13 frames. ], batch size: 56, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:56:16,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240400
2023-11-21 17:56:20,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1602646.6666666667, ans=0.0
2023-11-21 17:56:25,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.41 vs. limit=15.0
2023-11-21 17:56:34,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1602713.3333333333, ans=0.125
2023-11-21 17:56:52,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1602846.6666666667, ans=0.1
2023-11-21 17:57:16,252 INFO [train_asr.py:1221] (2/4) Epoch 20, batch 12000, loss[loss=0.06818, simple_loss=0.09162, pruned_loss=0.01433, audio_tagging_loss=0.008044, over 14817.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09465, pruned_loss=0.01591, audio_tagging_loss=0.009906, over 3046217.69 frames. ], batch size: 55, lr: 3.38e-03, grad_scale: 32.0
2023-11-21 17:57:16,253 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 17:57:43,781 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3468, 5.0235, 4.7584, 5.1912], device='cuda:2')
2023-11-21 17:57:57,979 INFO [train_asr.py:1253] (2/4) Epoch 20, validation: loss=0.06015, simple_loss=0.05214, pruned_loss=0.005246, audio_tagging_loss=0.02884, over 4681554.00 frames. 
2023-11-21 17:57:57,980 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 17:57:59,200 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240450
2023-11-21 17:58:07,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1602980.0, ans=0.125
2023-11-21 17:58:07,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1602980.0, ans=0.125
2023-11-21 17:58:11,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1603046.6666666667, ans=0.1
2023-11-21 17:58:58,668 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 0, loss[loss=0.08195, simple_loss=0.08869, pruned_loss=0.01152, audio_tagging_loss=0.02609, over 15280.00 frames. ], tot_loss[loss=0.08195, simple_loss=0.08869, pruned_loss=0.01152, audio_tagging_loss=0.02609, over 15280.00 frames. ], batch size: 57, lr: 3.30e-03, grad_scale: 32.0
2023-11-21 17:58:58,669 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 17:59:34,269 INFO [train_asr.py:1253] (2/4) Epoch 21, validation: loss=0.05942, simple_loss=0.05208, pruned_loss=0.00519, audio_tagging_loss=0.02819, over 4681554.00 frames. 
2023-11-21 17:59:34,270 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 17:59:46,184 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.163e+01 8.736e+01 1.018e+02 1.443e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 17:59:57,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1603200.0, ans=0.125
2023-11-21 18:00:04,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1603266.6666666667, ans=0.125
2023-11-21 18:00:11,001 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240500
2023-11-21 18:00:15,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1603333.3333333333, ans=0.125
2023-11-21 18:00:25,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1603400.0, ans=0.0
2023-11-21 18:00:28,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1603400.0, ans=0.04949747468305833
2023-11-21 18:00:28,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1603400.0, ans=0.1
2023-11-21 18:00:39,136 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 50, loss[loss=0.09173, simple_loss=0.1119, pruned_loss=0.02099, audio_tagging_loss=0.01479, over 15477.00 frames. ], tot_loss[loss=0.08254, simple_loss=0.09603, pruned_loss=0.01634, audio_tagging_loss=0.01818, over 687906.84 frames. ], batch size: 55, lr: 3.30e-03, grad_scale: 32.0
2023-11-21 18:01:12,060 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.21 vs. limit=10.0
2023-11-21 18:01:15,153 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240550
2023-11-21 18:01:43,925 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 100, loss[loss=0.0838, simple_loss=0.1046, pruned_loss=0.0173, audio_tagging_loss=0.01418, over 14659.00 frames. ], tot_loss[loss=0.08075, simple_loss=0.09478, pruned_loss=0.01576, audio_tagging_loss=0.0176, over 1208047.13 frames. ], batch size: 57, lr: 3.30e-03, grad_scale: 32.0
2023-11-21 18:01:54,767 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.709e+01 8.663e+01 9.262e+01 1.008e+02 1.245e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-21 18:02:09,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1603933.3333333333, ans=0.125
2023-11-21 18:02:19,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240600
2023-11-21 18:02:47,860 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 150, loss[loss=0.05715, simple_loss=0.06121, pruned_loss=0.009645, audio_tagging_loss=0.01689, over 14719.00 frames. ], tot_loss[loss=0.07843, simple_loss=0.0934, pruned_loss=0.0157, audio_tagging_loss=0.01602, over 1611253.90 frames. ], batch size: 60, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:02:55,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1604133.3333333333, ans=0.0
2023-11-21 18:02:57,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1604133.3333333333, ans=0.125
2023-11-21 18:03:20,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1604266.6666666667, ans=0.125
2023-11-21 18:03:24,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240650
2023-11-21 18:03:52,006 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 200, loss[loss=0.08928, simple_loss=0.1109, pruned_loss=0.02358, audio_tagging_loss=0.01025, over 15233.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09421, pruned_loss=0.0158, audio_tagging_loss=0.01411, over 1932388.25 frames. ], batch size: 56, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:03:59,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=1604466.6666666667, ans=0.1
2023-11-21 18:04:03,967 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.168e+01 8.831e+01 9.644e+01 1.295e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 18:04:07,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1604533.3333333333, ans=0.0
2023-11-21 18:04:27,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240700
2023-11-21 18:04:28,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=15.0
2023-11-21 18:04:32,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1604666.6666666667, ans=0.2
2023-11-21 18:04:39,793 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.51 vs. limit=15.0
2023-11-21 18:04:44,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1604733.3333333333, ans=0.125
2023-11-21 18:04:44,629 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.88 vs. limit=15.0
2023-11-21 18:04:49,411 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.21 vs. limit=15.0
2023-11-21 18:04:50,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1604733.3333333333, ans=0.1
2023-11-21 18:04:55,607 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 250, loss[loss=0.07584, simple_loss=0.09919, pruned_loss=0.01778, audio_tagging_loss=0.008464, over 14872.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09553, pruned_loss=0.01629, audio_tagging_loss=0.01257, over 2174798.63 frames. ], batch size: 57, lr: 3.30e-03, grad_scale: 8.0
2023-11-21 18:05:05,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1604800.0, ans=0.125
2023-11-21 18:05:08,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.31 vs. limit=6.0
2023-11-21 18:05:31,457 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240750
2023-11-21 18:05:54,022 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.31 vs. limit=15.0
2023-11-21 18:05:59,436 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 300, loss[loss=0.08466, simple_loss=0.1104, pruned_loss=0.02255, audio_tagging_loss=0.006922, over 15785.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09553, pruned_loss=0.0162, audio_tagging_loss=0.01172, over 2370382.56 frames. ], batch size: 55, lr: 3.30e-03, grad_scale: 8.0
2023-11-21 18:05:59,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1605133.3333333333, ans=0.09899494936611666
2023-11-21 18:06:00,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1605133.3333333333, ans=0.125
2023-11-21 18:06:10,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1605133.3333333333, ans=0.125
2023-11-21 18:06:14,093 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.194e+01 8.148e+01 8.758e+01 9.398e+01 1.354e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 18:06:20,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1605200.0, ans=0.2
2023-11-21 18:06:21,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1605200.0, ans=0.125
2023-11-21 18:06:35,651 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240800
2023-11-21 18:06:38,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1605333.3333333333, ans=0.0
2023-11-21 18:06:45,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1605333.3333333333, ans=0.2
2023-11-21 18:06:48,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1605333.3333333333, ans=0.125
2023-11-21 18:07:02,082 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.96 vs. limit=15.0
2023-11-21 18:07:04,351 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 350, loss[loss=0.06041, simple_loss=0.08401, pruned_loss=0.008361, audio_tagging_loss=0.01005, over 15038.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.09657, pruned_loss=0.01647, audio_tagging_loss=0.01105, over 2520445.08 frames. ], batch size: 56, lr: 3.30e-03, grad_scale: 8.0
2023-11-21 18:07:28,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1605600.0, ans=0.125
2023-11-21 18:07:39,919 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240850
2023-11-21 18:07:46,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1605666.6666666667, ans=0.0
2023-11-21 18:07:52,686 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.14 vs. limit=10.0
2023-11-21 18:07:58,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1605733.3333333333, ans=0.5
2023-11-21 18:08:08,095 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 400, loss[loss=0.08252, simple_loss=0.1067, pruned_loss=0.02031, audio_tagging_loss=0.008849, over 14665.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09634, pruned_loss=0.01637, audio_tagging_loss=0.01058, over 2639543.93 frames. ], batch size: 56, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:08:22,787 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.012e+01 8.744e+01 9.405e+01 1.302e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 18:08:29,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1605866.6666666667, ans=0.05
2023-11-21 18:08:44,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240900
2023-11-21 18:08:48,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1606000.0, ans=0.2
2023-11-21 18:09:09,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1606066.6666666667, ans=0.125
2023-11-21 18:09:12,841 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 450, loss[loss=0.07545, simple_loss=0.09572, pruned_loss=0.01845, audio_tagging_loss=0.009139, over 15206.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09572, pruned_loss=0.0164, audio_tagging_loss=0.01037, over 2724995.23 frames. ], batch size: 57, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:09:35,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1606200.0, ans=0.0
2023-11-21 18:09:41,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1606266.6666666667, ans=0.125
2023-11-21 18:09:49,246 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 240950
2023-11-21 18:10:17,046 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 500, loss[loss=0.08691, simple_loss=0.114, pruned_loss=0.022, audio_tagging_loss=0.007902, over 15917.00 frames. ], tot_loss[loss=0.07517, simple_loss=0.0968, pruned_loss=0.01664, audio_tagging_loss=0.01013, over 2798720.13 frames. ], batch size: 56, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:10:19,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1606466.6666666667, ans=0.125
2023-11-21 18:10:31,499 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.140e+01 8.109e+01 8.628e+01 9.380e+01 1.335e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 18:10:53,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241000
2023-11-21 18:10:57,342 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.34 vs. limit=22.5
2023-11-21 18:11:00,816 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.04 vs. limit=15.0
2023-11-21 18:11:13,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1606733.3333333333, ans=0.125
2023-11-21 18:11:19,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.59 vs. limit=12.0
2023-11-21 18:11:22,066 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 550, loss[loss=0.07048, simple_loss=0.09332, pruned_loss=0.01198, audio_tagging_loss=0.01184, over 14590.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09627, pruned_loss=0.01648, audio_tagging_loss=0.01008, over 2849705.97 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:11:26,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1606800.0, ans=0.1
2023-11-21 18:11:43,723 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.27 vs. limit=15.0
2023-11-21 18:11:57,689 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.35 vs. limit=15.0
2023-11-21 18:11:58,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241050
2023-11-21 18:12:00,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1607000.0, ans=0.125
2023-11-21 18:12:21,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1607066.6666666667, ans=0.2
2023-11-21 18:12:22,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1607066.6666666667, ans=0.125
2023-11-21 18:12:25,556 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 600, loss[loss=0.06104, simple_loss=0.08121, pruned_loss=0.01334, audio_tagging_loss=0.007089, over 15518.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09608, pruned_loss=0.01645, audio_tagging_loss=0.009972, over 2899273.69 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:12:37,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1607200.0, ans=0.05
2023-11-21 18:12:39,617 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.804e+01 8.139e+01 8.699e+01 9.445e+01 1.159e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 18:13:01,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241100
2023-11-21 18:13:13,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1607333.3333333333, ans=0.2
2023-11-21 18:13:13,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1607333.3333333333, ans=0.0
2023-11-21 18:13:26,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1607400.0, ans=0.1
2023-11-21 18:13:27,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1607400.0, ans=0.1
2023-11-21 18:13:30,510 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 650, loss[loss=0.08333, simple_loss=0.1091, pruned_loss=0.02081, audio_tagging_loss=0.007962, over 14922.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09607, pruned_loss=0.01658, audio_tagging_loss=0.009851, over 2932251.92 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:13:52,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1607533.3333333333, ans=0.125
2023-11-21 18:14:05,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241150
2023-11-21 18:14:05,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1607600.0, ans=0.05
2023-11-21 18:14:12,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1607666.6666666667, ans=0.2
2023-11-21 18:14:17,705 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.17 vs. limit=22.5
2023-11-21 18:14:23,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1607733.3333333333, ans=0.125
2023-11-21 18:14:26,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1607733.3333333333, ans=0.125
2023-11-21 18:14:34,276 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 700, loss[loss=0.07541, simple_loss=0.1002, pruned_loss=0.01692, audio_tagging_loss=0.008373, over 15029.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09585, pruned_loss=0.01623, audio_tagging_loss=0.009803, over 2960137.99 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:14:41,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1607800.0, ans=0.05
2023-11-21 18:14:48,423 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.012e+01 8.519e+01 9.265e+01 1.175e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 18:15:10,790 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241200
2023-11-21 18:15:20,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1608000.0, ans=0.07
2023-11-21 18:15:23,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1608000.0, ans=0.125
2023-11-21 18:15:26,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1608066.6666666667, ans=0.2
2023-11-21 18:15:33,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.79 vs. limit=12.0
2023-11-21 18:15:39,231 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 750, loss[loss=0.05528, simple_loss=0.06596, pruned_loss=0.01342, audio_tagging_loss=0.008873, over 14951.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09602, pruned_loss=0.0163, audio_tagging_loss=0.009753, over 2982455.91 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:15:43,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.64 vs. limit=22.5
2023-11-21 18:15:58,851 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.21 vs. limit=15.0
2023-11-21 18:16:07,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1608266.6666666667, ans=0.125
2023-11-21 18:16:14,954 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241250
2023-11-21 18:16:27,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1608333.3333333333, ans=0.1
2023-11-21 18:16:42,241 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 800, loss[loss=0.09518, simple_loss=0.1153, pruned_loss=0.02728, audio_tagging_loss=0.01025, over 14580.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09608, pruned_loss=0.01617, audio_tagging_loss=0.009787, over 3001791.44 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:16:55,777 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.30 vs. limit=6.0
2023-11-21 18:16:57,473 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.159e+01 8.981e+01 9.751e+01 1.353e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 18:17:02,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1608533.3333333333, ans=0.125
2023-11-21 18:17:10,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1608600.0, ans=0.1
2023-11-21 18:17:18,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1608600.0, ans=0.2
2023-11-21 18:17:18,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241300
2023-11-21 18:17:29,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1608666.6666666667, ans=0.0
2023-11-21 18:17:31,733 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.64 vs. limit=15.0
2023-11-21 18:17:36,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.98 vs. limit=22.5
2023-11-21 18:17:47,505 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 850, loss[loss=0.0605, simple_loss=0.08289, pruned_loss=0.01087, audio_tagging_loss=0.008179, over 15563.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09762, pruned_loss=0.01647, audio_tagging_loss=0.009759, over 3020552.81 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:18:06,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1608866.6666666667, ans=0.1
2023-11-21 18:18:08,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1608866.6666666667, ans=0.0
2023-11-21 18:18:13,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1608933.3333333333, ans=0.0
2023-11-21 18:18:19,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1608933.3333333333, ans=0.125
2023-11-21 18:18:19,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1608933.3333333333, ans=0.0
2023-11-21 18:18:23,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241350
2023-11-21 18:18:33,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1609000.0, ans=0.0
2023-11-21 18:18:36,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.02 vs. limit=15.0
2023-11-21 18:18:49,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1609066.6666666667, ans=0.125
2023-11-21 18:18:52,216 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 900, loss[loss=0.06724, simple_loss=0.09352, pruned_loss=0.01243, audio_tagging_loss=0.008047, over 15229.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.0977, pruned_loss=0.01639, audio_tagging_loss=0.009815, over 3025916.74 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:19:00,132 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.15 vs. limit=22.5
2023-11-21 18:19:05,367 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.217e+01 8.787e+01 9.484e+01 1.241e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 18:19:05,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1609200.0, ans=0.5
2023-11-21 18:19:09,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.09 vs. limit=15.0
2023-11-21 18:19:27,883 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241400
2023-11-21 18:19:49,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1609400.0, ans=0.125
2023-11-21 18:19:53,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1609400.0, ans=0.1
2023-11-21 18:19:55,361 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 950, loss[loss=0.06697, simple_loss=0.08728, pruned_loss=0.01228, audio_tagging_loss=0.01106, over 15206.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09743, pruned_loss=0.01646, audio_tagging_loss=0.009728, over 3027462.20 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:20:12,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.70 vs. limit=15.0
2023-11-21 18:20:25,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1609600.0, ans=0.0
2023-11-21 18:20:31,557 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241450
2023-11-21 18:20:42,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0
2023-11-21 18:21:00,401 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1000, loss[loss=0.06213, simple_loss=0.0769, pruned_loss=0.01256, audio_tagging_loss=0.01112, over 16404.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09753, pruned_loss=0.01644, audio_tagging_loss=0.009542, over 3038795.99 frames. ], batch size: 61, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:21:15,639 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.205e+01 8.818e+01 9.430e+01 1.136e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 18:21:17,229 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.790e-02
2023-11-21 18:21:26,841 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:21:35,613 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241500
2023-11-21 18:22:02,535 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:22:04,709 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1050, loss[loss=0.07814, simple_loss=0.1029, pruned_loss=0.01802, audio_tagging_loss=0.008653, over 15722.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09719, pruned_loss=0.0165, audio_tagging_loss=0.009479, over 3034212.93 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:22:10,363 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.69 vs. limit=15.0
2023-11-21 18:22:28,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1610266.6666666667, ans=0.125
2023-11-21 18:22:40,857 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241550
2023-11-21 18:23:08,024 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1100, loss[loss=0.07149, simple_loss=0.09325, pruned_loss=0.01664, audio_tagging_loss=0.008226, over 15227.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.0968, pruned_loss=0.01636, audio_tagging_loss=0.009415, over 3038978.16 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:23:10,525 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:23:13,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1610466.6666666667, ans=0.1
2023-11-21 18:23:15,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1610466.6666666667, ans=0.125
2023-11-21 18:23:15,223 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:23:23,976 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.562e+01 7.887e+01 8.672e+01 9.418e+01 1.186e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 18:23:25,996 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.41 vs. limit=10.0
2023-11-21 18:23:34,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1610600.0, ans=0.1
2023-11-21 18:23:35,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1610600.0, ans=0.125
2023-11-21 18:23:39,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1610600.0, ans=0.0
2023-11-21 18:23:41,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1610600.0, ans=0.125
2023-11-21 18:23:44,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241600
2023-11-21 18:23:48,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1610666.6666666667, ans=0.0
2023-11-21 18:23:56,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1610666.6666666667, ans=0.0
2023-11-21 18:24:00,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1610733.3333333333, ans=0.125
2023-11-21 18:24:12,792 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1150, loss[loss=0.08254, simple_loss=0.1048, pruned_loss=0.02036, audio_tagging_loss=0.00979, over 15610.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09657, pruned_loss=0.01635, audio_tagging_loss=0.00936, over 3043248.84 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:24:31,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1610866.6666666667, ans=0.0
2023-11-21 18:24:42,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1610933.3333333333, ans=0.0
2023-11-21 18:24:47,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1610933.3333333333, ans=0.0
2023-11-21 18:24:48,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241650
2023-11-21 18:25:04,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1611066.6666666667, ans=0.1
2023-11-21 18:25:06,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1611066.6666666667, ans=0.0
2023-11-21 18:25:17,314 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1200, loss[loss=0.05634, simple_loss=0.071, pruned_loss=0.01082, audio_tagging_loss=0.01002, over 15803.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09504, pruned_loss=0.01603, audio_tagging_loss=0.009344, over 3045681.53 frames. ], batch size: 60, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:25:17,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1611133.3333333333, ans=0.2
2023-11-21 18:25:33,024 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.048e+01 8.609e+01 9.236e+01 1.274e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 18:25:40,969 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.02 vs. limit=10.0
2023-11-21 18:25:45,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=15.0
2023-11-21 18:25:52,521 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241700
2023-11-21 18:25:57,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1611333.3333333333, ans=0.125
2023-11-21 18:26:17,323 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:26:20,654 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1250, loss[loss=0.05981, simple_loss=0.07529, pruned_loss=0.01286, audio_tagging_loss=0.009307, over 14604.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09486, pruned_loss=0.01602, audio_tagging_loss=0.009271, over 3039247.60 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:26:23,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.69 vs. limit=15.0
2023-11-21 18:26:24,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1611466.6666666667, ans=0.125
2023-11-21 18:26:26,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1611466.6666666667, ans=0.125
2023-11-21 18:26:31,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1611466.6666666667, ans=0.125
2023-11-21 18:26:48,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1611600.0, ans=0.0
2023-11-21 18:26:51,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1611600.0, ans=0.125
2023-11-21 18:26:57,048 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241750
2023-11-21 18:26:58,900 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0
2023-11-21 18:27:09,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1611666.6666666667, ans=0.1
2023-11-21 18:27:25,089 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1300, loss[loss=0.07692, simple_loss=0.1022, pruned_loss=0.01625, audio_tagging_loss=0.009583, over 14724.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09528, pruned_loss=0.01619, audio_tagging_loss=0.00927, over 3043893.76 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:27:41,453 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.105e+01 8.749e+01 9.378e+01 1.268e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 18:27:58,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1611933.3333333333, ans=0.125
2023-11-21 18:28:00,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241800
2023-11-21 18:28:10,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.56 vs. limit=15.0
2023-11-21 18:28:15,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1612066.6666666667, ans=0.0
2023-11-21 18:28:26,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1612066.6666666667, ans=0.1
2023-11-21 18:28:28,411 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1350, loss[loss=0.07817, simple_loss=0.1071, pruned_loss=0.01618, audio_tagging_loss=0.008447, over 15468.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09566, pruned_loss=0.01623, audio_tagging_loss=0.009242, over 3046907.03 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:29:05,057 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241850
2023-11-21 18:29:12,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1612333.3333333333, ans=0.125
2023-11-21 18:29:14,755 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:29:32,994 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1400, loss[loss=0.08812, simple_loss=0.1227, pruned_loss=0.01936, audio_tagging_loss=0.007397, over 15116.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.0958, pruned_loss=0.01635, audio_tagging_loss=0.009419, over 3050200.06 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:29:38,870 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.05 vs. limit=15.0
2023-11-21 18:29:50,232 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.505e+01 9.180e+01 9.942e+01 1.333e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-21 18:29:57,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1612600.0, ans=0.125
2023-11-21 18:29:59,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1612600.0, ans=0.1
2023-11-21 18:30:09,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241900
2023-11-21 18:30:22,864 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.12 vs. limit=12.0
2023-11-21 18:30:37,671 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1450, loss[loss=0.06072, simple_loss=0.07994, pruned_loss=0.01177, audio_tagging_loss=0.008977, over 14786.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09508, pruned_loss=0.0162, audio_tagging_loss=0.009561, over 3047147.58 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:30:42,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1612800.0, ans=0.0
2023-11-21 18:30:48,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1612866.6666666667, ans=0.125
2023-11-21 18:30:57,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1612866.6666666667, ans=0.1
2023-11-21 18:31:02,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1612933.3333333333, ans=0.125
2023-11-21 18:31:13,419 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 241950
2023-11-21 18:31:16,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1613000.0, ans=0.0
2023-11-21 18:31:25,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.53 vs. limit=15.0
2023-11-21 18:31:27,334 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.05 vs. limit=15.0
2023-11-21 18:31:41,555 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1500, loss[loss=0.06808, simple_loss=0.07718, pruned_loss=0.01528, audio_tagging_loss=0.01421, over 15206.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.09424, pruned_loss=0.01622, audio_tagging_loss=0.009672, over 3042442.78 frames. ], batch size: 59, lr: 3.29e-03, grad_scale: 8.0
2023-11-21 18:31:58,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1613200.0, ans=0.1
2023-11-21 18:31:59,578 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.238e+01 7.980e+01 8.647e+01 9.591e+01 1.272e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-21 18:32:01,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1613200.0, ans=0.125
2023-11-21 18:32:02,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1613200.0, ans=0.125
2023-11-21 18:32:13,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1613266.6666666667, ans=0.0
2023-11-21 18:32:18,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242000
2023-11-21 18:32:46,312 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1550, loss[loss=0.04538, simple_loss=0.05462, pruned_loss=0.005181, audio_tagging_loss=0.01289, over 14000.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09363, pruned_loss=0.01606, audio_tagging_loss=0.009834, over 3039380.27 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 8.0
2023-11-21 18:32:54,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1613466.6666666667, ans=0.0
2023-11-21 18:32:57,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1613533.3333333333, ans=0.1
2023-11-21 18:32:58,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1613533.3333333333, ans=0.125
2023-11-21 18:33:00,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1613533.3333333333, ans=0.125
2023-11-21 18:33:02,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1613533.3333333333, ans=0.025
2023-11-21 18:33:22,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242050
2023-11-21 18:33:42,810 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.46 vs. limit=15.0
2023-11-21 18:33:44,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.91 vs. limit=15.0
2023-11-21 18:33:50,190 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1600, loss[loss=0.06325, simple_loss=0.08633, pruned_loss=0.01077, audio_tagging_loss=0.009312, over 14556.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09478, pruned_loss=0.01631, audio_tagging_loss=0.009858, over 3033476.38 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:33:54,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0
2023-11-21 18:34:07,796 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.028e+01 8.717e+01 9.586e+01 1.995e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 18:34:17,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1613933.3333333333, ans=0.2
2023-11-21 18:34:26,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242100
2023-11-21 18:34:26,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.12 vs. limit=22.5
2023-11-21 18:34:54,119 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1650, loss[loss=0.07191, simple_loss=0.09906, pruned_loss=0.01403, audio_tagging_loss=0.008353, over 14552.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09406, pruned_loss=0.01609, audio_tagging_loss=0.009912, over 3041884.75 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:35:16,900 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.99 vs. limit=15.0
2023-11-21 18:35:20,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1614266.6666666667, ans=0.2
2023-11-21 18:35:31,165 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242150
2023-11-21 18:35:46,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1614400.0, ans=0.1
2023-11-21 18:35:58,534 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1700, loss[loss=0.05812, simple_loss=0.0642, pruned_loss=0.0116, audio_tagging_loss=0.01442, over 14681.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09479, pruned_loss=0.01625, audio_tagging_loss=0.009891, over 3041786.90 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:36:00,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1614466.6666666667, ans=0.125
2023-11-21 18:36:10,565 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.14 vs. limit=15.0
2023-11-21 18:36:12,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1614533.3333333333, ans=0.1
2023-11-21 18:36:13,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.17 vs. limit=15.0
2023-11-21 18:36:16,629 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.384e+01 8.955e+01 9.684e+01 1.332e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 18:36:28,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1614600.0, ans=0.0
2023-11-21 18:36:34,505 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242200
2023-11-21 18:36:34,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1614600.0, ans=0.025
2023-11-21 18:36:38,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1614666.6666666667, ans=0.125
2023-11-21 18:36:52,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.84 vs. limit=15.0
2023-11-21 18:37:02,898 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1750, loss[loss=0.07392, simple_loss=0.1056, pruned_loss=0.0134, audio_tagging_loss=0.007718, over 15416.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.0945, pruned_loss=0.01625, audio_tagging_loss=0.009819, over 3042717.18 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:37:21,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1614866.6666666667, ans=0.04949747468305833
2023-11-21 18:37:27,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1614933.3333333333, ans=0.125
2023-11-21 18:37:28,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.78 vs. limit=15.0
2023-11-21 18:37:38,629 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242250
2023-11-21 18:37:38,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1614933.3333333333, ans=0.0
2023-11-21 18:37:40,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1615000.0, ans=0.0
2023-11-21 18:37:46,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1615000.0, ans=0.2
2023-11-21 18:37:46,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=1615000.0, ans=0.2
2023-11-21 18:37:48,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1615000.0, ans=0.2
2023-11-21 18:38:00,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1615066.6666666667, ans=0.2
2023-11-21 18:38:07,275 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1800, loss[loss=0.04584, simple_loss=0.05394, pruned_loss=0.00791, audio_tagging_loss=0.01096, over 14974.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09473, pruned_loss=0.01619, audio_tagging_loss=0.00972, over 3041153.56 frames. ], batch size: 59, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:38:19,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1615200.0, ans=0.0
2023-11-21 18:38:25,235 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.166e+01 8.773e+01 9.532e+01 1.071e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 18:38:29,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1615200.0, ans=0.125
2023-11-21 18:38:29,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1615200.0, ans=0.125
2023-11-21 18:38:38,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.20 vs. limit=15.0
2023-11-21 18:38:39,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1615266.6666666667, ans=0.125
2023-11-21 18:38:39,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1615266.6666666667, ans=0.2
2023-11-21 18:38:43,721 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242300
2023-11-21 18:38:45,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1615333.3333333333, ans=0.125
2023-11-21 18:38:48,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1615333.3333333333, ans=0.07
2023-11-21 18:38:56,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1615333.3333333333, ans=0.0
2023-11-21 18:39:11,199 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1850, loss[loss=0.06032, simple_loss=0.08035, pruned_loss=0.0116, audio_tagging_loss=0.008545, over 14963.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09491, pruned_loss=0.01596, audio_tagging_loss=0.009652, over 3042972.28 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:39:18,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1615466.6666666667, ans=0.0
2023-11-21 18:39:31,502 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.08 vs. limit=15.0
2023-11-21 18:39:37,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1615600.0, ans=0.125
2023-11-21 18:39:37,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1615600.0, ans=0.125
2023-11-21 18:39:43,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1615600.0, ans=0.125
2023-11-21 18:39:45,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.12 vs. limit=15.0
2023-11-21 18:39:47,083 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242350
2023-11-21 18:40:03,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1615733.3333333333, ans=0.125
2023-11-21 18:40:06,232 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=3.606e-02
2023-11-21 18:40:14,375 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1900, loss[loss=0.0638, simple_loss=0.08606, pruned_loss=0.01001, audio_tagging_loss=0.01076, over 14692.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09505, pruned_loss=0.01606, audio_tagging_loss=0.009568, over 3045815.15 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:40:18,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.51 vs. limit=22.5
2023-11-21 18:40:22,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1615800.0, ans=0.1
2023-11-21 18:40:33,142 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.159e+01 8.678e+01 9.357e+01 1.521e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 18:40:50,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242400
2023-11-21 18:40:54,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.83 vs. limit=15.0
2023-11-21 18:41:18,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.64 vs. limit=15.0
2023-11-21 18:41:19,209 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 1950, loss[loss=0.04926, simple_loss=0.06404, pruned_loss=0.008065, audio_tagging_loss=0.009172, over 14667.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.095, pruned_loss=0.01595, audio_tagging_loss=0.009409, over 3047841.73 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:41:48,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1616266.6666666667, ans=0.125
2023-11-21 18:41:54,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242450
2023-11-21 18:42:21,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1616400.0, ans=0.125
2023-11-21 18:42:23,261 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2000, loss[loss=0.07464, simple_loss=0.1025, pruned_loss=0.01525, audio_tagging_loss=0.008138, over 14605.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09487, pruned_loss=0.01612, audio_tagging_loss=0.009411, over 3049590.57 frames. ], batch size: 53, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:42:40,263 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 7.872e+01 8.647e+01 9.317e+01 2.008e+02, threshold=1.729e+02, percent-clipped=1.0
2023-11-21 18:42:46,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1616533.3333333333, ans=0.125
2023-11-21 18:42:58,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242500
2023-11-21 18:43:01,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1616666.6666666667, ans=0.125
2023-11-21 18:43:02,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1616666.6666666667, ans=0.125
2023-11-21 18:43:16,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1616733.3333333333, ans=0.2
2023-11-21 18:43:24,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.68 vs. limit=22.5
2023-11-21 18:43:26,311 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2050, loss[loss=0.08024, simple_loss=0.1102, pruned_loss=0.01739, audio_tagging_loss=0.007768, over 15692.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09541, pruned_loss=0.01618, audio_tagging_loss=0.0093, over 3042808.53 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:43:45,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1616866.6666666667, ans=0.0
2023-11-21 18:43:46,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1616866.6666666667, ans=0.2
2023-11-21 18:43:47,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1616866.6666666667, ans=0.0
2023-11-21 18:43:51,874 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:44:03,243 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242550
2023-11-21 18:44:11,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1617000.0, ans=0.0
2023-11-21 18:44:31,776 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2100, loss[loss=0.06872, simple_loss=0.0869, pruned_loss=0.01573, audio_tagging_loss=0.009541, over 16234.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09517, pruned_loss=0.01622, audio_tagging_loss=0.009279, over 3048089.90 frames. ], batch size: 61, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:44:50,753 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 8.232e+01 8.682e+01 9.296e+01 1.860e+02, threshold=1.736e+02, percent-clipped=1.0
2023-11-21 18:45:04,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1617266.6666666667, ans=0.125
2023-11-21 18:45:06,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242600
2023-11-21 18:45:14,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1617333.3333333333, ans=0.0
2023-11-21 18:45:36,222 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2150, loss[loss=0.06428, simple_loss=0.07906, pruned_loss=0.01417, audio_tagging_loss=0.01058, over 14212.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09529, pruned_loss=0.01629, audio_tagging_loss=0.009215, over 3048200.47 frames. ], batch size: 54, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:45:43,797 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:45:45,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1617466.6666666667, ans=0.0
2023-11-21 18:45:45,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.48 vs. limit=12.0
2023-11-21 18:45:46,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1617466.6666666667, ans=0.07
2023-11-21 18:46:12,469 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242650
2023-11-21 18:46:14,889 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:46:39,532 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2200, loss[loss=0.06964, simple_loss=0.1001, pruned_loss=0.01228, audio_tagging_loss=0.00729, over 15858.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09584, pruned_loss=0.01632, audio_tagging_loss=0.009255, over 3047656.77 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:46:41,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=15.0
2023-11-21 18:46:58,468 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0
2023-11-21 18:46:58,820 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.003e+01 8.660e+01 9.579e+01 1.643e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 18:46:59,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1617866.6666666667, ans=0.0
2023-11-21 18:47:01,314 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.26 vs. limit=22.5
2023-11-21 18:47:04,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1617933.3333333333, ans=0.07
2023-11-21 18:47:15,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242700
2023-11-21 18:47:32,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.15 vs. limit=22.5
2023-11-21 18:47:32,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1618066.6666666667, ans=0.0
2023-11-21 18:47:39,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.01 vs. limit=12.0
2023-11-21 18:47:43,671 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2250, loss[loss=0.06608, simple_loss=0.08413, pruned_loss=0.01548, audio_tagging_loss=0.008531, over 16908.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09519, pruned_loss=0.0163, audio_tagging_loss=0.009404, over 3049619.62 frames. ], batch size: 65, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:47:47,962 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.20 vs. limit=15.0
2023-11-21 18:48:10,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1618266.6666666667, ans=0.125
2023-11-21 18:48:13,177 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.73 vs. limit=15.0
2023-11-21 18:48:18,716 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242750
2023-11-21 18:48:31,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1618333.3333333333, ans=0.125
2023-11-21 18:48:34,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1618400.0, ans=0.2
2023-11-21 18:48:44,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1618400.0, ans=15.0
2023-11-21 18:48:47,271 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2300, loss[loss=0.07756, simple_loss=0.1035, pruned_loss=0.01453, audio_tagging_loss=0.01127, over 15410.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09545, pruned_loss=0.01625, audio_tagging_loss=0.00952, over 3051352.36 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:48:48,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1618466.6666666667, ans=10.0
2023-11-21 18:48:55,465 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:48:55,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1618466.6666666667, ans=15.0
2023-11-21 18:48:56,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1618466.6666666667, ans=0.0
2023-11-21 18:49:06,243 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.387e+01 7.925e+01 8.512e+01 9.086e+01 1.194e+02, threshold=1.702e+02, percent-clipped=0.0
2023-11-21 18:49:21,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=1618600.0, ans=0.2
2023-11-21 18:49:23,565 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242800
2023-11-21 18:49:27,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1618666.6666666667, ans=10.0
2023-11-21 18:49:33,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1618666.6666666667, ans=0.0
2023-11-21 18:49:42,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1618733.3333333333, ans=0.1
2023-11-21 18:49:44,489 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:49:44,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1618733.3333333333, ans=0.1
2023-11-21 18:49:48,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1618733.3333333333, ans=0.1
2023-11-21 18:49:51,761 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2350, loss[loss=0.08356, simple_loss=0.1153, pruned_loss=0.01823, audio_tagging_loss=0.007677, over 16062.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09656, pruned_loss=0.01628, audio_tagging_loss=0.009444, over 3052808.47 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:49:58,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1618800.0, ans=0.0
2023-11-21 18:50:12,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1618866.6666666667, ans=0.0
2023-11-21 18:50:28,235 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242850
2023-11-21 18:50:28,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1618933.3333333333, ans=0.2
2023-11-21 18:50:38,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1619000.0, ans=0.0
2023-11-21 18:50:46,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1619066.6666666667, ans=0.125
2023-11-21 18:50:46,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1619066.6666666667, ans=0.09899494936611666
2023-11-21 18:50:47,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1619066.6666666667, ans=0.125
2023-11-21 18:50:56,101 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2400, loss[loss=0.04533, simple_loss=0.05379, pruned_loss=0.005503, audio_tagging_loss=0.01293, over 16277.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.0955, pruned_loss=0.01601, audio_tagging_loss=0.009657, over 3061906.62 frames. ], batch size: 62, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:50:58,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1619133.3333333333, ans=0.1
2023-11-21 18:51:15,445 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.343e+01 8.783e+01 9.778e+01 1.689e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 18:51:20,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1619266.6666666667, ans=0.0
2023-11-21 18:51:30,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1619266.6666666667, ans=22.5
2023-11-21 18:51:31,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242900
2023-11-21 18:51:36,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1619333.3333333333, ans=0.125
2023-11-21 18:51:38,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1619333.3333333333, ans=0.125
2023-11-21 18:51:47,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1619400.0, ans=0.0
2023-11-21 18:51:52,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1619400.0, ans=0.0
2023-11-21 18:51:59,516 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2450, loss[loss=0.08951, simple_loss=0.1028, pruned_loss=0.02388, audio_tagging_loss=0.01421, over 14632.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09529, pruned_loss=0.01615, audio_tagging_loss=0.009695, over 3053109.65 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:52:29,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1619600.0, ans=0.1
2023-11-21 18:52:36,218 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 242950
2023-11-21 18:52:36,526 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:52:46,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1619666.6666666667, ans=0.1
2023-11-21 18:52:57,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1619733.3333333333, ans=0.05
2023-11-21 18:52:57,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1619733.3333333333, ans=0.1
2023-11-21 18:52:58,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1619733.3333333333, ans=0.125
2023-11-21 18:53:02,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1619800.0, ans=0.1
2023-11-21 18:53:03,992 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2500, loss[loss=0.08519, simple_loss=0.1153, pruned_loss=0.01815, audio_tagging_loss=0.009418, over 14934.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09527, pruned_loss=0.01614, audio_tagging_loss=0.009797, over 3043154.93 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:53:10,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.79 vs. limit=22.5
2023-11-21 18:53:16,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1619866.6666666667, ans=0.2
2023-11-21 18:53:20,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1619866.6666666667, ans=0.125
2023-11-21 18:53:23,259 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.882e+01 8.197e+01 8.814e+01 9.365e+01 1.246e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 18:53:31,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.22 vs. limit=10.0
2023-11-21 18:53:39,837 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243000
2023-11-21 18:54:05,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1620066.6666666667, ans=0.125
2023-11-21 18:54:08,160 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2550, loss[loss=0.06776, simple_loss=0.08355, pruned_loss=0.0166, audio_tagging_loss=0.009377, over 14200.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09517, pruned_loss=0.01611, audio_tagging_loss=0.009669, over 3047466.44 frames. ], batch size: 54, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:54:44,354 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243050
2023-11-21 18:54:55,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1620333.3333333333, ans=0.125
2023-11-21 18:55:06,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1620400.0, ans=0.125
2023-11-21 18:55:10,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1620400.0, ans=0.2
2023-11-21 18:55:12,128 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2600, loss[loss=0.08152, simple_loss=0.1131, pruned_loss=0.01837, audio_tagging_loss=0.006615, over 15286.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09527, pruned_loss=0.01627, audio_tagging_loss=0.009455, over 3045372.26 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:55:32,970 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.728e+01 8.396e+01 8.905e+01 9.438e+01 1.443e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 18:55:34,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1620533.3333333333, ans=0.2
2023-11-21 18:55:39,858 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0
2023-11-21 18:55:45,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1620600.0, ans=0.125
2023-11-21 18:55:48,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243100
2023-11-21 18:55:58,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1620666.6666666667, ans=0.1
2023-11-21 18:56:01,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1620666.6666666667, ans=0.0
2023-11-21 18:56:03,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1620733.3333333333, ans=0.125
2023-11-21 18:56:16,584 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2650, loss[loss=0.08539, simple_loss=0.1025, pruned_loss=0.02177, audio_tagging_loss=0.01236, over 15609.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09494, pruned_loss=0.01628, audio_tagging_loss=0.00948, over 3038252.56 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 18:56:17,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1620800.0, ans=0.0
2023-11-21 18:56:18,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.35 vs. limit=12.0
2023-11-21 18:56:38,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1620866.6666666667, ans=0.0
2023-11-21 18:56:52,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243150
2023-11-21 18:56:58,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1621000.0, ans=0.125
2023-11-21 18:56:59,075 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.36 vs. limit=12.0
2023-11-21 18:57:06,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1621066.6666666667, ans=0.0
2023-11-21 18:57:19,994 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2700, loss[loss=0.05792, simple_loss=0.06924, pruned_loss=0.01057, audio_tagging_loss=0.01274, over 15360.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09475, pruned_loss=0.01611, audio_tagging_loss=0.009498, over 3045610.36 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 18:57:41,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 7.974e+01 8.822e+01 9.240e+01 1.303e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 18:57:51,344 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=12.0
2023-11-21 18:57:56,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243200
2023-11-21 18:58:02,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1621333.3333333333, ans=0.125
2023-11-21 18:58:17,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1621400.0, ans=0.2
2023-11-21 18:58:24,262 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2750, loss[loss=0.07621, simple_loss=0.1068, pruned_loss=0.01568, audio_tagging_loss=0.007124, over 14988.00 frames. ], tot_loss[loss=0.07311, simple_loss=0.0953, pruned_loss=0.01606, audio_tagging_loss=0.009399, over 3046177.30 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 18:58:32,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1621466.6666666667, ans=0.125
2023-11-21 18:58:52,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1621600.0, ans=0.125
2023-11-21 18:59:00,238 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243250
2023-11-21 18:59:00,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.33 vs. limit=22.5
2023-11-21 18:59:17,904 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:59:28,123 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2800, loss[loss=0.03319, simple_loss=0.03874, pruned_loss=0.005016, audio_tagging_loss=0.008807, over 14559.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09471, pruned_loss=0.01598, audio_tagging_loss=0.009419, over 3050210.48 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:59:30,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1621800.0, ans=0.125
2023-11-21 18:59:50,356 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 7.876e+01 8.639e+01 9.327e+01 1.140e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 18:59:56,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1621933.3333333333, ans=0.2
2023-11-21 19:00:03,987 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243300
2023-11-21 19:00:05,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1622000.0, ans=0.0
2023-11-21 19:00:11,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1622000.0, ans=0.0
2023-11-21 19:00:16,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1622000.0, ans=0.0
2023-11-21 19:00:19,152 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:00:31,564 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2850, loss[loss=0.06859, simple_loss=0.0903, pruned_loss=0.0135, audio_tagging_loss=0.009942, over 15352.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09417, pruned_loss=0.01581, audio_tagging_loss=0.00938, over 3048589.14 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:00:36,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1622133.3333333333, ans=0.125
2023-11-21 19:01:06,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243350
2023-11-21 19:01:10,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1622333.3333333333, ans=0.125
2023-11-21 19:01:10,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1622333.3333333333, ans=0.1
2023-11-21 19:01:24,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1622400.0, ans=0.0
2023-11-21 19:01:34,244 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2900, loss[loss=0.05661, simple_loss=0.07838, pruned_loss=0.01041, audio_tagging_loss=0.007003, over 14610.00 frames. ], tot_loss[loss=0.07153, simple_loss=0.09312, pruned_loss=0.0155, audio_tagging_loss=0.009477, over 3047372.17 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:01:48,926 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.79 vs. limit=15.0
2023-11-21 19:01:55,217 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.905e+01 8.105e+01 8.663e+01 9.336e+01 1.251e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 19:02:09,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243400
2023-11-21 19:02:38,003 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 2950, loss[loss=0.07652, simple_loss=0.1015, pruned_loss=0.01584, audio_tagging_loss=0.009941, over 14449.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09432, pruned_loss=0.01566, audio_tagging_loss=0.009454, over 3048412.01 frames. ], batch size: 54, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:02:42,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1622800.0, ans=0.125
2023-11-21 19:02:42,222 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.53 vs. limit=15.0
2023-11-21 19:03:02,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1622933.3333333333, ans=0.125
2023-11-21 19:03:11,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1622933.3333333333, ans=0.2
2023-11-21 19:03:13,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243450
2023-11-21 19:03:19,608 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.58 vs. limit=22.5
2023-11-21 19:03:29,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1623066.6666666667, ans=0.125
2023-11-21 19:03:41,269 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3000, loss[loss=0.05361, simple_loss=0.06175, pruned_loss=0.008925, audio_tagging_loss=0.01381, over 14615.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09449, pruned_loss=0.01575, audio_tagging_loss=0.009602, over 3048220.55 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:03:41,269 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 19:04:17,256 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8042, 4.9113, 4.9984, 4.8499], device='cuda:2')
2023-11-21 19:04:24,770 INFO [train_asr.py:1253] (2/4) Epoch 21, validation: loss=0.0594, simple_loss=0.05205, pruned_loss=0.005197, audio_tagging_loss=0.02817, over 4681554.00 frames. 
2023-11-21 19:04:24,771 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 19:04:40,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1623200.0, ans=0.1
2023-11-21 19:04:40,858 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.65 vs. limit=22.5
2023-11-21 19:04:47,417 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.111e+01 8.701e+01 9.542e+01 1.276e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 19:04:55,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1623266.6666666667, ans=0.125
2023-11-21 19:05:00,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243500
2023-11-21 19:05:02,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1623333.3333333333, ans=0.0
2023-11-21 19:05:10,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0
2023-11-21 19:05:28,940 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3050, loss[loss=0.09161, simple_loss=0.1183, pruned_loss=0.0229, audio_tagging_loss=0.009574, over 14645.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09446, pruned_loss=0.01569, audio_tagging_loss=0.00963, over 3050088.86 frames. ], batch size: 54, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:06:05,520 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:06:05,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243550
2023-11-21 19:06:08,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.56 vs. limit=10.0
2023-11-21 19:06:12,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1623666.6666666667, ans=0.125
2023-11-21 19:06:32,843 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3100, loss[loss=0.0752, simple_loss=0.09564, pruned_loss=0.01668, audio_tagging_loss=0.0107, over 16137.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09505, pruned_loss=0.01608, audio_tagging_loss=0.009716, over 3052870.82 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:06:51,297 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.26 vs. limit=15.0
2023-11-21 19:06:56,504 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.034e+01 8.558e+01 9.488e+01 1.458e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 19:07:09,256 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243600
2023-11-21 19:07:12,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1624000.0, ans=0.015
2023-11-21 19:07:20,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1624000.0, ans=0.0
2023-11-21 19:07:24,795 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.30 vs. limit=10.0
2023-11-21 19:07:38,450 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3150, loss[loss=0.08678, simple_loss=0.1214, pruned_loss=0.01804, audio_tagging_loss=0.008025, over 15988.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09543, pruned_loss=0.01622, audio_tagging_loss=0.009794, over 3052960.93 frames. ], batch size: 60, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:07:43,469 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:07:57,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1624200.0, ans=0.0
2023-11-21 19:08:13,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243650
2023-11-21 19:08:34,654 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.48 vs. limit=15.0
2023-11-21 19:08:42,945 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3200, loss[loss=0.05652, simple_loss=0.07765, pruned_loss=0.009714, audio_tagging_loss=0.007981, over 14269.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09588, pruned_loss=0.01631, audio_tagging_loss=0.009792, over 3057403.86 frames. ], batch size: 54, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:09:04,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.386e+01 8.002e+01 8.673e+01 9.467e+01 1.702e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 19:09:15,922 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.300e-02
2023-11-21 19:09:18,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243700
2023-11-21 19:09:33,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1624733.3333333333, ans=0.0
2023-11-21 19:09:42,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1624733.3333333333, ans=0.125
2023-11-21 19:09:45,227 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3250, loss[loss=0.0715, simple_loss=0.08923, pruned_loss=0.01627, audio_tagging_loss=0.01061, over 15300.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09552, pruned_loss=0.01618, audio_tagging_loss=0.009822, over 3055488.01 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:10:20,907 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243750
2023-11-21 19:10:22,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1625000.0, ans=0.125
2023-11-21 19:10:48,916 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3300, loss[loss=0.05074, simple_loss=0.0683, pruned_loss=0.009067, audio_tagging_loss=0.007519, over 14423.00 frames. ], tot_loss[loss=0.07423, simple_loss=0.09604, pruned_loss=0.01637, audio_tagging_loss=0.009845, over 3049642.44 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:10:57,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1625133.3333333333, ans=0.0
2023-11-21 19:10:59,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1625133.3333333333, ans=0.09899494936611666
2023-11-21 19:11:11,996 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 8.264e+01 8.839e+01 9.527e+01 1.267e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 19:11:19,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1625266.6666666667, ans=0.1
2023-11-21 19:11:24,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243800
2023-11-21 19:11:53,925 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3350, loss[loss=0.07351, simple_loss=0.09315, pruned_loss=0.01685, audio_tagging_loss=0.01008, over 15175.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09614, pruned_loss=0.01632, audio_tagging_loss=0.009734, over 3052735.41 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:12:07,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1625533.3333333333, ans=0.125
2023-11-21 19:12:18,214 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.76 vs. limit=15.0
2023-11-21 19:12:20,639 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.67 vs. limit=15.0
2023-11-21 19:12:29,501 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243850
2023-11-21 19:12:38,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1625666.6666666667, ans=0.0
2023-11-21 19:12:54,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.15 vs. limit=10.0
2023-11-21 19:12:55,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1625733.3333333333, ans=0.125
2023-11-21 19:12:57,940 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3400, loss[loss=0.07088, simple_loss=0.09904, pruned_loss=0.01447, audio_tagging_loss=0.006892, over 15652.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09665, pruned_loss=0.01646, audio_tagging_loss=0.009657, over 3055577.14 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:12:59,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1625800.0, ans=0.125
2023-11-21 19:13:15,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1625866.6666666667, ans=15.0
2023-11-21 19:13:17,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1625866.6666666667, ans=0.125
2023-11-21 19:13:20,637 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.826e+01 8.123e+01 8.713e+01 9.440e+01 3.289e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 19:13:24,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1625933.3333333333, ans=0.2
2023-11-21 19:13:34,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243900
2023-11-21 19:13:38,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1626000.0, ans=0.125
2023-11-21 19:14:01,819 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3450, loss[loss=0.06578, simple_loss=0.08358, pruned_loss=0.01589, audio_tagging_loss=0.008101, over 14968.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09668, pruned_loss=0.01648, audio_tagging_loss=0.00956, over 3053396.76 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:14:02,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1626133.3333333333, ans=0.5
2023-11-21 19:14:30,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1626266.6666666667, ans=0.0
2023-11-21 19:14:34,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.69 vs. limit=15.0
2023-11-21 19:14:37,963 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 243950
2023-11-21 19:14:38,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1626266.6666666667, ans=0.0
2023-11-21 19:14:48,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1626333.3333333333, ans=0.04949747468305833
2023-11-21 19:14:51,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-21 19:15:04,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1626400.0, ans=0.125
2023-11-21 19:15:05,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.35 vs. limit=22.5
2023-11-21 19:15:06,974 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3500, loss[loss=0.0933, simple_loss=0.1226, pruned_loss=0.02474, audio_tagging_loss=0.007247, over 15815.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09729, pruned_loss=0.01646, audio_tagging_loss=0.009445, over 3050083.71 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:15:09,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1626466.6666666667, ans=0.0
2023-11-21 19:15:29,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.309e+01 8.089e+01 8.726e+01 9.622e+01 1.634e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 19:15:37,850 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:15:41,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1626600.0, ans=0.125
2023-11-21 19:15:42,696 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244000
2023-11-21 19:15:59,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1626666.6666666667, ans=0.0
2023-11-21 19:16:03,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1626733.3333333333, ans=0.125
2023-11-21 19:16:08,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1626733.3333333333, ans=0.0
2023-11-21 19:16:09,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1626733.3333333333, ans=0.125
2023-11-21 19:16:13,968 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3550, loss[loss=0.07872, simple_loss=0.1021, pruned_loss=0.01813, audio_tagging_loss=0.009563, over 14949.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09632, pruned_loss=0.0164, audio_tagging_loss=0.009458, over 3047623.02 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:16:15,546 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:16:51,052 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244050
2023-11-21 19:17:07,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1627066.6666666667, ans=0.0
2023-11-21 19:17:18,062 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3600, loss[loss=0.0743, simple_loss=0.09424, pruned_loss=0.01807, audio_tagging_loss=0.009109, over 15568.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09625, pruned_loss=0.01646, audio_tagging_loss=0.009417, over 3046942.57 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:17:26,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1627133.3333333333, ans=0.125
2023-11-21 19:17:29,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1627133.3333333333, ans=0.0
2023-11-21 19:17:42,711 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.963e+01 8.176e+01 8.799e+01 9.565e+01 1.406e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 19:17:49,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1627266.6666666667, ans=0.2
2023-11-21 19:17:53,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244100
2023-11-21 19:18:00,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1627333.3333333333, ans=0.125
2023-11-21 19:18:21,754 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3650, loss[loss=0.07285, simple_loss=0.0905, pruned_loss=0.01564, audio_tagging_loss=0.01196, over 16343.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09607, pruned_loss=0.01636, audio_tagging_loss=0.009375, over 3058605.89 frames. ], batch size: 63, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:18:36,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1627533.3333333333, ans=0.5
2023-11-21 19:18:49,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1627600.0, ans=0.125
2023-11-21 19:18:51,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1627600.0, ans=0.125
2023-11-21 19:18:57,929 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244150
2023-11-21 19:19:17,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.17 vs. limit=15.0
2023-11-21 19:19:24,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1627733.3333333333, ans=0.2
2023-11-21 19:19:26,330 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3700, loss[loss=0.07125, simple_loss=0.08887, pruned_loss=0.01662, audio_tagging_loss=0.0102, over 15804.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09576, pruned_loss=0.01633, audio_tagging_loss=0.009325, over 3056454.18 frames. ], batch size: 62, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:19:31,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1627800.0, ans=0.0
2023-11-21 19:19:44,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.64 vs. limit=12.0
2023-11-21 19:19:48,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1627866.6666666667, ans=0.0
2023-11-21 19:19:50,828 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.276e+01 8.879e+01 9.943e+01 1.926e+02, threshold=1.776e+02, percent-clipped=1.0
2023-11-21 19:19:53,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1627933.3333333333, ans=0.125
2023-11-21 19:20:02,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244200
2023-11-21 19:20:05,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1628000.0, ans=0.0
2023-11-21 19:20:08,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1628000.0, ans=0.125
2023-11-21 19:20:12,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1628000.0, ans=0.04949747468305833
2023-11-21 19:20:17,658 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.70 vs. limit=10.0
2023-11-21 19:20:18,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1628066.6666666667, ans=0.0
2023-11-21 19:20:26,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.51 vs. limit=15.0
2023-11-21 19:20:26,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1628066.6666666667, ans=10.0
2023-11-21 19:20:30,379 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3750, loss[loss=0.09167, simple_loss=0.1264, pruned_loss=0.01976, audio_tagging_loss=0.0087, over 16049.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.0961, pruned_loss=0.0164, audio_tagging_loss=0.009363, over 3058378.53 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:21:02,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1628266.6666666667, ans=0.125
2023-11-21 19:21:02,481 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:21:07,011 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244250
2023-11-21 19:21:11,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1628333.3333333333, ans=0.125
2023-11-21 19:21:14,280 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:21:35,247 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3800, loss[loss=0.08465, simple_loss=0.111, pruned_loss=0.02111, audio_tagging_loss=0.008026, over 15060.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09639, pruned_loss=0.01636, audio_tagging_loss=0.009462, over 3054246.15 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:21:43,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1628466.6666666667, ans=0.0
2023-11-21 19:21:57,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1628533.3333333333, ans=0.0
2023-11-21 19:21:59,381 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.245e+01 8.842e+01 9.701e+01 1.541e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 19:22:00,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1628600.0, ans=0.1
2023-11-21 19:22:10,658 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244300
2023-11-21 19:22:11,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1628666.6666666667, ans=0.125
2023-11-21 19:22:13,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1628666.6666666667, ans=0.2
2023-11-21 19:22:18,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.22 vs. limit=6.0
2023-11-21 19:22:38,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1628800.0, ans=0.09899494936611666
2023-11-21 19:22:39,728 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3850, loss[loss=0.06207, simple_loss=0.07554, pruned_loss=0.01181, audio_tagging_loss=0.01248, over 15926.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09698, pruned_loss=0.01641, audio_tagging_loss=0.009478, over 3057511.05 frames. ], batch size: 59, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:22:42,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.39 vs. limit=22.5
2023-11-21 19:22:54,650 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:23:06,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1628933.3333333333, ans=0.1
2023-11-21 19:23:10,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1628933.3333333333, ans=0.0
2023-11-21 19:23:15,509 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244350
2023-11-21 19:23:22,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1629000.0, ans=0.2
2023-11-21 19:23:35,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1629066.6666666667, ans=0.125
2023-11-21 19:23:43,358 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3900, loss[loss=0.06949, simple_loss=0.09729, pruned_loss=0.01316, audio_tagging_loss=0.007683, over 15625.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09582, pruned_loss=0.01634, audio_tagging_loss=0.009628, over 3050496.44 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:23:43,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1629133.3333333333, ans=0.1
2023-11-21 19:23:44,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1629133.3333333333, ans=0.0
2023-11-21 19:23:59,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1629200.0, ans=0.0
2023-11-21 19:24:08,371 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.034e+01 8.736e+01 9.358e+01 1.610e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 19:24:20,171 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244400
2023-11-21 19:24:28,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1629333.3333333333, ans=0.125
2023-11-21 19:24:32,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.35 vs. limit=15.0
2023-11-21 19:24:48,561 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 3950, loss[loss=0.0837, simple_loss=0.1044, pruned_loss=0.01917, audio_tagging_loss=0.01233, over 14573.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09491, pruned_loss=0.01615, audio_tagging_loss=0.009798, over 3054962.85 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:25:06,165 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:25:18,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1629600.0, ans=0.0
2023-11-21 19:25:23,593 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244450
2023-11-21 19:25:36,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1629666.6666666667, ans=10.0
2023-11-21 19:25:52,435 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4000, loss[loss=0.09321, simple_loss=0.1248, pruned_loss=0.02489, audio_tagging_loss=0.005935, over 14420.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09696, pruned_loss=0.01692, audio_tagging_loss=0.009755, over 3063392.14 frames. ], batch size: 53, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:25:58,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1629800.0, ans=0.2
2023-11-21 19:26:13,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.50 vs. limit=15.0
2023-11-21 19:26:16,055 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.271e+01 8.886e+01 9.884e+01 1.294e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 19:26:28,138 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244500
2023-11-21 19:26:30,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1630000.0, ans=0.125
2023-11-21 19:26:51,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1630066.6666666667, ans=0.125
2023-11-21 19:26:56,260 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4050, loss[loss=0.06935, simple_loss=0.09335, pruned_loss=0.01166, audio_tagging_loss=0.01102, over 15320.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09745, pruned_loss=0.01692, audio_tagging_loss=0.009762, over 3056991.56 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:26:58,663 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:27:14,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1630200.0, ans=0.125
2023-11-21 19:27:32,284 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244550
2023-11-21 19:27:41,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1630333.3333333333, ans=0.2
2023-11-21 19:27:45,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1630333.3333333333, ans=0.1
2023-11-21 19:27:58,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1630400.0, ans=0.0
2023-11-21 19:28:00,859 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4100, loss[loss=0.06383, simple_loss=0.09036, pruned_loss=0.01022, audio_tagging_loss=0.008432, over 15094.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.09634, pruned_loss=0.01661, audio_tagging_loss=0.009755, over 3042623.50 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:28:24,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.whiten.whitening_limit, batch_count=1630533.3333333333, ans=12.0
2023-11-21 19:28:25,946 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.257e+01 8.384e+01 8.914e+01 9.573e+01 1.296e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 19:28:26,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1630600.0, ans=0.0
2023-11-21 19:28:30,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1630600.0, ans=0.125
2023-11-21 19:28:36,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244600
2023-11-21 19:28:52,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.75 vs. limit=15.0
2023-11-21 19:29:06,009 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4150, loss[loss=0.07467, simple_loss=0.1027, pruned_loss=0.01703, audio_tagging_loss=0.006277, over 15459.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09635, pruned_loss=0.01658, audio_tagging_loss=0.009633, over 3049025.49 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:29:21,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1630866.6666666667, ans=0.1
2023-11-21 19:29:21,092 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:29:28,592 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.69 vs. limit=15.0
2023-11-21 19:29:42,543 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244650
2023-11-21 19:29:53,037 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:30:05,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1631066.6666666667, ans=0.0
2023-11-21 19:30:06,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1631066.6666666667, ans=0.125
2023-11-21 19:30:10,067 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4200, loss[loss=0.06297, simple_loss=0.07972, pruned_loss=0.01212, audio_tagging_loss=0.01099, over 15679.00 frames. ], tot_loss[loss=0.07411, simple_loss=0.09644, pruned_loss=0.01643, audio_tagging_loss=0.009459, over 3048939.07 frames. ], batch size: 61, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:30:30,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1631200.0, ans=0.125
2023-11-21 19:30:32,996 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:30:36,268 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 7.875e+01 8.455e+01 9.386e+01 1.251e+02, threshold=1.691e+02, percent-clipped=0.0
2023-11-21 19:30:46,927 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244700
2023-11-21 19:31:00,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.02 vs. limit=22.5
2023-11-21 19:31:08,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1631400.0, ans=0.0
2023-11-21 19:31:12,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1631400.0, ans=0.09899494936611666
2023-11-21 19:31:15,167 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4250, loss[loss=0.07818, simple_loss=0.09482, pruned_loss=0.01879, audio_tagging_loss=0.01198, over 14801.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09724, pruned_loss=0.01653, audio_tagging_loss=0.00932, over 3050453.89 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:31:18,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1631466.6666666667, ans=0.1
2023-11-21 19:31:22,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1631466.6666666667, ans=0.2
2023-11-21 19:31:50,286 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244750
2023-11-21 19:31:52,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1631666.6666666667, ans=0.125
2023-11-21 19:32:01,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1631666.6666666667, ans=0.0
2023-11-21 19:32:18,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1631800.0, ans=0.1
2023-11-21 19:32:19,219 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4300, loss[loss=0.0918, simple_loss=0.1245, pruned_loss=0.02034, audio_tagging_loss=0.009222, over 15056.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.09792, pruned_loss=0.01673, audio_tagging_loss=0.009371, over 3050439.80 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:32:20,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1631800.0, ans=0.2
2023-11-21 19:32:35,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1631866.6666666667, ans=0.125
2023-11-21 19:32:37,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1631866.6666666667, ans=0.125
2023-11-21 19:32:41,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1631866.6666666667, ans=0.125
2023-11-21 19:32:43,221 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.395e+01 8.866e+01 9.565e+01 1.376e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 19:32:55,048 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244800
2023-11-21 19:33:08,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1632000.0, ans=0.1
2023-11-21 19:33:13,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1632066.6666666667, ans=0.1
2023-11-21 19:33:16,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1632066.6666666667, ans=0.2
2023-11-21 19:33:19,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1632066.6666666667, ans=0.125
2023-11-21 19:33:22,511 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4350, loss[loss=0.06796, simple_loss=0.09436, pruned_loss=0.01312, audio_tagging_loss=0.007655, over 15646.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09772, pruned_loss=0.01656, audio_tagging_loss=0.009315, over 3053780.73 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:33:44,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1632200.0, ans=0.0
2023-11-21 19:33:44,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1632200.0, ans=0.125
2023-11-21 19:33:48,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1632266.6666666667, ans=0.2
2023-11-21 19:33:58,889 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244850
2023-11-21 19:34:02,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=1632333.3333333333, ans=0.05
2023-11-21 19:34:25,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1632466.6666666667, ans=0.2
2023-11-21 19:34:27,327 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4400, loss[loss=0.08629, simple_loss=0.1072, pruned_loss=0.02223, audio_tagging_loss=0.01044, over 15962.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09751, pruned_loss=0.01658, audio_tagging_loss=0.009336, over 3051525.10 frames. ], batch size: 61, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:34:30,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1632466.6666666667, ans=0.1
2023-11-21 19:34:52,768 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.144e+01 8.724e+01 9.531e+01 1.294e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 19:35:02,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244900
2023-11-21 19:35:05,771 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-21 19:35:10,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1632666.6666666667, ans=0.125
2023-11-21 19:35:32,193 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4450, loss[loss=0.05563, simple_loss=0.07561, pruned_loss=0.009771, audio_tagging_loss=0.008051, over 15050.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09705, pruned_loss=0.01658, audio_tagging_loss=0.009381, over 3054727.44 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:35:35,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1632800.0, ans=0.0
2023-11-21 19:35:45,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1632866.6666666667, ans=0.2
2023-11-21 19:35:46,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1632866.6666666667, ans=0.125
2023-11-21 19:35:54,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1632866.6666666667, ans=0.125
2023-11-21 19:35:54,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1632866.6666666667, ans=0.2
2023-11-21 19:36:02,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1632933.3333333333, ans=0.0
2023-11-21 19:36:07,874 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 244950
2023-11-21 19:36:29,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1633066.6666666667, ans=0.0
2023-11-21 19:36:35,219 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4500, loss[loss=0.07901, simple_loss=0.1058, pruned_loss=0.01846, audio_tagging_loss=0.007633, over 16119.00 frames. ], tot_loss[loss=0.07433, simple_loss=0.09706, pruned_loss=0.01651, audio_tagging_loss=0.009291, over 3052530.21 frames. ], batch size: 59, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:36:36,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1633133.3333333333, ans=0.2
2023-11-21 19:36:42,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.72 vs. limit=15.0
2023-11-21 19:36:53,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1633200.0, ans=0.125
2023-11-21 19:36:54,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1633200.0, ans=0.5
2023-11-21 19:37:01,318 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.155e+01 8.993e+01 9.811e+01 1.324e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-21 19:37:06,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1633266.6666666667, ans=0.0
2023-11-21 19:37:11,796 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245000
2023-11-21 19:37:39,071 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4550, loss[loss=0.06606, simple_loss=0.08447, pruned_loss=0.01001, audio_tagging_loss=0.01381, over 15264.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09582, pruned_loss=0.01618, audio_tagging_loss=0.009394, over 3045260.25 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:37:54,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1633533.3333333333, ans=0.2
2023-11-21 19:38:11,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1633600.0, ans=0.0
2023-11-21 19:38:16,154 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245050
2023-11-21 19:38:28,294 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:38:28,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1633666.6666666667, ans=0.2
2023-11-21 19:38:29,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1633733.3333333333, ans=0.125
2023-11-21 19:38:44,480 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4600, loss[loss=0.0724, simple_loss=0.1032, pruned_loss=0.01312, audio_tagging_loss=0.007669, over 15958.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09634, pruned_loss=0.01641, audio_tagging_loss=0.009499, over 3047772.41 frames. ], batch size: 55, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:38:45,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1633800.0, ans=0.0
2023-11-21 19:38:47,036 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.42 vs. limit=15.0
2023-11-21 19:38:57,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1633866.6666666667, ans=0.125
2023-11-21 19:39:03,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.61 vs. limit=22.5
2023-11-21 19:39:09,272 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.725e+01 8.114e+01 8.737e+01 9.348e+01 1.187e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 19:39:19,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245100
2023-11-21 19:39:21,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.32 vs. limit=15.0
2023-11-21 19:39:41,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1634066.6666666667, ans=0.1
2023-11-21 19:39:41,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1634066.6666666667, ans=0.125
2023-11-21 19:39:48,071 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4650, loss[loss=0.08805, simple_loss=0.1255, pruned_loss=0.01896, audio_tagging_loss=0.006358, over 16546.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09547, pruned_loss=0.01628, audio_tagging_loss=0.009573, over 3043817.76 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:39:50,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1634133.3333333333, ans=0.0
2023-11-21 19:39:56,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1634133.3333333333, ans=0.0
2023-11-21 19:40:02,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1634200.0, ans=0.0
2023-11-21 19:40:03,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1634200.0, ans=0.2
2023-11-21 19:40:05,826 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.88 vs. limit=22.5
2023-11-21 19:40:13,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1634266.6666666667, ans=0.125
2023-11-21 19:40:20,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1634266.6666666667, ans=0.0
2023-11-21 19:40:24,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245150
2023-11-21 19:40:40,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1634400.0, ans=0.5
2023-11-21 19:40:50,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1634466.6666666667, ans=0.0
2023-11-21 19:40:50,950 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4700, loss[loss=0.0602, simple_loss=0.0816, pruned_loss=0.009791, audio_tagging_loss=0.009614, over 16356.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09502, pruned_loss=0.01613, audio_tagging_loss=0.009584, over 3048499.96 frames. ], batch size: 62, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:40:52,419 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:41:17,146 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.203e+01 8.093e+01 8.785e+01 9.721e+01 1.198e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 19:41:27,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245200
2023-11-21 19:41:33,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1634666.6666666667, ans=0.0
2023-11-21 19:41:39,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1634666.6666666667, ans=0.125
2023-11-21 19:41:45,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1634733.3333333333, ans=0.125
2023-11-21 19:41:51,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1634733.3333333333, ans=0.1
2023-11-21 19:41:55,451 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4750, loss[loss=0.05251, simple_loss=0.06625, pruned_loss=0.009289, audio_tagging_loss=0.0101, over 14479.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09489, pruned_loss=0.01599, audio_tagging_loss=0.009625, over 3054762.70 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:42:03,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1634800.0, ans=0.5
2023-11-21 19:42:04,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1634800.0, ans=0.1
2023-11-21 19:42:07,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.34 vs. limit=12.0
2023-11-21 19:42:13,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1634866.6666666667, ans=22.5
2023-11-21 19:42:23,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1634933.3333333333, ans=0.0
2023-11-21 19:42:29,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.80 vs. limit=15.0
2023-11-21 19:42:30,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245250
2023-11-21 19:42:30,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1634933.3333333333, ans=0.125
2023-11-21 19:42:48,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1635066.6666666667, ans=0.035
2023-11-21 19:42:49,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1635066.6666666667, ans=0.1
2023-11-21 19:42:59,113 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4800, loss[loss=0.05127, simple_loss=0.06004, pruned_loss=0.009943, audio_tagging_loss=0.01131, over 14774.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09494, pruned_loss=0.01608, audio_tagging_loss=0.009752, over 3049180.10 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:43:00,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1635133.3333333333, ans=0.0
2023-11-21 19:43:12,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1635200.0, ans=0.0
2023-11-21 19:43:15,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.69 vs. limit=6.0
2023-11-21 19:43:25,997 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.114e+01 8.825e+01 9.753e+01 1.179e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 19:43:32,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1635266.6666666667, ans=0.125
2023-11-21 19:43:34,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245300
2023-11-21 19:43:59,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1635400.0, ans=0.125
2023-11-21 19:44:02,542 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4850, loss[loss=0.06306, simple_loss=0.06931, pruned_loss=0.01737, audio_tagging_loss=0.01103, over 16033.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09563, pruned_loss=0.01624, audio_tagging_loss=0.009821, over 3051598.77 frames. ], batch size: 63, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:44:04,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1635466.6666666667, ans=0.0
2023-11-21 19:44:11,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1635466.6666666667, ans=0.2
2023-11-21 19:44:29,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1635600.0, ans=0.125
2023-11-21 19:44:39,443 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245350
2023-11-21 19:44:39,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1635600.0, ans=0.1
2023-11-21 19:44:47,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1635666.6666666667, ans=0.2
2023-11-21 19:44:48,568 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.77 vs. limit=15.0
2023-11-21 19:45:03,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1635733.3333333333, ans=0.1
2023-11-21 19:45:07,177 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4900, loss[loss=0.07523, simple_loss=0.1046, pruned_loss=0.01718, audio_tagging_loss=0.005753, over 14420.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09577, pruned_loss=0.0161, audio_tagging_loss=0.009602, over 3050537.46 frames. ], batch size: 55, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:45:16,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1635800.0, ans=0.125
2023-11-21 19:45:27,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1635866.6666666667, ans=0.125
2023-11-21 19:45:34,466 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.513e+01 7.987e+01 8.445e+01 9.023e+01 1.315e+02, threshold=1.689e+02, percent-clipped=0.0
2023-11-21 19:45:43,201 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245400
2023-11-21 19:46:01,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1636066.6666666667, ans=0.125
2023-11-21 19:46:05,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=22.5
2023-11-21 19:46:12,599 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 4950, loss[loss=0.0895, simple_loss=0.1178, pruned_loss=0.02273, audio_tagging_loss=0.007879, over 16027.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09628, pruned_loss=0.0163, audio_tagging_loss=0.009455, over 3052286.38 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:46:12,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1636133.3333333333, ans=0.5
2023-11-21 19:46:16,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1636133.3333333333, ans=0.05
2023-11-21 19:46:21,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1636133.3333333333, ans=0.0
2023-11-21 19:46:32,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1636200.0, ans=0.0
2023-11-21 19:46:33,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1636200.0, ans=0.0
2023-11-21 19:46:38,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1636266.6666666667, ans=0.0
2023-11-21 19:46:48,307 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245450
2023-11-21 19:46:49,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1636333.3333333333, ans=0.1
2023-11-21 19:46:54,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1636333.3333333333, ans=15.0
2023-11-21 19:47:16,830 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5000, loss[loss=0.06125, simple_loss=0.07459, pruned_loss=0.01372, audio_tagging_loss=0.01023, over 16132.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09588, pruned_loss=0.0161, audio_tagging_loss=0.009354, over 3046352.88 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:47:26,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1636466.6666666667, ans=0.0
2023-11-21 19:47:44,392 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.789e+01 8.147e+01 8.751e+01 9.572e+01 1.278e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 19:47:53,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245500
2023-11-21 19:47:59,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1636666.6666666667, ans=0.125
2023-11-21 19:48:03,359 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.765e-02
2023-11-21 19:48:08,660 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.07 vs. limit=22.5
2023-11-21 19:48:21,042 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5050, loss[loss=0.08627, simple_loss=0.1196, pruned_loss=0.02122, audio_tagging_loss=0.005233, over 16197.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09621, pruned_loss=0.01623, audio_tagging_loss=0.009242, over 3043808.76 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:48:42,622 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.04 vs. limit=22.5
2023-11-21 19:48:56,555 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245550
2023-11-21 19:49:25,096 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5100, loss[loss=0.1083, simple_loss=0.1378, pruned_loss=0.02921, audio_tagging_loss=0.01013, over 15751.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09596, pruned_loss=0.01626, audio_tagging_loss=0.009228, over 3045962.08 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:49:30,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1637133.3333333333, ans=0.125
2023-11-21 19:49:47,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1637200.0, ans=0.0
2023-11-21 19:49:51,620 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.776e+01 7.985e+01 8.659e+01 9.540e+01 1.416e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 19:50:00,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245600
2023-11-21 19:50:00,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1637266.6666666667, ans=0.0
2023-11-21 19:50:03,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1637333.3333333333, ans=0.0
2023-11-21 19:50:09,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1637333.3333333333, ans=0.125
2023-11-21 19:50:15,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1637400.0, ans=0.2
2023-11-21 19:50:29,311 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5150, loss[loss=0.08832, simple_loss=0.1173, pruned_loss=0.01996, audio_tagging_loss=0.009711, over 14646.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09502, pruned_loss=0.01614, audio_tagging_loss=0.009324, over 3038425.98 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 19:50:36,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1637466.6666666667, ans=0.0
2023-11-21 19:50:46,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1637533.3333333333, ans=0.0
2023-11-21 19:50:53,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1637600.0, ans=0.0
2023-11-21 19:51:05,584 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245650
2023-11-21 19:51:12,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1637666.6666666667, ans=0.125
2023-11-21 19:51:12,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1637666.6666666667, ans=0.125
2023-11-21 19:51:16,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1637666.6666666667, ans=0.125
2023-11-21 19:51:33,902 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5200, loss[loss=0.05997, simple_loss=0.07849, pruned_loss=0.0126, audio_tagging_loss=0.008127, over 16028.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09544, pruned_loss=0.01607, audio_tagging_loss=0.009289, over 3040271.57 frames. ], batch size: 61, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:51:41,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1637800.0, ans=0.0
2023-11-21 19:51:53,461 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.56 vs. limit=15.0
2023-11-21 19:52:01,156 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 7.936e+01 8.716e+01 9.499e+01 2.064e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 19:52:02,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1637933.3333333333, ans=0.125
2023-11-21 19:52:09,267 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245700
2023-11-21 19:52:10,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1638000.0, ans=0.0
2023-11-21 19:52:28,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1638066.6666666667, ans=0.125
2023-11-21 19:52:38,024 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5250, loss[loss=0.07418, simple_loss=0.1007, pruned_loss=0.0148, audio_tagging_loss=0.009015, over 15931.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09582, pruned_loss=0.0162, audio_tagging_loss=0.009284, over 3036255.54 frames. ], batch size: 60, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:52:51,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1638200.0, ans=0.2
2023-11-21 19:52:54,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.79 vs. limit=10.0
2023-11-21 19:53:14,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245750
2023-11-21 19:53:16,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1638333.3333333333, ans=0.2
2023-11-21 19:53:18,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1638333.3333333333, ans=0.05
2023-11-21 19:53:18,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1638333.3333333333, ans=0.2
2023-11-21 19:53:41,696 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5300, loss[loss=0.08418, simple_loss=0.1149, pruned_loss=0.01987, audio_tagging_loss=0.006847, over 14546.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09568, pruned_loss=0.01613, audio_tagging_loss=0.009305, over 3031022.00 frames. ], batch size: 52, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:53:44,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1638466.6666666667, ans=0.125
2023-11-21 19:53:54,239 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.89 vs. limit=15.0
2023-11-21 19:53:56,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1638533.3333333333, ans=0.1
2023-11-21 19:54:07,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1638600.0, ans=0.1
2023-11-21 19:54:07,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1638600.0, ans=0.125
2023-11-21 19:54:10,759 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.198e+01 8.861e+01 9.238e+01 1.215e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-21 19:54:18,859 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245800
2023-11-21 19:54:30,311 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:54:42,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=1638733.3333333333, ans=0.025
2023-11-21 19:54:42,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1638733.3333333333, ans=0.0
2023-11-21 19:54:47,325 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5350, loss[loss=0.09243, simple_loss=0.1251, pruned_loss=0.02368, audio_tagging_loss=0.006216, over 14899.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.0965, pruned_loss=0.0164, audio_tagging_loss=0.009337, over 3038815.61 frames. ], batch size: 53, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:55:03,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1638866.6666666667, ans=0.1
2023-11-21 19:55:09,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1638866.6666666667, ans=0.1
2023-11-21 19:55:09,920 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.17 vs. limit=15.0
2023-11-21 19:55:10,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1638866.6666666667, ans=0.1
2023-11-21 19:55:23,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245850
2023-11-21 19:55:46,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1639066.6666666667, ans=0.0
2023-11-21 19:55:52,130 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5400, loss[loss=0.08065, simple_loss=0.1055, pruned_loss=0.01868, audio_tagging_loss=0.009194, over 15235.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09584, pruned_loss=0.01622, audio_tagging_loss=0.009415, over 3044724.93 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:55:59,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1639133.3333333333, ans=0.0
2023-11-21 19:56:15,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1639200.0, ans=0.95
2023-11-21 19:56:17,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1639266.6666666667, ans=0.125
2023-11-21 19:56:19,853 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.098e+01 8.835e+01 9.334e+01 1.205e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 19:56:28,473 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245900
2023-11-21 19:56:55,945 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5450, loss[loss=0.07697, simple_loss=0.1028, pruned_loss=0.016, audio_tagging_loss=0.009588, over 15400.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09652, pruned_loss=0.01651, audio_tagging_loss=0.009474, over 3047471.94 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:57:32,999 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 245950
2023-11-21 19:58:00,374 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5500, loss[loss=0.07146, simple_loss=0.08825, pruned_loss=0.01987, audio_tagging_loss=0.007464, over 16159.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09605, pruned_loss=0.0163, audio_tagging_loss=0.009505, over 3051486.10 frames. ], batch size: 63, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:58:09,382 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.48 vs. limit=15.0
2023-11-21 19:58:11,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1639800.0, ans=0.0
2023-11-21 19:58:23,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1639866.6666666667, ans=0.1
2023-11-21 19:58:28,913 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.083e+01 8.649e+01 9.254e+01 1.187e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 19:58:29,698 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.35 vs. limit=15.0
2023-11-21 19:58:36,351 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246000
2023-11-21 19:58:54,286 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.33 vs. limit=22.5
2023-11-21 19:59:04,546 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5550, loss[loss=0.09762, simple_loss=0.1375, pruned_loss=0.0218, audio_tagging_loss=0.00709, over 15600.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.0962, pruned_loss=0.01637, audio_tagging_loss=0.009592, over 3035225.91 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:59:08,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1640133.3333333333, ans=0.1
2023-11-21 19:59:22,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1640200.0, ans=0.125
2023-11-21 19:59:23,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1640200.0, ans=15.0
2023-11-21 19:59:25,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0
2023-11-21 19:59:26,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1640200.0, ans=0.0
2023-11-21 19:59:40,109 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246050
2023-11-21 19:59:50,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0
2023-11-21 20:00:08,738 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5600, loss[loss=0.06732, simple_loss=0.09527, pruned_loss=0.01289, audio_tagging_loss=0.006801, over 14817.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09567, pruned_loss=0.01605, audio_tagging_loss=0.00976, over 3039843.13 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:00:35,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1640600.0, ans=0.1
2023-11-21 20:00:37,209 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.033e+01 8.931e+01 9.623e+01 1.146e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 20:00:44,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246100
2023-11-21 20:00:52,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1640666.6666666667, ans=0.125
2023-11-21 20:00:53,813 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:00:58,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1640733.3333333333, ans=0.1
2023-11-21 20:01:11,931 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5650, loss[loss=0.07994, simple_loss=0.1008, pruned_loss=0.02211, audio_tagging_loss=0.007442, over 14787.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09645, pruned_loss=0.01631, audio_tagging_loss=0.009706, over 3044413.70 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:01:14,323 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.97 vs. limit=15.0
2023-11-21 20:01:19,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1640800.0, ans=0.2
2023-11-21 20:01:34,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1640866.6666666667, ans=0.125
2023-11-21 20:01:37,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1640933.3333333333, ans=0.04949747468305833
2023-11-21 20:01:48,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246150
2023-11-21 20:02:15,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1641133.3333333333, ans=0.125
2023-11-21 20:02:16,869 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5700, loss[loss=0.08779, simple_loss=0.1179, pruned_loss=0.01976, audio_tagging_loss=0.009083, over 15595.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09583, pruned_loss=0.01609, audio_tagging_loss=0.009727, over 3048035.02 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:02:44,514 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.226e+01 8.873e+01 9.610e+01 1.173e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 20:02:52,779 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246200
2023-11-21 20:02:54,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1641333.3333333333, ans=0.125
2023-11-21 20:03:21,885 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5750, loss[loss=0.08613, simple_loss=0.1137, pruned_loss=0.02208, audio_tagging_loss=0.007206, over 15783.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09563, pruned_loss=0.01615, audio_tagging_loss=0.009634, over 3039185.52 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:03:42,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1641533.3333333333, ans=0.125
2023-11-21 20:03:58,235 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246250
2023-11-21 20:04:20,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.48 vs. limit=15.0
2023-11-21 20:04:25,647 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5800, loss[loss=0.08766, simple_loss=0.1192, pruned_loss=0.02193, audio_tagging_loss=0.006156, over 15532.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09544, pruned_loss=0.01601, audio_tagging_loss=0.009518, over 3049672.39 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:04:49,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1641866.6666666667, ans=0.125
2023-11-21 20:04:54,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1641933.3333333333, ans=0.125
2023-11-21 20:04:55,493 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.128e+01 8.477e+01 9.291e+01 1.218e+02, threshold=1.695e+02, percent-clipped=0.0
2023-11-21 20:04:55,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1641933.3333333333, ans=0.125
2023-11-21 20:05:01,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246300
2023-11-21 20:05:08,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1642000.0, ans=0.125
2023-11-21 20:05:11,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1642000.0, ans=0.125
2023-11-21 20:05:29,860 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5850, loss[loss=0.09274, simple_loss=0.1299, pruned_loss=0.02098, audio_tagging_loss=0.0068, over 15571.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09505, pruned_loss=0.01584, audio_tagging_loss=0.009494, over 3045795.74 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:05:31,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1642133.3333333333, ans=0.2
2023-11-21 20:05:47,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1642200.0, ans=0.0
2023-11-21 20:05:55,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1642266.6666666667, ans=0.125
2023-11-21 20:06:05,298 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246350
2023-11-21 20:06:29,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1642400.0, ans=0.1
2023-11-21 20:06:34,579 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5900, loss[loss=0.08658, simple_loss=0.1179, pruned_loss=0.01986, audio_tagging_loss=0.007785, over 16224.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09561, pruned_loss=0.01585, audio_tagging_loss=0.009428, over 3048128.90 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:06:47,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1642533.3333333333, ans=0.125
2023-11-21 20:06:53,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1642533.3333333333, ans=0.0
2023-11-21 20:07:04,006 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.116e+01 8.807e+01 9.373e+01 1.126e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 20:07:10,158 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246400
2023-11-21 20:07:13,584 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0
2023-11-21 20:07:15,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1642666.6666666667, ans=0.1
2023-11-21 20:07:17,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1642666.6666666667, ans=0.125
2023-11-21 20:07:24,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1642733.3333333333, ans=0.125
2023-11-21 20:07:37,638 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 5950, loss[loss=0.08121, simple_loss=0.1163, pruned_loss=0.01584, audio_tagging_loss=0.007248, over 14677.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09506, pruned_loss=0.01574, audio_tagging_loss=0.009455, over 3052140.25 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:07:45,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1642800.0, ans=0.125
2023-11-21 20:07:48,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1642800.0, ans=0.125
2023-11-21 20:08:14,379 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246450
2023-11-21 20:08:19,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1643000.0, ans=0.2
2023-11-21 20:08:30,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1643066.6666666667, ans=0.125
2023-11-21 20:08:31,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1643066.6666666667, ans=0.0
2023-11-21 20:08:42,288 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6000, loss[loss=0.08056, simple_loss=0.1012, pruned_loss=0.01824, audio_tagging_loss=0.01173, over 15360.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09555, pruned_loss=0.01594, audio_tagging_loss=0.009461, over 3049385.24 frames. ], batch size: 60, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:08:42,289 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 20:09:23,101 INFO [train_asr.py:1253] (2/4) Epoch 21, validation: loss=0.05951, simple_loss=0.05205, pruned_loss=0.005242, audio_tagging_loss=0.02825, over 4681554.00 frames. 
2023-11-21 20:09:23,101 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 20:09:40,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.65 vs. limit=15.0
2023-11-21 20:09:46,525 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:09:48,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1643266.6666666667, ans=0.2
2023-11-21 20:09:52,395 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 7.949e+01 8.551e+01 9.154e+01 1.132e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-21 20:09:58,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246500
2023-11-21 20:10:01,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1643333.3333333333, ans=0.125
2023-11-21 20:10:09,440 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:10:13,582 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:10:26,630 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6050, loss[loss=0.06855, simple_loss=0.08742, pruned_loss=0.01459, audio_tagging_loss=0.01026, over 16138.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09514, pruned_loss=0.01586, audio_tagging_loss=0.009495, over 3053187.57 frames. ], batch size: 60, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:10:38,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1643533.3333333333, ans=0.125
2023-11-21 20:10:39,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1643533.3333333333, ans=0.1
2023-11-21 20:10:53,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1643600.0, ans=0.125
2023-11-21 20:10:55,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.34 vs. limit=15.0
2023-11-21 20:10:57,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1643600.0, ans=0.1
2023-11-21 20:11:02,841 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246550
2023-11-21 20:11:04,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1643666.6666666667, ans=0.0
2023-11-21 20:11:20,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1643733.3333333333, ans=0.125
2023-11-21 20:11:30,227 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.63 vs. limit=22.5
2023-11-21 20:11:30,728 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6100, loss[loss=0.07522, simple_loss=0.1002, pruned_loss=0.01612, audio_tagging_loss=0.009007, over 15214.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09465, pruned_loss=0.01584, audio_tagging_loss=0.00949, over 3049756.26 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:11:32,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1643800.0, ans=0.125
2023-11-21 20:11:59,492 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.16 vs. limit=22.5
2023-11-21 20:12:03,546 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 7.866e+01 8.426e+01 9.427e+01 1.141e+02, threshold=1.685e+02, percent-clipped=0.0
2023-11-21 20:12:05,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.75 vs. limit=10.0
2023-11-21 20:12:06,075 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246600
2023-11-21 20:12:22,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.92 vs. limit=15.0
2023-11-21 20:12:26,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1644066.6666666667, ans=0.125
2023-11-21 20:12:33,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1644133.3333333333, ans=0.2
2023-11-21 20:12:34,400 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6150, loss[loss=0.08573, simple_loss=0.1229, pruned_loss=0.0173, audio_tagging_loss=0.006967, over 14775.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09517, pruned_loss=0.01618, audio_tagging_loss=0.009565, over 3045430.57 frames. ], batch size: 52, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:12:45,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1644133.3333333333, ans=0.125
2023-11-21 20:13:04,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1644266.6666666667, ans=0.125
2023-11-21 20:13:10,137 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246650
2023-11-21 20:13:10,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1644266.6666666667, ans=0.1
2023-11-21 20:13:12,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1644333.3333333333, ans=0.1
2023-11-21 20:13:13,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1644333.3333333333, ans=0.125
2023-11-21 20:13:22,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.18 vs. limit=15.0
2023-11-21 20:13:24,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1644400.0, ans=0.2
2023-11-21 20:13:37,953 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6200, loss[loss=0.06071, simple_loss=0.07784, pruned_loss=0.01145, audio_tagging_loss=0.01034, over 15101.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09354, pruned_loss=0.01594, audio_tagging_loss=0.009645, over 3045488.62 frames. ], batch size: 60, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:14:04,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1644600.0, ans=0.125
2023-11-21 20:14:11,063 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.261e+01 8.724e+01 9.317e+01 1.600e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 20:14:13,599 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246700
2023-11-21 20:14:18,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.86 vs. limit=22.5
2023-11-21 20:14:19,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1644666.6666666667, ans=0.1
2023-11-21 20:14:39,085 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.55 vs. limit=15.0
2023-11-21 20:14:42,101 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6250, loss[loss=0.06469, simple_loss=0.07541, pruned_loss=0.01132, audio_tagging_loss=0.01567, over 14634.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09403, pruned_loss=0.01588, audio_tagging_loss=0.009731, over 3047233.75 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:15:17,625 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246750
2023-11-21 20:15:23,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1645000.0, ans=0.1
2023-11-21 20:15:23,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1645000.0, ans=0.125
2023-11-21 20:15:32,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.86 vs. limit=8.0
2023-11-21 20:15:45,981 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6300, loss[loss=0.07849, simple_loss=0.09589, pruned_loss=0.01779, audio_tagging_loss=0.01276, over 14618.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09467, pruned_loss=0.01592, audio_tagging_loss=0.009758, over 3048330.08 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:16:08,006 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0
2023-11-21 20:16:17,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1645266.6666666667, ans=0.0
2023-11-21 20:16:19,039 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.125e+01 8.837e+01 9.702e+01 1.272e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 20:16:21,598 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246800
2023-11-21 20:16:45,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1645400.0, ans=0.2
2023-11-21 20:16:50,197 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6350, loss[loss=0.07022, simple_loss=0.08717, pruned_loss=0.01879, audio_tagging_loss=0.007846, over 14603.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09431, pruned_loss=0.01591, audio_tagging_loss=0.009761, over 3046768.79 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:17:24,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1645600.0, ans=0.125
2023-11-21 20:17:26,557 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246850
2023-11-21 20:17:26,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1645600.0, ans=0.125
2023-11-21 20:17:40,202 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:17:55,213 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6400, loss[loss=0.0719, simple_loss=0.09189, pruned_loss=0.01333, audio_tagging_loss=0.01262, over 14767.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09465, pruned_loss=0.01606, audio_tagging_loss=0.009777, over 3044654.87 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:18:15,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.61 vs. limit=15.0
2023-11-21 20:18:26,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1645933.3333333333, ans=0.0
2023-11-21 20:18:27,617 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.810e+01 7.956e+01 8.478e+01 9.124e+01 1.124e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-21 20:18:30,213 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246900
2023-11-21 20:18:51,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1646066.6666666667, ans=0.1
2023-11-21 20:18:56,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1646066.6666666667, ans=0.125
2023-11-21 20:18:58,460 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6450, loss[loss=0.06346, simple_loss=0.07645, pruned_loss=0.01458, audio_tagging_loss=0.01066, over 15204.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09362, pruned_loss=0.01598, audio_tagging_loss=0.009887, over 3041412.26 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:19:10,361 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:19:25,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1646266.6666666667, ans=0.1
2023-11-21 20:19:29,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1646266.6666666667, ans=0.125
2023-11-21 20:19:31,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1646266.6666666667, ans=0.0
2023-11-21 20:19:34,781 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 246950
2023-11-21 20:19:38,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1646333.3333333333, ans=0.125
2023-11-21 20:20:02,060 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6500, loss[loss=0.06996, simple_loss=0.09394, pruned_loss=0.01469, audio_tagging_loss=0.008295, over 14040.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.0938, pruned_loss=0.01606, audio_tagging_loss=0.009805, over 3035299.51 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:20:07,811 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.80 vs. limit=15.0
2023-11-21 20:20:09,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1646466.6666666667, ans=0.0
2023-11-21 20:20:13,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1646533.3333333333, ans=0.2
2023-11-21 20:20:35,572 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.211e+01 8.118e+01 8.877e+01 9.588e+01 1.151e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 20:20:38,769 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247000
2023-11-21 20:20:44,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.14 vs. limit=15.0
2023-11-21 20:20:47,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1646666.6666666667, ans=0.125
2023-11-21 20:21:07,068 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6550, loss[loss=0.07238, simple_loss=0.09325, pruned_loss=0.0143, audio_tagging_loss=0.01145, over 15308.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09314, pruned_loss=0.01575, audio_tagging_loss=0.009685, over 3039701.70 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:21:08,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1646800.0, ans=0.125
2023-11-21 20:21:34,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.59 vs. limit=15.0
2023-11-21 20:21:43,201 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247050
2023-11-21 20:21:51,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.58 vs. limit=12.0
2023-11-21 20:22:11,489 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6600, loss[loss=0.0698, simple_loss=0.0935, pruned_loss=0.01349, audio_tagging_loss=0.009567, over 16263.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09417, pruned_loss=0.01587, audio_tagging_loss=0.009503, over 3046056.27 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:22:17,837 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.39 vs. limit=15.0
2023-11-21 20:22:30,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1647200.0, ans=0.0
2023-11-21 20:22:31,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1647200.0, ans=0.125
2023-11-21 20:22:40,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1647266.6666666667, ans=0.125
2023-11-21 20:22:44,404 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.213e+01 8.717e+01 9.748e+01 1.869e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 20:22:47,599 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247100
2023-11-21 20:22:54,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1647333.3333333333, ans=0.125
2023-11-21 20:22:54,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.whiten.whitening_limit, batch_count=1647333.3333333333, ans=12.0
2023-11-21 20:22:55,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1647333.3333333333, ans=0.125
2023-11-21 20:23:15,530 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6650, loss[loss=0.09047, simple_loss=0.1153, pruned_loss=0.02459, audio_tagging_loss=0.008202, over 16573.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09422, pruned_loss=0.01595, audio_tagging_loss=0.009479, over 3047415.42 frames. ], batch size: 60, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:23:17,644 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.05 vs. limit=22.5
2023-11-21 20:23:18,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1647466.6666666667, ans=0.125
2023-11-21 20:23:34,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1647533.3333333333, ans=0.0
2023-11-21 20:23:36,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1647533.3333333333, ans=0.0
2023-11-21 20:23:47,608 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.03 vs. limit=22.5
2023-11-21 20:23:50,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247150
2023-11-21 20:23:52,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1647666.6666666667, ans=0.125
2023-11-21 20:23:56,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1647666.6666666667, ans=0.125
2023-11-21 20:24:12,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1647733.3333333333, ans=0.2
2023-11-21 20:24:18,045 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6700, loss[loss=0.07217, simple_loss=0.1033, pruned_loss=0.01164, audio_tagging_loss=0.008861, over 14815.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09494, pruned_loss=0.01609, audio_tagging_loss=0.009481, over 3046867.74 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:24:32,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1647866.6666666667, ans=0.125
2023-11-21 20:24:34,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1647866.6666666667, ans=0.1
2023-11-21 20:24:51,829 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.102e+01 8.678e+01 9.415e+01 1.201e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 20:24:54,337 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247200
2023-11-21 20:25:01,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1648000.0, ans=0.2
2023-11-21 20:25:06,314 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.52 vs. limit=15.0
2023-11-21 20:25:10,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1648066.6666666667, ans=0.0
2023-11-21 20:25:22,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.87 vs. limit=10.0
2023-11-21 20:25:23,162 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6750, loss[loss=0.08227, simple_loss=0.1106, pruned_loss=0.01987, audio_tagging_loss=0.007104, over 15057.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09384, pruned_loss=0.01599, audio_tagging_loss=0.009554, over 3040312.15 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:25:24,784 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:25:36,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1648200.0, ans=0.125
2023-11-21 20:25:58,242 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247250
2023-11-21 20:25:59,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1648333.3333333333, ans=0.0
2023-11-21 20:26:09,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1648333.3333333333, ans=0.025
2023-11-21 20:26:14,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1648400.0, ans=0.125
2023-11-21 20:26:17,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.10 vs. limit=22.5
2023-11-21 20:26:21,563 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:26:26,347 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6800, loss[loss=0.09335, simple_loss=0.1243, pruned_loss=0.0245, audio_tagging_loss=0.0067, over 15668.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09414, pruned_loss=0.01607, audio_tagging_loss=0.009591, over 3041249.76 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:26:37,876 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.99 vs. limit=10.0
2023-11-21 20:26:45,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.18 vs. limit=15.0
2023-11-21 20:26:47,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1648533.3333333333, ans=0.125
2023-11-21 20:26:59,643 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 7.919e+01 8.607e+01 9.295e+01 1.340e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 20:27:02,158 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247300
2023-11-21 20:27:29,481 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6850, loss[loss=0.08147, simple_loss=0.1075, pruned_loss=0.01871, audio_tagging_loss=0.009033, over 16451.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09408, pruned_loss=0.01591, audio_tagging_loss=0.00948, over 3042431.96 frames. ], batch size: 60, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:28:03,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1648933.3333333333, ans=0.0
2023-11-21 20:28:06,040 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247350
2023-11-21 20:28:16,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1649000.0, ans=0.0
2023-11-21 20:28:16,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1649000.0, ans=10.0
2023-11-21 20:28:25,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1649066.6666666667, ans=15.0
2023-11-21 20:28:33,860 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6900, loss[loss=0.05689, simple_loss=0.07446, pruned_loss=0.01181, audio_tagging_loss=0.007852, over 15674.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09495, pruned_loss=0.01593, audio_tagging_loss=0.009343, over 3043342.52 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:28:52,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1649200.0, ans=0.125
2023-11-21 20:28:52,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1649200.0, ans=0.0
2023-11-21 20:28:59,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1649266.6666666667, ans=0.0
2023-11-21 20:28:59,828 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.16 vs. limit=15.0
2023-11-21 20:29:08,792 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.897e+01 8.008e+01 8.614e+01 9.356e+01 1.357e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 20:29:08,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247400
2023-11-21 20:29:23,604 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:29:37,997 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 6950, loss[loss=0.08613, simple_loss=0.1134, pruned_loss=0.02037, audio_tagging_loss=0.009078, over 14922.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09589, pruned_loss=0.01622, audio_tagging_loss=0.009361, over 3044987.15 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:30:10,365 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.53 vs. limit=10.0
2023-11-21 20:30:13,493 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247450
2023-11-21 20:30:41,582 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7000, loss[loss=0.07167, simple_loss=0.09934, pruned_loss=0.01283, audio_tagging_loss=0.009166, over 14550.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09557, pruned_loss=0.01606, audio_tagging_loss=0.009454, over 3043637.31 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:30:41,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1649800.0, ans=10.0
2023-11-21 20:30:48,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1649800.0, ans=0.125
2023-11-21 20:30:52,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1649800.0, ans=0.2
2023-11-21 20:31:05,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1649866.6666666667, ans=0.1
2023-11-21 20:31:18,723 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 7.928e+01 8.493e+01 9.155e+01 1.182e+02, threshold=1.699e+02, percent-clipped=0.0
2023-11-21 20:31:18,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247500
2023-11-21 20:31:32,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1650066.6666666667, ans=0.2
2023-11-21 20:31:36,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1650066.6666666667, ans=0.125
2023-11-21 20:31:47,131 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7050, loss[loss=0.06955, simple_loss=0.09049, pruned_loss=0.01606, audio_tagging_loss=0.008247, over 15923.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.0949, pruned_loss=0.01596, audio_tagging_loss=0.009572, over 3045802.42 frames. ], batch size: 60, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:31:51,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1650133.3333333333, ans=0.025
2023-11-21 20:31:53,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1650133.3333333333, ans=0.0
2023-11-21 20:32:00,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1650200.0, ans=0.0
2023-11-21 20:32:08,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.71 vs. limit=6.0
2023-11-21 20:32:09,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1650200.0, ans=0.125
2023-11-21 20:32:09,451 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:32:09,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1650200.0, ans=0.07
2023-11-21 20:32:11,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1650266.6666666667, ans=0.125
2023-11-21 20:32:22,525 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247550
2023-11-21 20:32:49,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1650400.0, ans=0.125
2023-11-21 20:32:51,827 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7100, loss[loss=0.08627, simple_loss=0.1069, pruned_loss=0.02252, audio_tagging_loss=0.01028, over 14914.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09508, pruned_loss=0.01624, audio_tagging_loss=0.009576, over 3051115.45 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:32:54,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1650466.6666666667, ans=0.2
2023-11-21 20:33:09,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.27 vs. limit=15.0
2023-11-21 20:33:27,107 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.093e+01 8.713e+01 9.346e+01 1.337e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 20:33:27,289 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247600
2023-11-21 20:33:28,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1650666.6666666667, ans=0.0
2023-11-21 20:33:30,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1650666.6666666667, ans=0.0
2023-11-21 20:33:41,200 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:33:46,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1650733.3333333333, ans=0.1
2023-11-21 20:33:55,492 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7150, loss[loss=0.0972, simple_loss=0.1351, pruned_loss=0.02224, audio_tagging_loss=0.00739, over 16028.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09459, pruned_loss=0.01614, audio_tagging_loss=0.00967, over 3049112.97 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:33:56,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0
2023-11-21 20:33:59,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1650800.0, ans=0.125
2023-11-21 20:34:11,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1650866.6666666667, ans=0.0
2023-11-21 20:34:26,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1650933.3333333333, ans=0.0
2023-11-21 20:34:32,130 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247650
2023-11-21 20:34:42,336 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.23 vs. limit=15.0
2023-11-21 20:34:44,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1651000.0, ans=0.1
2023-11-21 20:34:45,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1651066.6666666667, ans=0.125
2023-11-21 20:34:50,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0
2023-11-21 20:34:54,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1651066.6666666667, ans=0.1
2023-11-21 20:34:56,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1651066.6666666667, ans=0.0
2023-11-21 20:35:00,013 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7200, loss[loss=0.08625, simple_loss=0.11, pruned_loss=0.02034, audio_tagging_loss=0.01092, over 15573.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09479, pruned_loss=0.01605, audio_tagging_loss=0.009709, over 3051197.82 frames. ], batch size: 60, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:35:35,368 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.484e+01 9.159e+01 1.010e+02 1.295e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-21 20:35:35,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247700
2023-11-21 20:35:41,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1651333.3333333333, ans=0.0
2023-11-21 20:35:51,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1651400.0, ans=0.125
2023-11-21 20:35:52,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1651400.0, ans=0.125
2023-11-21 20:35:55,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1651400.0, ans=0.0
2023-11-21 20:36:03,888 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7250, loss[loss=0.0602, simple_loss=0.07109, pruned_loss=0.01338, audio_tagging_loss=0.01128, over 15115.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09463, pruned_loss=0.01597, audio_tagging_loss=0.00982, over 3046135.14 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:36:06,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1651466.6666666667, ans=0.0
2023-11-21 20:36:06,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1651466.6666666667, ans=0.0
2023-11-21 20:36:27,867 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:36:32,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1651600.0, ans=0.125
2023-11-21 20:36:39,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247750
2023-11-21 20:36:56,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1651733.3333333333, ans=0.0
2023-11-21 20:37:00,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1651733.3333333333, ans=0.125
2023-11-21 20:37:07,611 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7300, loss[loss=0.06124, simple_loss=0.07411, pruned_loss=0.01281, audio_tagging_loss=0.01138, over 15616.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09526, pruned_loss=0.01609, audio_tagging_loss=0.009695, over 3046314.65 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:37:16,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1651800.0, ans=0.125
2023-11-21 20:37:26,823 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.29 vs. limit=15.0
2023-11-21 20:37:30,525 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.23 vs. limit=6.0
2023-11-21 20:37:43,498 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.100e+01 8.051e+01 8.700e+01 9.385e+01 1.347e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 20:37:43,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247800
2023-11-21 20:37:53,930 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.47 vs. limit=15.0
2023-11-21 20:38:05,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1652066.6666666667, ans=0.0
2023-11-21 20:38:07,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.14 vs. limit=15.0
2023-11-21 20:38:12,558 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7350, loss[loss=0.07434, simple_loss=0.1017, pruned_loss=0.01663, audio_tagging_loss=0.006878, over 16494.00 frames. ], tot_loss[loss=0.07311, simple_loss=0.09509, pruned_loss=0.01603, audio_tagging_loss=0.009533, over 3047428.53 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:38:16,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1652133.3333333333, ans=0.125
2023-11-21 20:38:25,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1652200.0, ans=0.2
2023-11-21 20:38:29,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=15.0
2023-11-21 20:38:34,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1652200.0, ans=0.125
2023-11-21 20:38:43,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.07 vs. limit=10.0
2023-11-21 20:38:47,959 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247850
2023-11-21 20:38:54,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1652333.3333333333, ans=0.1
2023-11-21 20:38:56,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1652333.3333333333, ans=0.125
2023-11-21 20:38:56,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1652333.3333333333, ans=0.035
2023-11-21 20:39:07,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1652400.0, ans=0.0
2023-11-21 20:39:09,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1652400.0, ans=0.125
2023-11-21 20:39:15,984 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7400, loss[loss=0.1125, simple_loss=0.1468, pruned_loss=0.03104, audio_tagging_loss=0.008041, over 15219.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09626, pruned_loss=0.01629, audio_tagging_loss=0.009475, over 3051733.53 frames. ], batch size: 54, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:39:22,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1652466.6666666667, ans=0.2
2023-11-21 20:39:51,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.103e+01 8.809e+01 9.626e+01 1.321e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 20:39:51,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247900
2023-11-21 20:39:53,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1652666.6666666667, ans=0.0
2023-11-21 20:39:55,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1652666.6666666667, ans=0.0
2023-11-21 20:40:04,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.39 vs. limit=22.5
2023-11-21 20:40:18,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1652800.0, ans=0.0
2023-11-21 20:40:18,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1652800.0, ans=0.5
2023-11-21 20:40:19,652 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7450, loss[loss=0.07471, simple_loss=0.1054, pruned_loss=0.01348, audio_tagging_loss=0.00853, over 14795.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09741, pruned_loss=0.01649, audio_tagging_loss=0.009346, over 3057362.53 frames. ], batch size: 52, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:40:28,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1652800.0, ans=0.0
2023-11-21 20:40:29,900 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.07 vs. limit=15.0
2023-11-21 20:40:41,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1652866.6666666667, ans=0.0
2023-11-21 20:40:51,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1652933.3333333333, ans=0.0
2023-11-21 20:40:55,219 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 247950
2023-11-21 20:40:59,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1653000.0, ans=0.125
2023-11-21 20:41:21,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1653133.3333333333, ans=0.2
2023-11-21 20:41:22,594 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7500, loss[loss=0.06748, simple_loss=0.08181, pruned_loss=0.01286, audio_tagging_loss=0.01372, over 14712.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09718, pruned_loss=0.01634, audio_tagging_loss=0.009375, over 3061052.85 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:41:28,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1653133.3333333333, ans=0.125
2023-11-21 20:41:37,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1653200.0, ans=0.1
2023-11-21 20:41:43,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1653200.0, ans=0.125
2023-11-21 20:41:57,959 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.210e+01 8.790e+01 9.485e+01 1.319e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 20:41:58,171 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248000
2023-11-21 20:42:00,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1653333.3333333333, ans=0.125
2023-11-21 20:42:22,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1653400.0, ans=0.125
2023-11-21 20:42:28,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1653466.6666666667, ans=0.125
2023-11-21 20:42:29,568 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7550, loss[loss=0.06256, simple_loss=0.07997, pruned_loss=0.01073, audio_tagging_loss=0.01185, over 16356.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09682, pruned_loss=0.01634, audio_tagging_loss=0.009369, over 3062060.20 frames. ], batch size: 64, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:43:05,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248050
2023-11-21 20:43:06,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1653666.6666666667, ans=0.2
2023-11-21 20:43:10,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1653666.6666666667, ans=0.125
2023-11-21 20:43:15,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1653666.6666666667, ans=0.07
2023-11-21 20:43:18,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.80 vs. limit=15.0
2023-11-21 20:43:19,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1653733.3333333333, ans=0.2
2023-11-21 20:43:25,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1653733.3333333333, ans=0.125
2023-11-21 20:43:32,804 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7600, loss[loss=0.07007, simple_loss=0.09832, pruned_loss=0.01201, audio_tagging_loss=0.008898, over 15515.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.0964, pruned_loss=0.01631, audio_tagging_loss=0.009362, over 3070418.44 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:43:43,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1653800.0, ans=0.0
2023-11-21 20:43:54,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1653866.6666666667, ans=0.0
2023-11-21 20:44:08,224 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.151e+01 8.758e+01 9.560e+01 1.334e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 20:44:08,371 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248100
2023-11-21 20:44:09,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1653933.3333333333, ans=0.2
2023-11-21 20:44:17,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1654000.0, ans=0.1
2023-11-21 20:44:36,139 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7650, loss[loss=0.08603, simple_loss=0.1102, pruned_loss=0.02268, audio_tagging_loss=0.008226, over 15793.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09594, pruned_loss=0.01625, audio_tagging_loss=0.009456, over 3060118.58 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:44:48,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1654200.0, ans=0.125
2023-11-21 20:44:59,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1654200.0, ans=0.1
2023-11-21 20:45:11,794 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248150
2023-11-21 20:45:13,829 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.88 vs. limit=22.5
2023-11-21 20:45:37,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1654400.0, ans=0.5
2023-11-21 20:45:40,009 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7700, loss[loss=0.08135, simple_loss=0.111, pruned_loss=0.0187, audio_tagging_loss=0.007168, over 15279.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09559, pruned_loss=0.0161, audio_tagging_loss=0.009434, over 3062021.94 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:45:43,274 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.52 vs. limit=10.0
2023-11-21 20:45:46,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1654466.6666666667, ans=0.125
2023-11-21 20:45:48,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.33 vs. limit=15.0
2023-11-21 20:45:56,990 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.92 vs. limit=15.0
2023-11-21 20:46:04,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1654600.0, ans=0.95
2023-11-21 20:46:15,483 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.008e+01 8.521e+01 9.342e+01 1.167e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 20:46:15,630 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248200
2023-11-21 20:46:42,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.58 vs. limit=15.0
2023-11-21 20:46:43,446 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7750, loss[loss=0.05648, simple_loss=0.07222, pruned_loss=0.0123, audio_tagging_loss=0.00807, over 16355.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09524, pruned_loss=0.01611, audio_tagging_loss=0.009484, over 3062798.66 frames. ], batch size: 66, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:47:13,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1654933.3333333333, ans=0.125
2023-11-21 20:47:19,530 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248250
2023-11-21 20:47:46,957 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7800, loss[loss=0.06288, simple_loss=0.0736, pruned_loss=0.01604, audio_tagging_loss=0.01004, over 14799.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09495, pruned_loss=0.01601, audio_tagging_loss=0.00943, over 3058036.01 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:48:08,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1655200.0, ans=0.125
2023-11-21 20:48:08,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1655200.0, ans=0.125
2023-11-21 20:48:12,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1655266.6666666667, ans=0.0
2023-11-21 20:48:19,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1655266.6666666667, ans=0.025
2023-11-21 20:48:23,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248300
2023-11-21 20:48:24,375 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.033e+01 8.572e+01 9.400e+01 1.167e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 20:48:41,420 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.54 vs. limit=12.0
2023-11-21 20:48:43,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1655400.0, ans=0.125
2023-11-21 20:48:50,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1655466.6666666667, ans=0.1
2023-11-21 20:48:51,007 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7850, loss[loss=0.0567, simple_loss=0.06441, pruned_loss=0.01077, audio_tagging_loss=0.01372, over 14702.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09536, pruned_loss=0.01618, audio_tagging_loss=0.009536, over 3051896.18 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:49:04,628 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.66 vs. limit=15.0
2023-11-21 20:49:25,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248350
2023-11-21 20:49:53,825 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7900, loss[loss=0.08288, simple_loss=0.1115, pruned_loss=0.01805, audio_tagging_loss=0.009067, over 15723.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09518, pruned_loss=0.01626, audio_tagging_loss=0.009638, over 3052334.36 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:50:01,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten.whitening_limit, batch_count=1655800.0, ans=15.0
2023-11-21 20:50:20,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1655933.3333333333, ans=0.125
2023-11-21 20:50:21,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1655933.3333333333, ans=0.09899494936611666
2023-11-21 20:50:23,963 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:50:29,890 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248400
2023-11-21 20:50:32,500 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.166e+01 8.800e+01 9.422e+01 1.274e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 20:50:39,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1656000.0, ans=0.125
2023-11-21 20:50:43,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1656000.0, ans=0.125
2023-11-21 20:50:57,273 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 7950, loss[loss=0.06453, simple_loss=0.08885, pruned_loss=0.009149, audio_tagging_loss=0.01096, over 14747.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09493, pruned_loss=0.01614, audio_tagging_loss=0.009669, over 3052081.42 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:50:57,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1656133.3333333333, ans=0.0
2023-11-21 20:50:59,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1656133.3333333333, ans=0.1
2023-11-21 20:51:13,905 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:51:21,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1656200.0, ans=0.125
2023-11-21 20:51:29,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1656266.6666666667, ans=0.125
2023-11-21 20:51:34,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248450
2023-11-21 20:51:37,928 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:51:46,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1656333.3333333333, ans=0.2
2023-11-21 20:51:47,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1656400.0, ans=0.125
2023-11-21 20:51:56,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.14 vs. limit=15.0
2023-11-21 20:52:02,358 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8000, loss[loss=0.06618, simple_loss=0.08065, pruned_loss=0.01521, audio_tagging_loss=0.01065, over 14916.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09435, pruned_loss=0.01609, audio_tagging_loss=0.009793, over 3044933.35 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:52:06,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1656466.6666666667, ans=0.0
2023-11-21 20:52:35,183 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.50 vs. limit=15.0
2023-11-21 20:52:37,004 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248500
2023-11-21 20:52:37,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1656600.0, ans=0.0
2023-11-21 20:52:38,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1656666.6666666667, ans=0.125
2023-11-21 20:52:39,307 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.016e+01 8.965e+01 9.786e+01 1.299e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 20:53:05,615 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8050, loss[loss=0.1008, simple_loss=0.1318, pruned_loss=0.02771, audio_tagging_loss=0.007138, over 15581.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.0945, pruned_loss=0.01596, audio_tagging_loss=0.009854, over 3044042.89 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:53:05,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1656800.0, ans=0.125
2023-11-21 20:53:05,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1656800.0, ans=0.125
2023-11-21 20:53:16,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1656866.6666666667, ans=0.1
2023-11-21 20:53:38,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1656933.3333333333, ans=0.0
2023-11-21 20:53:41,137 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248550
2023-11-21 20:53:50,541 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:54:08,577 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8100, loss[loss=0.08382, simple_loss=0.1164, pruned_loss=0.01835, audio_tagging_loss=0.007267, over 15114.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09513, pruned_loss=0.01603, audio_tagging_loss=0.0097, over 3042409.26 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:54:23,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1657200.0, ans=0.125
2023-11-21 20:54:26,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1657200.0, ans=0.125
2023-11-21 20:54:29,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1657200.0, ans=0.125
2023-11-21 20:54:44,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248600
2023-11-21 20:54:47,356 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.022e+01 8.595e+01 9.139e+01 1.165e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 20:55:02,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1657400.0, ans=0.2
2023-11-21 20:55:12,715 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8150, loss[loss=0.07383, simple_loss=0.1008, pruned_loss=0.01533, audio_tagging_loss=0.008115, over 15341.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09552, pruned_loss=0.01612, audio_tagging_loss=0.009478, over 3048098.54 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:55:31,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1657533.3333333333, ans=0.04949747468305833
2023-11-21 20:55:47,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248650
2023-11-21 20:56:05,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1657733.3333333333, ans=0.125
2023-11-21 20:56:16,188 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8200, loss[loss=0.0867, simple_loss=0.1205, pruned_loss=0.01758, audio_tagging_loss=0.008861, over 15396.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09596, pruned_loss=0.01631, audio_tagging_loss=0.00945, over 3046445.11 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:56:16,243 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:56:32,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1657866.6666666667, ans=0.1
2023-11-21 20:56:33,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.94 vs. limit=10.0
2023-11-21 20:56:36,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1657866.6666666667, ans=0.035
2023-11-21 20:56:38,395 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.15 vs. limit=10.0
2023-11-21 20:56:47,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1657933.3333333333, ans=0.125
2023-11-21 20:56:50,630 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248700
2023-11-21 20:56:52,860 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.429e+01 8.006e+01 8.771e+01 9.726e+01 1.237e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 20:56:56,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1658000.0, ans=0.125
2023-11-21 20:57:18,322 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8250, loss[loss=0.06761, simple_loss=0.08647, pruned_loss=0.01472, audio_tagging_loss=0.009654, over 15806.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09677, pruned_loss=0.01643, audio_tagging_loss=0.009323, over 3049356.88 frames. ], batch size: 60, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:57:27,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1658133.3333333333, ans=0.1
2023-11-21 20:57:30,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.78 vs. limit=15.0
2023-11-21 20:57:34,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1658200.0, ans=0.0
2023-11-21 20:57:36,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1658200.0, ans=0.1
2023-11-21 20:57:53,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1658266.6666666667, ans=0.09899494936611666
2023-11-21 20:57:54,930 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248750
2023-11-21 20:58:08,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.20 vs. limit=15.0
2023-11-21 20:58:22,531 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8300, loss[loss=0.07536, simple_loss=0.09915, pruned_loss=0.0145, audio_tagging_loss=0.01128, over 15418.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09618, pruned_loss=0.01615, audio_tagging_loss=0.009381, over 3048765.98 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:58:23,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.44 vs. limit=15.0
2023-11-21 20:58:40,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1658533.3333333333, ans=0.125
2023-11-21 20:58:41,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1658533.3333333333, ans=0.1
2023-11-21 20:58:44,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1658533.3333333333, ans=0.125
2023-11-21 20:58:48,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0
2023-11-21 20:58:57,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1658600.0, ans=0.125
2023-11-21 20:58:58,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248800
2023-11-21 20:58:59,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1658666.6666666667, ans=0.2
2023-11-21 20:59:00,789 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.249e+01 8.152e+01 8.713e+01 9.319e+01 1.326e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 20:59:19,864 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.83 vs. limit=15.0
2023-11-21 20:59:27,564 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8350, loss[loss=0.05389, simple_loss=0.06456, pruned_loss=0.01126, audio_tagging_loss=0.01034, over 15028.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.0949, pruned_loss=0.01593, audio_tagging_loss=0.009412, over 3040942.31 frames. ], batch size: 57, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:59:27,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1658800.0, ans=0.1
2023-11-21 20:59:31,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1658800.0, ans=0.1
2023-11-21 20:59:51,453 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.92 vs. limit=10.0
2023-11-21 20:59:52,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1658933.3333333333, ans=0.1
2023-11-21 21:00:02,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248850
2023-11-21 21:00:02,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1658933.3333333333, ans=0.0
2023-11-21 21:00:14,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1659000.0, ans=0.0
2023-11-21 21:00:22,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1659066.6666666667, ans=0.2
2023-11-21 21:00:30,754 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8400, loss[loss=0.09464, simple_loss=0.1187, pruned_loss=0.0273, audio_tagging_loss=0.007967, over 14896.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09485, pruned_loss=0.01587, audio_tagging_loss=0.009431, over 3037720.87 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 32.0
2023-11-21 21:00:32,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1659133.3333333333, ans=0.1
2023-11-21 21:01:06,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248900
2023-11-21 21:01:09,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1659333.3333333333, ans=0.0
2023-11-21 21:01:10,209 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.422e+01 7.990e+01 8.773e+01 9.433e+01 1.119e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 21:01:10,751 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.69 vs. limit=15.0
2023-11-21 21:01:29,965 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.41 vs. limit=15.0
2023-11-21 21:01:33,137 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8450, loss[loss=0.1015, simple_loss=0.131, pruned_loss=0.02758, audio_tagging_loss=0.008462, over 15111.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09605, pruned_loss=0.01619, audio_tagging_loss=0.009261, over 3036858.40 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:01:33,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1659466.6666666667, ans=0.2
2023-11-21 21:01:41,885 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.15 vs. limit=15.0
2023-11-21 21:01:46,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1659533.3333333333, ans=0.09899494936611666
2023-11-21 21:01:47,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1659533.3333333333, ans=0.125
2023-11-21 21:02:08,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 248950
2023-11-21 21:02:09,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1659600.0, ans=0.125
2023-11-21 21:02:09,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=15.0
2023-11-21 21:02:11,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1659666.6666666667, ans=0.1
2023-11-21 21:02:16,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1659666.6666666667, ans=0.09899494936611666
2023-11-21 21:02:30,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1659733.3333333333, ans=0.1
2023-11-21 21:02:36,337 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8500, loss[loss=0.05969, simple_loss=0.0749, pruned_loss=0.01224, audio_tagging_loss=0.01, over 14793.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09562, pruned_loss=0.01615, audio_tagging_loss=0.009333, over 3035272.86 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:02:54,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1659866.6666666667, ans=0.125
2023-11-21 21:03:01,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1659933.3333333333, ans=0.1
2023-11-21 21:03:12,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249000
2023-11-21 21:03:12,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1659933.3333333333, ans=0.0
2023-11-21 21:03:13,284 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.29 vs. limit=15.0
2023-11-21 21:03:15,821 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.08 vs. limit=15.0
2023-11-21 21:03:16,325 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.238e+01 7.932e+01 8.481e+01 9.270e+01 1.196e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-21 21:03:25,301 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:03:32,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1660066.6666666667, ans=0.125
2023-11-21 21:03:41,018 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8550, loss[loss=0.06913, simple_loss=0.08685, pruned_loss=0.017, audio_tagging_loss=0.008696, over 15382.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09599, pruned_loss=0.01615, audio_tagging_loss=0.00931, over 3037089.27 frames. ], batch size: 59, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:03:41,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1660133.3333333333, ans=0.125
2023-11-21 21:03:58,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1660200.0, ans=0.125
2023-11-21 21:04:08,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1660266.6666666667, ans=0.1
2023-11-21 21:04:16,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249050
2023-11-21 21:04:32,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1660400.0, ans=0.125
2023-11-21 21:04:32,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1660400.0, ans=0.2
2023-11-21 21:04:37,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1660400.0, ans=0.125
2023-11-21 21:04:43,494 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8600, loss[loss=0.09627, simple_loss=0.1318, pruned_loss=0.02315, audio_tagging_loss=0.00722, over 15072.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09633, pruned_loss=0.01615, audio_tagging_loss=0.009421, over 3042320.58 frames. ], batch size: 53, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:04:49,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1660466.6666666667, ans=0.0
2023-11-21 21:04:50,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0
2023-11-21 21:04:50,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1660466.6666666667, ans=0.0
2023-11-21 21:05:04,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1660533.3333333333, ans=0.0
2023-11-21 21:05:16,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1660600.0, ans=0.125
2023-11-21 21:05:20,118 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249100
2023-11-21 21:05:20,740 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.84 vs. limit=22.5
2023-11-21 21:05:24,797 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.123e+01 8.590e+01 9.454e+01 1.253e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 21:05:27,939 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.88 vs. limit=22.5
2023-11-21 21:05:47,754 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8650, loss[loss=0.07532, simple_loss=0.09738, pruned_loss=0.01748, audio_tagging_loss=0.009147, over 16073.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09664, pruned_loss=0.01632, audio_tagging_loss=0.009399, over 3046647.80 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:05:50,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1660800.0, ans=0.125
2023-11-21 21:06:13,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1660933.3333333333, ans=0.025
2023-11-21 21:06:21,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1660933.3333333333, ans=0.04949747468305833
2023-11-21 21:06:22,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.68 vs. limit=15.0
2023-11-21 21:06:23,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249150
2023-11-21 21:06:51,505 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8700, loss[loss=0.09375, simple_loss=0.1157, pruned_loss=0.02478, audio_tagging_loss=0.01114, over 15850.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09666, pruned_loss=0.01624, audio_tagging_loss=0.009627, over 3049261.68 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:07:01,355 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0
2023-11-21 21:07:24,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1661266.6666666667, ans=0.125
2023-11-21 21:07:27,378 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249200
2023-11-21 21:07:32,752 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.120e+01 8.963e+01 9.729e+01 1.356e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 21:07:40,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.79 vs. limit=12.0
2023-11-21 21:07:40,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.48 vs. limit=22.5
2023-11-21 21:07:55,268 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8750, loss[loss=0.1008, simple_loss=0.1347, pruned_loss=0.02544, audio_tagging_loss=0.008054, over 16120.00 frames. ], tot_loss[loss=0.07414, simple_loss=0.09654, pruned_loss=0.01621, audio_tagging_loss=0.009661, over 3044877.45 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:08:31,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249250
2023-11-21 21:08:43,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1661666.6666666667, ans=0.2
2023-11-21 21:08:45,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1661733.3333333333, ans=0.125
2023-11-21 21:08:52,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1661733.3333333333, ans=0.0
2023-11-21 21:08:57,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1661733.3333333333, ans=0.125
2023-11-21 21:08:59,675 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8800, loss[loss=0.07189, simple_loss=0.09164, pruned_loss=0.01608, audio_tagging_loss=0.009993, over 14616.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09767, pruned_loss=0.01638, audio_tagging_loss=0.009662, over 3044038.75 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:09:09,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1661800.0, ans=0.2
2023-11-21 21:09:15,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1661866.6666666667, ans=0.0
2023-11-21 21:09:35,365 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249300
2023-11-21 21:09:40,741 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.249e+01 8.864e+01 9.702e+01 1.215e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 21:09:41,444 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.34 vs. limit=15.0
2023-11-21 21:10:03,866 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8850, loss[loss=0.05699, simple_loss=0.06857, pruned_loss=0.01119, audio_tagging_loss=0.01152, over 16778.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.097, pruned_loss=0.01609, audio_tagging_loss=0.00963, over 3055289.77 frames. ], batch size: 66, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:10:15,541 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:10:39,610 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249350
2023-11-21 21:10:48,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1662333.3333333333, ans=0.125
2023-11-21 21:10:51,253 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:10:59,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1662400.0, ans=0.2
2023-11-21 21:11:00,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1662400.0, ans=0.125
2023-11-21 21:11:06,477 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:11:06,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1662466.6666666667, ans=0.0
2023-11-21 21:11:07,400 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8900, loss[loss=0.06743, simple_loss=0.08331, pruned_loss=0.01446, audio_tagging_loss=0.01131, over 14725.00 frames. ], tot_loss[loss=0.07398, simple_loss=0.09658, pruned_loss=0.01619, audio_tagging_loss=0.009495, over 3046681.14 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:11:09,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1662466.6666666667, ans=0.2
2023-11-21 21:11:37,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1662600.0, ans=0.2
2023-11-21 21:11:43,070 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249400
2023-11-21 21:11:49,422 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.502e+01 8.199e+01 8.684e+01 9.586e+01 1.559e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 21:12:12,031 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 8950, loss[loss=0.07497, simple_loss=0.08885, pruned_loss=0.01692, audio_tagging_loss=0.01363, over 15612.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09593, pruned_loss=0.01604, audio_tagging_loss=0.009394, over 3049912.57 frames. ], batch size: 63, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:12:17,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.44 vs. limit=22.5
2023-11-21 21:12:46,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1662933.3333333333, ans=0.0
2023-11-21 21:12:47,149 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249450
2023-11-21 21:13:00,028 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.48 vs. limit=22.5
2023-11-21 21:13:01,121 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.23 vs. limit=22.5
2023-11-21 21:13:01,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.65 vs. limit=15.0
2023-11-21 21:13:09,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1663066.6666666667, ans=0.125
2023-11-21 21:13:15,723 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9000, loss[loss=0.08312, simple_loss=0.1074, pruned_loss=0.02146, audio_tagging_loss=0.007957, over 14496.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.09703, pruned_loss=0.01627, audio_tagging_loss=0.009248, over 3053237.13 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:13:15,723 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 21:13:56,859 INFO [train_asr.py:1253] (2/4) Epoch 21, validation: loss=0.06003, simple_loss=0.05194, pruned_loss=0.005168, audio_tagging_loss=0.0289, over 4681554.00 frames. 
2023-11-21 21:13:56,859 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 21:14:32,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249500
2023-11-21 21:14:38,733 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 7.983e+01 8.777e+01 1.012e+02 1.174e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 21:14:58,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1663400.0, ans=0.125
2023-11-21 21:15:01,130 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9050, loss[loss=0.07994, simple_loss=0.111, pruned_loss=0.01763, audio_tagging_loss=0.006817, over 15739.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09672, pruned_loss=0.01629, audio_tagging_loss=0.009251, over 3048929.62 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:15:11,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1663466.6666666667, ans=0.1
2023-11-21 21:15:30,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.63 vs. limit=22.5
2023-11-21 21:15:36,761 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249550
2023-11-21 21:15:42,571 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:15:48,060 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.12 vs. limit=15.0
2023-11-21 21:15:58,943 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:16:04,951 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9100, loss[loss=0.1057, simple_loss=0.1395, pruned_loss=0.03006, audio_tagging_loss=0.005875, over 15664.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09634, pruned_loss=0.01615, audio_tagging_loss=0.009252, over 3041810.85 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:16:28,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1663866.6666666667, ans=0.1
2023-11-21 21:16:40,994 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249600
2023-11-21 21:16:47,980 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.138e+01 8.810e+01 9.417e+01 1.142e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 21:17:06,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1664066.6666666667, ans=0.04949747468305833
2023-11-21 21:17:08,748 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9150, loss[loss=0.05719, simple_loss=0.08032, pruned_loss=0.009379, audio_tagging_loss=0.007653, over 16195.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.0966, pruned_loss=0.01623, audio_tagging_loss=0.009282, over 3041103.77 frames. ], batch size: 62, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:17:08,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1664133.3333333333, ans=0.0
2023-11-21 21:17:28,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1664200.0, ans=0.0
2023-11-21 21:17:31,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1664200.0, ans=0.1
2023-11-21 21:17:46,005 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249650
2023-11-21 21:17:55,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=12.0
2023-11-21 21:17:56,563 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.53 vs. limit=10.0
2023-11-21 21:18:11,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1664400.0, ans=0.125
2023-11-21 21:18:14,540 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9200, loss[loss=0.04762, simple_loss=0.06052, pruned_loss=0.007255, audio_tagging_loss=0.0101, over 16390.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09579, pruned_loss=0.01594, audio_tagging_loss=0.00934, over 3042974.99 frames. ], batch size: 64, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:18:19,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1664466.6666666667, ans=0.1
2023-11-21 21:18:31,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1664533.3333333333, ans=0.0
2023-11-21 21:18:39,227 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.87 vs. limit=15.0
2023-11-21 21:18:43,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1664600.0, ans=0.0
2023-11-21 21:18:49,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249700
2023-11-21 21:18:56,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.81 vs. limit=10.0
2023-11-21 21:18:56,919 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.394e+01 8.125e+01 8.694e+01 9.340e+01 1.162e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 21:19:12,835 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.50 vs. limit=15.0
2023-11-21 21:19:19,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.66 vs. limit=15.0
2023-11-21 21:19:19,596 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9250, loss[loss=0.09185, simple_loss=0.124, pruned_loss=0.02165, audio_tagging_loss=0.008217, over 15563.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.0962, pruned_loss=0.01604, audio_tagging_loss=0.009417, over 3042286.76 frames. ], batch size: 60, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:19:22,732 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.42 vs. limit=15.0
2023-11-21 21:19:41,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1664866.6666666667, ans=0.125
2023-11-21 21:19:55,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249750
2023-11-21 21:20:00,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1665000.0, ans=0.0
2023-11-21 21:20:03,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1665000.0, ans=0.0
2023-11-21 21:20:23,908 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9300, loss[loss=0.06712, simple_loss=0.1001, pruned_loss=0.009254, audio_tagging_loss=0.007797, over 14650.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09674, pruned_loss=0.01627, audio_tagging_loss=0.009375, over 3041633.85 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:20:39,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.60 vs. limit=15.0
2023-11-21 21:20:45,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1665200.0, ans=0.125
2023-11-21 21:20:50,789 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:20:59,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1665266.6666666667, ans=0.0
2023-11-21 21:20:59,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1665266.6666666667, ans=0.0
2023-11-21 21:21:00,359 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249800
2023-11-21 21:21:04,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1665333.3333333333, ans=0.0
2023-11-21 21:21:06,634 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.134e+01 8.648e+01 9.186e+01 1.314e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 21:21:21,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1665400.0, ans=0.125
2023-11-21 21:21:29,041 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9350, loss[loss=0.09471, simple_loss=0.1303, pruned_loss=0.02293, audio_tagging_loss=0.006639, over 15622.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09665, pruned_loss=0.01627, audio_tagging_loss=0.009481, over 3046643.05 frames. ], batch size: 57, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:21:41,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1665533.3333333333, ans=0.0
2023-11-21 21:22:03,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1665600.0, ans=0.125
2023-11-21 21:22:04,377 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249850
2023-11-21 21:22:07,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1665666.6666666667, ans=0.125
2023-11-21 21:22:09,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1665666.6666666667, ans=0.0
2023-11-21 21:22:33,850 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9400, loss[loss=0.07904, simple_loss=0.1062, pruned_loss=0.01601, audio_tagging_loss=0.009908, over 15337.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09642, pruned_loss=0.0161, audio_tagging_loss=0.009459, over 3044867.13 frames. ], batch size: 59, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:22:43,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1665800.0, ans=0.125
2023-11-21 21:22:55,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1665866.6666666667, ans=0.125
2023-11-21 21:23:09,725 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249900
2023-11-21 21:23:15,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1666000.0, ans=0.0
2023-11-21 21:23:16,397 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.499e+01 7.941e+01 8.716e+01 9.449e+01 2.230e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 21:23:34,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1666066.6666666667, ans=0.125
2023-11-21 21:23:35,618 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:23:37,630 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.34 vs. limit=15.0
2023-11-21 21:23:38,009 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9450, loss[loss=0.06381, simple_loss=0.08246, pruned_loss=0.01316, audio_tagging_loss=0.009421, over 15854.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09696, pruned_loss=0.01631, audio_tagging_loss=0.009482, over 3042104.49 frames. ], batch size: 60, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:23:44,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1666133.3333333333, ans=0.0
2023-11-21 21:23:49,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1666133.3333333333, ans=0.2
2023-11-21 21:23:55,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1666200.0, ans=0.125
2023-11-21 21:24:09,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1666266.6666666667, ans=0.1
2023-11-21 21:24:10,145 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.12 vs. limit=10.0
2023-11-21 21:24:14,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 249950
2023-11-21 21:24:15,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1666333.3333333333, ans=0.125
2023-11-21 21:24:17,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1666333.3333333333, ans=0.0
2023-11-21 21:24:25,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1666333.3333333333, ans=0.07
2023-11-21 21:24:31,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1666400.0, ans=0.0
2023-11-21 21:24:31,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1666400.0, ans=0.07
2023-11-21 21:24:42,633 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9500, loss[loss=0.05659, simple_loss=0.0663, pruned_loss=0.01306, audio_tagging_loss=0.01039, over 14713.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.0966, pruned_loss=0.01629, audio_tagging_loss=0.009648, over 3049987.90 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:25:06,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1666533.3333333333, ans=0.125
2023-11-21 21:25:10,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1666600.0, ans=0.1
2023-11-21 21:25:13,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1666600.0, ans=0.0
2023-11-21 21:25:17,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1666600.0, ans=0.0
2023-11-21 21:25:17,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1666600.0, ans=0.125
2023-11-21 21:25:17,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1666600.0, ans=0.125
2023-11-21 21:25:18,567 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250000
2023-11-21 21:25:20,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1666666.6666666667, ans=0.125
2023-11-21 21:25:22,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1666666.6666666667, ans=0.1
2023-11-21 21:25:23,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1666666.6666666667, ans=0.125
2023-11-21 21:25:24,226 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.78 vs. limit=22.5
2023-11-21 21:25:26,596 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.269e+01 8.539e+01 8.975e+01 9.633e+01 1.238e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-21 21:25:48,001 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9550, loss[loss=0.06094, simple_loss=0.07273, pruned_loss=0.008771, audio_tagging_loss=0.0158, over 14989.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09691, pruned_loss=0.01633, audio_tagging_loss=0.009701, over 3049285.48 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:25:57,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1666800.0, ans=0.125
2023-11-21 21:26:18,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1666933.3333333333, ans=0.0
2023-11-21 21:26:24,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250050
2023-11-21 21:26:28,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1667000.0, ans=0.0
2023-11-21 21:26:31,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1667000.0, ans=0.0
2023-11-21 21:26:53,394 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9600, loss[loss=0.07277, simple_loss=0.09347, pruned_loss=0.01245, audio_tagging_loss=0.01359, over 14558.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09626, pruned_loss=0.01613, audio_tagging_loss=0.009756, over 3049162.00 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:27:20,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1667266.6666666667, ans=0.125
2023-11-21 21:27:29,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1667266.6666666667, ans=0.125
2023-11-21 21:27:30,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250100
2023-11-21 21:27:32,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1667333.3333333333, ans=0.2
2023-11-21 21:27:38,200 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 8.305e+01 8.780e+01 9.351e+01 2.082e+02, threshold=1.756e+02, percent-clipped=1.0
2023-11-21 21:27:40,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1667333.3333333333, ans=0.0
2023-11-21 21:27:56,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1667400.0, ans=0.125
2023-11-21 21:27:56,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.19 vs. limit=10.0
2023-11-21 21:27:58,573 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9650, loss[loss=0.0677, simple_loss=0.0917, pruned_loss=0.01354, audio_tagging_loss=0.008308, over 15541.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09661, pruned_loss=0.01626, audio_tagging_loss=0.009644, over 3047933.42 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:28:03,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1667466.6666666667, ans=0.125
2023-11-21 21:28:09,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.51 vs. limit=15.0
2023-11-21 21:28:30,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1667600.0, ans=0.125
2023-11-21 21:28:34,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250150
2023-11-21 21:28:59,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1667733.3333333333, ans=0.125
2023-11-21 21:29:03,974 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9700, loss[loss=0.08937, simple_loss=0.1235, pruned_loss=0.02001, audio_tagging_loss=0.007619, over 15581.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.0975, pruned_loss=0.01631, audio_tagging_loss=0.009454, over 3055269.34 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:29:12,237 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.45 vs. limit=15.0
2023-11-21 21:29:18,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1667866.6666666667, ans=0.125
2023-11-21 21:29:20,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1667866.6666666667, ans=0.125
2023-11-21 21:29:28,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1667933.3333333333, ans=0.07
2023-11-21 21:29:40,206 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250200
2023-11-21 21:29:47,782 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.250e+01 8.691e+01 9.260e+01 1.203e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 21:30:04,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1668066.6666666667, ans=0.125
2023-11-21 21:30:09,329 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9750, loss[loss=0.08772, simple_loss=0.1199, pruned_loss=0.01972, audio_tagging_loss=0.008064, over 15730.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09727, pruned_loss=0.01638, audio_tagging_loss=0.009387, over 3052094.31 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:30:20,762 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:30:22,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1668200.0, ans=0.0
2023-11-21 21:30:29,317 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.44 vs. limit=22.5
2023-11-21 21:30:38,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1668266.6666666667, ans=0.1
2023-11-21 21:30:38,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1668266.6666666667, ans=0.125
2023-11-21 21:30:46,097 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250250
2023-11-21 21:30:52,440 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:31:13,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1668466.6666666667, ans=0.2
2023-11-21 21:31:14,295 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9800, loss[loss=0.07784, simple_loss=0.108, pruned_loss=0.01455, audio_tagging_loss=0.009281, over 15852.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09619, pruned_loss=0.0162, audio_tagging_loss=0.009448, over 3054782.67 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:31:21,747 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.65 vs. limit=15.0
2023-11-21 21:31:22,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1668466.6666666667, ans=0.0
2023-11-21 21:31:44,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1668600.0, ans=0.125
2023-11-21 21:31:50,404 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250300
2023-11-21 21:31:58,402 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 7.987e+01 8.487e+01 9.363e+01 1.387e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-21 21:32:06,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1668733.3333333333, ans=0.0
2023-11-21 21:32:11,917 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:32:13,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1668733.3333333333, ans=0.09899494936611666
2023-11-21 21:32:19,230 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9850, loss[loss=0.06149, simple_loss=0.07548, pruned_loss=0.01169, audio_tagging_loss=0.01206, over 14737.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.0957, pruned_loss=0.01611, audio_tagging_loss=0.009478, over 3049689.14 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:32:29,901 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:32:39,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1668866.6666666667, ans=0.0
2023-11-21 21:32:55,329 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250350
2023-11-21 21:33:00,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1669000.0, ans=0.0
2023-11-21 21:33:12,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1669066.6666666667, ans=0.05
2023-11-21 21:33:23,704 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9900, loss[loss=0.06943, simple_loss=0.09079, pruned_loss=0.01604, audio_tagging_loss=0.007999, over 14864.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09633, pruned_loss=0.01614, audio_tagging_loss=0.009365, over 3050164.17 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:33:28,110 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.70 vs. limit=15.0
2023-11-21 21:33:35,007 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.80 vs. limit=15.0
2023-11-21 21:33:58,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1669266.6666666667, ans=0.125
2023-11-21 21:34:00,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250400
2023-11-21 21:34:07,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1669333.3333333333, ans=0.125
2023-11-21 21:34:08,495 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.181e+01 8.919e+01 9.368e+01 2.171e+02, threshold=1.784e+02, percent-clipped=1.0
2023-11-21 21:34:25,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1669400.0, ans=0.125
2023-11-21 21:34:28,649 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 9950, loss[loss=0.06492, simple_loss=0.08442, pruned_loss=0.01129, audio_tagging_loss=0.01142, over 14472.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09638, pruned_loss=0.01608, audio_tagging_loss=0.009347, over 3044455.90 frames. ], batch size: 54, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:34:40,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.32 vs. limit=15.0
2023-11-21 21:34:50,390 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:34:52,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1669533.3333333333, ans=0.125
2023-11-21 21:35:04,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250450
2023-11-21 21:35:28,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1669733.3333333333, ans=0.04949747468305833
2023-11-21 21:35:30,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1669733.3333333333, ans=15.0
2023-11-21 21:35:33,063 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10000, loss[loss=0.05782, simple_loss=0.07957, pruned_loss=0.009778, audio_tagging_loss=0.008257, over 15327.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09474, pruned_loss=0.01586, audio_tagging_loss=0.009333, over 3046370.15 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:35:37,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1669800.0, ans=0.2
2023-11-21 21:35:51,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.13 vs. limit=15.0
2023-11-21 21:35:59,065 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.87 vs. limit=15.0
2023-11-21 21:36:09,486 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250500
2023-11-21 21:36:16,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1670000.0, ans=0.125
2023-11-21 21:36:17,279 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.484e+01 8.244e+01 8.958e+01 9.849e+01 1.581e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-21 21:36:37,531 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10050, loss[loss=0.07173, simple_loss=0.09079, pruned_loss=0.01626, audio_tagging_loss=0.01007, over 14546.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09484, pruned_loss=0.01596, audio_tagging_loss=0.009454, over 3042172.56 frames. ], batch size: 54, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:37:01,100 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.65 vs. limit=10.0
2023-11-21 21:37:05,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1670266.6666666667, ans=0.125
2023-11-21 21:37:13,265 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250550
2023-11-21 21:37:13,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1670266.6666666667, ans=0.0
2023-11-21 21:37:13,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1670266.6666666667, ans=0.125
2023-11-21 21:37:28,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1670400.0, ans=0.05
2023-11-21 21:37:39,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1670466.6666666667, ans=0.125
2023-11-21 21:37:40,401 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10100, loss[loss=0.1006, simple_loss=0.1444, pruned_loss=0.02014, audio_tagging_loss=0.00821, over 17765.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.0951, pruned_loss=0.01587, audio_tagging_loss=0.009516, over 3045848.83 frames. ], batch size: 63, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:38:06,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1670600.0, ans=0.125
2023-11-21 21:38:10,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1670600.0, ans=0.125
2023-11-21 21:38:12,526 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0
2023-11-21 21:38:16,948 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250600
2023-11-21 21:38:25,701 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.286e+01 8.086e+01 8.639e+01 9.260e+01 1.282e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 21:38:27,790 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-21 21:38:31,770 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:38:33,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1670733.3333333333, ans=0.125
2023-11-21 21:38:35,702 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0
2023-11-21 21:38:36,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1670733.3333333333, ans=0.2
2023-11-21 21:38:43,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1670733.3333333333, ans=0.1
2023-11-21 21:38:45,759 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10150, loss[loss=0.0817, simple_loss=0.1178, pruned_loss=0.01374, audio_tagging_loss=0.00904, over 15885.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09562, pruned_loss=0.01592, audio_tagging_loss=0.00961, over 3046140.67 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:38:54,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1670800.0, ans=0.2
2023-11-21 21:39:14,496 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:39:20,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250650
2023-11-21 21:39:25,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1671000.0, ans=0.0
2023-11-21 21:39:37,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1671066.6666666667, ans=0.125
2023-11-21 21:39:38,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1671066.6666666667, ans=0.125
2023-11-21 21:39:49,973 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10200, loss[loss=0.08254, simple_loss=0.1108, pruned_loss=0.01709, audio_tagging_loss=0.01003, over 15810.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.0959, pruned_loss=0.01608, audio_tagging_loss=0.009613, over 3050806.60 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:39:55,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1671133.3333333333, ans=0.125
2023-11-21 21:39:58,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1671133.3333333333, ans=0.125
2023-11-21 21:40:00,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1671133.3333333333, ans=0.1
2023-11-21 21:40:12,560 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:40:16,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1671266.6666666667, ans=0.2
2023-11-21 21:40:26,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250700
2023-11-21 21:40:28,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1671333.3333333333, ans=0.125
2023-11-21 21:40:35,574 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.067e+01 8.565e+01 9.348e+01 1.300e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 21:40:44,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1671400.0, ans=0.125
2023-11-21 21:40:45,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1671400.0, ans=0.0
2023-11-21 21:40:54,046 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10250, loss[loss=0.06717, simple_loss=0.08535, pruned_loss=0.01422, audio_tagging_loss=0.01027, over 16547.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09584, pruned_loss=0.01616, audio_tagging_loss=0.009687, over 3049879.34 frames. ], batch size: 62, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:41:00,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1671466.6666666667, ans=0.125
2023-11-21 21:41:07,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1671533.3333333333, ans=0.0
2023-11-21 21:41:08,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1671533.3333333333, ans=0.125
2023-11-21 21:41:13,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1671533.3333333333, ans=0.0
2023-11-21 21:41:27,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1671600.0, ans=0.2
2023-11-21 21:41:30,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250750
2023-11-21 21:41:42,876 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.88 vs. limit=15.0
2023-11-21 21:41:58,077 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10300, loss[loss=0.06688, simple_loss=0.09321, pruned_loss=0.0104, audio_tagging_loss=0.009879, over 15933.00 frames. ], tot_loss[loss=0.07414, simple_loss=0.09643, pruned_loss=0.0162, audio_tagging_loss=0.009728, over 3051074.29 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:42:33,107 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250800
2023-11-21 21:42:37,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1672000.0, ans=0.125
2023-11-21 21:42:40,963 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.48 vs. limit=22.5
2023-11-21 21:42:42,610 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.124e+01 8.842e+01 9.369e+01 1.260e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 21:42:56,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1672066.6666666667, ans=0.125
2023-11-21 21:42:57,299 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.39 vs. limit=22.5
2023-11-21 21:43:03,275 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10350, loss[loss=0.07463, simple_loss=0.09638, pruned_loss=0.01501, audio_tagging_loss=0.01143, over 14121.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09629, pruned_loss=0.01615, audio_tagging_loss=0.009787, over 3044755.42 frames. ], batch size: 54, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:43:39,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250850
2023-11-21 21:43:50,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1672333.3333333333, ans=0.1
2023-11-21 21:43:51,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1672333.3333333333, ans=0.1
2023-11-21 21:44:07,042 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10400, loss[loss=0.0611, simple_loss=0.07859, pruned_loss=0.0104, audio_tagging_loss=0.01141, over 14714.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09496, pruned_loss=0.01596, audio_tagging_loss=0.009944, over 3042996.49 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:44:20,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.59 vs. limit=15.0
2023-11-21 21:44:37,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1672600.0, ans=0.0
2023-11-21 21:44:43,936 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250900
2023-11-21 21:44:45,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1672666.6666666667, ans=0.2
2023-11-21 21:44:52,353 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.219e+01 8.868e+01 9.725e+01 1.469e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 21:45:12,299 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10450, loss[loss=0.05069, simple_loss=0.0615, pruned_loss=0.01035, audio_tagging_loss=0.009587, over 14692.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09448, pruned_loss=0.016, audio_tagging_loss=0.009846, over 3038657.90 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:45:13,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1672800.0, ans=0.0
2023-11-21 21:45:39,123 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=11.33 vs. limit=15.0
2023-11-21 21:45:39,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1672933.3333333333, ans=0.125
2023-11-21 21:45:43,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1672933.3333333333, ans=0.125
2023-11-21 21:45:48,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 250950
2023-11-21 21:45:48,577 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=15.97 vs. limit=15.0
2023-11-21 21:45:51,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1673000.0, ans=0.07
2023-11-21 21:45:54,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1673000.0, ans=0.125
2023-11-21 21:45:56,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1673000.0, ans=0.125
2023-11-21 21:46:13,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1673066.6666666667, ans=0.125
2023-11-21 21:46:17,698 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10500, loss[loss=0.07118, simple_loss=0.1033, pruned_loss=0.01265, audio_tagging_loss=0.006894, over 15308.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09426, pruned_loss=0.01576, audio_tagging_loss=0.009752, over 3035382.96 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:46:18,034 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:46:48,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1673266.6666666667, ans=0.125
2023-11-21 21:46:54,139 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251000
2023-11-21 21:46:57,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.05 vs. limit=15.0
2023-11-21 21:47:01,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1673333.3333333333, ans=0.0
2023-11-21 21:47:05,261 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.294e+01 8.913e+01 9.437e+01 1.275e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 21:47:12,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1673400.0, ans=0.125
2023-11-21 21:47:20,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1673400.0, ans=0.1
2023-11-21 21:47:23,406 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10550, loss[loss=0.1049, simple_loss=0.135, pruned_loss=0.02672, audio_tagging_loss=0.0107, over 14631.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09439, pruned_loss=0.01584, audio_tagging_loss=0.009699, over 3032542.83 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:48:00,766 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251050
2023-11-21 21:48:20,884 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:48:28,727 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10600, loss[loss=0.06693, simple_loss=0.07479, pruned_loss=0.01642, audio_tagging_loss=0.01311, over 14777.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.095, pruned_loss=0.01604, audio_tagging_loss=0.009622, over 3027191.39 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:48:50,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.89 vs. limit=22.5
2023-11-21 21:48:53,407 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.89 vs. limit=15.0
2023-11-21 21:48:59,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.83 vs. limit=15.0
2023-11-21 21:49:05,066 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251100
2023-11-21 21:49:07,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.31 vs. limit=5.0
2023-11-21 21:49:09,503 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.59 vs. limit=15.0
2023-11-21 21:49:15,226 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.229e+01 8.713e+01 9.313e+01 1.246e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 21:49:18,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1674000.0, ans=10.0
2023-11-21 21:49:33,260 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10650, loss[loss=0.06964, simple_loss=0.09608, pruned_loss=0.01498, audio_tagging_loss=0.006621, over 16145.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.09514, pruned_loss=0.01597, audio_tagging_loss=0.009591, over 3037535.71 frames. ], batch size: 62, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:49:48,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1674200.0, ans=0.125
2023-11-21 21:49:50,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1674200.0, ans=0.125
2023-11-21 21:50:06,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1674266.6666666667, ans=0.0
2023-11-21 21:50:09,595 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251150
2023-11-21 21:50:38,632 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10700, loss[loss=0.06682, simple_loss=0.09004, pruned_loss=0.01367, audio_tagging_loss=0.008125, over 16129.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09555, pruned_loss=0.01601, audio_tagging_loss=0.009594, over 3038349.06 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:50:54,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.54 vs. limit=15.0
2023-11-21 21:51:14,764 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251200
2023-11-21 21:51:22,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1674666.6666666667, ans=0.125
2023-11-21 21:51:25,543 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 8.099e+01 8.796e+01 9.421e+01 3.240e+02, threshold=1.759e+02, percent-clipped=1.0
2023-11-21 21:51:25,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1674666.6666666667, ans=0.0
2023-11-21 21:51:43,386 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10750, loss[loss=0.06482, simple_loss=0.08912, pruned_loss=0.0137, audio_tagging_loss=0.006565, over 15372.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09489, pruned_loss=0.01597, audio_tagging_loss=0.009584, over 3042255.14 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:51:53,490 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.04 vs. limit=12.0
2023-11-21 21:52:19,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251250
2023-11-21 21:52:20,238 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.03 vs. limit=15.0
2023-11-21 21:52:22,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1675000.0, ans=0.0
2023-11-21 21:52:47,588 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10800, loss[loss=0.06911, simple_loss=0.07996, pruned_loss=0.01698, audio_tagging_loss=0.01216, over 14606.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09419, pruned_loss=0.01571, audio_tagging_loss=0.009609, over 3039230.73 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:52:49,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1675133.3333333333, ans=0.125
2023-11-21 21:53:06,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1675200.0, ans=0.0
2023-11-21 21:53:23,701 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251300
2023-11-21 21:53:29,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1675333.3333333333, ans=0.125
2023-11-21 21:53:34,036 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.266e+01 8.735e+01 9.337e+01 1.123e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 21:53:42,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1675400.0, ans=0.1
2023-11-21 21:53:52,957 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10850, loss[loss=0.08175, simple_loss=0.1047, pruned_loss=0.02097, audio_tagging_loss=0.008414, over 15268.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09463, pruned_loss=0.01579, audio_tagging_loss=0.009579, over 3032005.68 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:53:53,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1675466.6666666667, ans=0.125
2023-11-21 21:54:15,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1675533.3333333333, ans=0.125
2023-11-21 21:54:28,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251350
2023-11-21 21:54:37,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1675666.6666666667, ans=0.0
2023-11-21 21:54:37,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1675666.6666666667, ans=0.125
2023-11-21 21:54:51,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1675733.3333333333, ans=0.125
2023-11-21 21:54:52,753 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:54:56,560 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10900, loss[loss=0.08527, simple_loss=0.1129, pruned_loss=0.02032, audio_tagging_loss=0.00852, over 14895.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09448, pruned_loss=0.01579, audio_tagging_loss=0.009639, over 3033095.42 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:54:57,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.79 vs. limit=15.0
2023-11-21 21:55:20,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1675866.6666666667, ans=0.125
2023-11-21 21:55:28,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1675933.3333333333, ans=0.0
2023-11-21 21:55:33,594 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251400
2023-11-21 21:55:39,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1676000.0, ans=0.125
2023-11-21 21:55:43,667 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.153e+01 8.721e+01 9.251e+01 1.356e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 21:56:02,088 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 10950, loss[loss=0.06755, simple_loss=0.08568, pruned_loss=0.0137, audio_tagging_loss=0.01101, over 16512.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09457, pruned_loss=0.01572, audio_tagging_loss=0.009697, over 3034363.21 frames. ], batch size: 60, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:56:03,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.61 vs. limit=15.0
2023-11-21 21:56:04,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1676133.3333333333, ans=0.125
2023-11-21 21:56:06,102 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:56:06,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1676133.3333333333, ans=0.09899494936611666
2023-11-21 21:56:16,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1676200.0, ans=0.125
2023-11-21 21:56:17,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1676200.0, ans=0.05
2023-11-21 21:56:29,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1676266.6666666667, ans=15.0
2023-11-21 21:56:34,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1676266.6666666667, ans=0.0
2023-11-21 21:56:35,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1676266.6666666667, ans=0.125
2023-11-21 21:56:37,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251450
2023-11-21 21:56:51,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1676333.3333333333, ans=0.1
2023-11-21 21:57:06,329 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11000, loss[loss=0.07012, simple_loss=0.09045, pruned_loss=0.0152, audio_tagging_loss=0.009697, over 14606.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09455, pruned_loss=0.01565, audio_tagging_loss=0.009704, over 3034369.71 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:57:16,909 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:57:30,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.88 vs. limit=6.0
2023-11-21 21:57:31,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=22.5
2023-11-21 21:57:34,027 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.92 vs. limit=6.0
2023-11-21 21:57:36,076 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:57:39,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1676600.0, ans=0.2
2023-11-21 21:57:41,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251500
2023-11-21 21:57:53,219 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.659e+01 8.207e+01 8.781e+01 9.534e+01 1.219e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-21 21:57:54,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1676666.6666666667, ans=0.125
2023-11-21 21:58:09,748 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11050, loss[loss=0.08792, simple_loss=0.1178, pruned_loss=0.02031, audio_tagging_loss=0.008697, over 14946.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09425, pruned_loss=0.01557, audio_tagging_loss=0.009753, over 3041031.57 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:58:24,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1676866.6666666667, ans=0.0
2023-11-21 21:58:45,409 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251550
2023-11-21 21:59:03,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1677066.6666666667, ans=0.0
2023-11-21 21:59:08,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1677066.6666666667, ans=0.0
2023-11-21 21:59:14,239 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11100, loss[loss=0.0803, simple_loss=0.1092, pruned_loss=0.01485, audio_tagging_loss=0.01085, over 15317.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09435, pruned_loss=0.01562, audio_tagging_loss=0.009818, over 3040335.79 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:59:20,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1677133.3333333333, ans=0.07
2023-11-21 21:59:40,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1677266.6666666667, ans=0.2
2023-11-21 21:59:50,056 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251600
2023-11-21 21:59:54,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1677333.3333333333, ans=0.0
2023-11-21 21:59:59,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1677333.3333333333, ans=0.0
2023-11-21 22:00:02,563 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.524e+01 9.104e+01 1.000e+02 2.938e+02, threshold=1.821e+02, percent-clipped=1.0
2023-11-21 22:00:18,950 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11150, loss[loss=0.04223, simple_loss=0.04405, pruned_loss=0.005872, audio_tagging_loss=0.01433, over 16250.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09457, pruned_loss=0.01581, audio_tagging_loss=0.009878, over 3045574.10 frames. ], batch size: 63, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:00:31,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1677533.3333333333, ans=0.1
2023-11-21 22:00:50,360 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:00:55,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251650
2023-11-21 22:01:22,782 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11200, loss[loss=0.05338, simple_loss=0.06083, pruned_loss=0.007549, audio_tagging_loss=0.01541, over 15501.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09473, pruned_loss=0.0159, audio_tagging_loss=0.009903, over 3048373.50 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:01:24,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1677800.0, ans=0.125
2023-11-21 22:01:30,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1677800.0, ans=0.04949747468305833
2023-11-21 22:01:42,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1677866.6666666667, ans=0.0
2023-11-21 22:01:59,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251700
2023-11-21 22:02:10,488 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 7.989e+01 8.616e+01 9.280e+01 1.646e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 22:02:28,528 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11250, loss[loss=0.08794, simple_loss=0.1151, pruned_loss=0.01991, audio_tagging_loss=0.0105, over 15721.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09402, pruned_loss=0.01586, audio_tagging_loss=0.00993, over 3053019.02 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:02:28,933 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:02:53,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1678266.6666666667, ans=0.0
2023-11-21 22:03:03,361 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251750
2023-11-21 22:03:14,914 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.69 vs. limit=15.0
2023-11-21 22:03:31,990 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11300, loss[loss=0.0954, simple_loss=0.12, pruned_loss=0.02443, audio_tagging_loss=0.01095, over 13885.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09431, pruned_loss=0.01587, audio_tagging_loss=0.009683, over 3048076.89 frames. ], batch size: 54, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:03:39,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1678466.6666666667, ans=0.0
2023-11-21 22:03:52,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1678533.3333333333, ans=0.1
2023-11-21 22:04:01,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1678600.0, ans=0.125
2023-11-21 22:04:02,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1678600.0, ans=0.1
2023-11-21 22:04:08,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251800
2023-11-21 22:04:08,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1678600.0, ans=0.0
2023-11-21 22:04:12,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1678666.6666666667, ans=0.1
2023-11-21 22:04:19,970 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.032e+01 8.626e+01 9.245e+01 1.282e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 22:04:25,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1678733.3333333333, ans=0.1
2023-11-21 22:04:36,719 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11350, loss[loss=0.07137, simple_loss=0.083, pruned_loss=0.01752, audio_tagging_loss=0.01235, over 15562.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09488, pruned_loss=0.01614, audio_tagging_loss=0.009585, over 3044222.48 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:05:12,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251850
2023-11-21 22:05:20,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1679000.0, ans=0.125
2023-11-21 22:05:29,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1679066.6666666667, ans=0.0
2023-11-21 22:05:40,483 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11400, loss[loss=0.05826, simple_loss=0.06139, pruned_loss=0.01426, audio_tagging_loss=0.01331, over 15016.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09399, pruned_loss=0.01599, audio_tagging_loss=0.009512, over 3042143.07 frames. ], batch size: 57, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:05:42,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1679133.3333333333, ans=0.1
2023-11-21 22:05:49,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.65 vs. limit=15.0
2023-11-21 22:06:02,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1679200.0, ans=0.0
2023-11-21 22:06:15,744 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251900
2023-11-21 22:06:20,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1679333.3333333333, ans=0.125
2023-11-21 22:06:27,612 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.296e+01 8.125e+01 8.630e+01 9.501e+01 1.337e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 22:06:28,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1679333.3333333333, ans=0.125
2023-11-21 22:06:40,873 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:06:43,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1679466.6666666667, ans=0.2
2023-11-21 22:06:44,322 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11450, loss[loss=0.08887, simple_loss=0.1076, pruned_loss=0.02695, audio_tagging_loss=0.008099, over 14098.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09478, pruned_loss=0.01609, audio_tagging_loss=0.00952, over 3050695.73 frames. ], batch size: 54, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:06:57,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1679533.3333333333, ans=0.0
2023-11-21 22:07:03,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1679533.3333333333, ans=0.125
2023-11-21 22:07:05,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1679533.3333333333, ans=0.95
2023-11-21 22:07:09,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1679600.0, ans=0.125
2023-11-21 22:07:21,330 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 251950
2023-11-21 22:07:22,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1679666.6666666667, ans=0.125
2023-11-21 22:07:26,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1679666.6666666667, ans=0.95
2023-11-21 22:07:29,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1679666.6666666667, ans=0.1
2023-11-21 22:07:48,096 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=7.76 vs. limit=12.0
2023-11-21 22:07:48,703 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11500, loss[loss=0.06056, simple_loss=0.07961, pruned_loss=0.01241, audio_tagging_loss=0.00834, over 16735.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.09464, pruned_loss=0.01594, audio_tagging_loss=0.009393, over 3047708.76 frames. ], batch size: 65, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:07:59,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1679866.6666666667, ans=0.125
2023-11-21 22:08:20,084 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-21 22:08:25,015 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252000
2023-11-21 22:08:34,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1680000.0, ans=0.1
2023-11-21 22:08:39,021 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.783e+01 7.996e+01 8.554e+01 9.237e+01 1.174e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 22:08:56,376 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11550, loss[loss=0.06746, simple_loss=0.08965, pruned_loss=0.01465, audio_tagging_loss=0.00798, over 15508.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09475, pruned_loss=0.0159, audio_tagging_loss=0.009331, over 3048124.44 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:08:57,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1680133.3333333333, ans=0.025
2023-11-21 22:09:08,755 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.71 vs. limit=22.5
2023-11-21 22:09:09,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1680200.0, ans=0.125
2023-11-21 22:09:13,639 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.82 vs. limit=10.0
2023-11-21 22:09:20,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1680200.0, ans=0.0
2023-11-21 22:09:24,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1680266.6666666667, ans=0.125
2023-11-21 22:09:31,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252050
2023-11-21 22:09:34,289 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:09:49,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.16 vs. limit=10.0
2023-11-21 22:09:59,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1680466.6666666667, ans=0.125
2023-11-21 22:10:00,341 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11600, loss[loss=0.07605, simple_loss=0.0992, pruned_loss=0.01921, audio_tagging_loss=0.007234, over 14617.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09602, pruned_loss=0.01609, audio_tagging_loss=0.0093, over 3049471.21 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:10:18,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1680533.3333333333, ans=0.125
2023-11-21 22:10:29,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1680600.0, ans=0.0
2023-11-21 22:10:36,282 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252100
2023-11-21 22:10:47,824 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.780e+01 8.258e+01 8.947e+01 9.707e+01 1.482e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 22:10:48,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1680666.6666666667, ans=0.125
2023-11-21 22:11:04,539 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11650, loss[loss=0.07825, simple_loss=0.1041, pruned_loss=0.0203, audio_tagging_loss=0.005901, over 14940.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09591, pruned_loss=0.01607, audio_tagging_loss=0.009322, over 3051960.26 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:11:18,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1680866.6666666667, ans=0.125
2023-11-21 22:11:18,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.18 vs. limit=22.5
2023-11-21 22:11:25,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1680866.6666666667, ans=0.125
2023-11-21 22:11:40,805 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252150
2023-11-21 22:11:54,261 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.93 vs. limit=12.0
2023-11-21 22:12:02,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1681066.6666666667, ans=0.125
2023-11-21 22:12:02,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1681066.6666666667, ans=0.1
2023-11-21 22:12:08,170 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11700, loss[loss=0.08425, simple_loss=0.108, pruned_loss=0.02184, audio_tagging_loss=0.008423, over 16207.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09613, pruned_loss=0.016, audio_tagging_loss=0.009305, over 3051244.16 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:12:39,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1681266.6666666667, ans=0.125
2023-11-21 22:12:41,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1681266.6666666667, ans=0.1
2023-11-21 22:12:44,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1681266.6666666667, ans=0.0
2023-11-21 22:12:45,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252200
2023-11-21 22:12:56,195 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.227e+01 8.797e+01 9.747e+01 1.315e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 22:13:06,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1681400.0, ans=0.04949747468305833
2023-11-21 22:13:14,117 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11750, loss[loss=0.08785, simple_loss=0.1166, pruned_loss=0.02215, audio_tagging_loss=0.007399, over 15870.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09493, pruned_loss=0.01573, audio_tagging_loss=0.009489, over 3037962.86 frames. ], batch size: 57, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:13:19,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1681466.6666666667, ans=0.125
2023-11-21 22:13:21,700 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.14 vs. limit=15.0
2023-11-21 22:13:23,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1681466.6666666667, ans=0.5
2023-11-21 22:13:29,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1681533.3333333333, ans=0.1
2023-11-21 22:13:32,545 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.86 vs. limit=15.0
2023-11-21 22:13:45,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1681600.0, ans=0.0
2023-11-21 22:13:50,099 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252250
2023-11-21 22:14:08,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1681733.3333333333, ans=0.0
2023-11-21 22:14:19,021 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11800, loss[loss=0.1038, simple_loss=0.1449, pruned_loss=0.02439, audio_tagging_loss=0.006943, over 17009.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09492, pruned_loss=0.01581, audio_tagging_loss=0.009491, over 3037689.06 frames. ], batch size: 61, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:14:24,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.99 vs. limit=12.0
2023-11-21 22:14:25,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1681800.0, ans=0.0
2023-11-21 22:14:25,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1681800.0, ans=0.125
2023-11-21 22:14:34,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1681866.6666666667, ans=0.1
2023-11-21 22:14:42,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1681866.6666666667, ans=0.1
2023-11-21 22:14:49,951 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=15.0
2023-11-21 22:14:54,850 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252300
2023-11-21 22:15:00,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1682000.0, ans=0.125
2023-11-21 22:15:07,670 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 7.961e+01 8.555e+01 9.322e+01 1.275e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 22:15:07,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1682000.0, ans=0.0
2023-11-21 22:15:15,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1682066.6666666667, ans=0.125
2023-11-21 22:15:22,863 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11850, loss[loss=0.07315, simple_loss=0.09439, pruned_loss=0.01585, audio_tagging_loss=0.0101, over 16134.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09467, pruned_loss=0.01592, audio_tagging_loss=0.009607, over 3043984.03 frames. ], batch size: 61, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:15:39,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1682200.0, ans=0.125
2023-11-21 22:15:42,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1682200.0, ans=0.125
2023-11-21 22:15:57,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1682266.6666666667, ans=0.125
2023-11-21 22:16:00,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252350
2023-11-21 22:16:00,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1682266.6666666667, ans=0.0
2023-11-21 22:16:05,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1682333.3333333333, ans=0.125
2023-11-21 22:16:07,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1682333.3333333333, ans=0.09899494936611666
2023-11-21 22:16:14,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.43 vs. limit=12.0
2023-11-21 22:16:27,836 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11900, loss[loss=0.06598, simple_loss=0.08753, pruned_loss=0.01133, audio_tagging_loss=0.01088, over 15404.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09378, pruned_loss=0.0157, audio_tagging_loss=0.009702, over 3043359.31 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:16:46,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1682533.3333333333, ans=0.125
2023-11-21 22:16:47,024 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:16:48,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.09 vs. limit=6.0
2023-11-21 22:17:04,121 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252400
2023-11-21 22:17:17,491 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.240e+01 7.885e+01 8.457e+01 9.193e+01 1.226e+02, threshold=1.691e+02, percent-clipped=0.0
2023-11-21 22:17:33,580 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 11950, loss[loss=0.06217, simple_loss=0.0793, pruned_loss=0.01229, audio_tagging_loss=0.01022, over 15746.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09451, pruned_loss=0.01597, audio_tagging_loss=0.009724, over 3049789.05 frames. ], batch size: 60, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:17:40,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1682800.0, ans=0.1
2023-11-21 22:17:45,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1682866.6666666667, ans=0.1
2023-11-21 22:18:09,287 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252450
2023-11-21 22:18:16,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1683000.0, ans=0.125
2023-11-21 22:18:17,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1683000.0, ans=0.125
2023-11-21 22:18:33,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1683066.6666666667, ans=0.0
2023-11-21 22:18:36,447 INFO [train_asr.py:1221] (2/4) Epoch 21, batch 12000, loss[loss=0.08011, simple_loss=0.1093, pruned_loss=0.01668, audio_tagging_loss=0.008791, over 14242.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09526, pruned_loss=0.016, audio_tagging_loss=0.009722, over 3040991.26 frames. ], batch size: 55, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:18:36,448 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 22:19:06,984 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4397, 3.6785, 2.5753, 3.7556], device='cuda:2')
2023-11-21 22:19:19,213 INFO [train_asr.py:1253] (2/4) Epoch 21, validation: loss=0.05938, simple_loss=0.05195, pruned_loss=0.005201, audio_tagging_loss=0.02821, over 4681554.00 frames. 
2023-11-21 22:19:19,214 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 22:19:22,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.71 vs. limit=22.5
2023-11-21 22:19:23,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1683133.3333333333, ans=0.0
2023-11-21 22:19:41,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1683200.0, ans=0.125
2023-11-21 22:20:22,679 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 0, loss[loss=0.08811, simple_loss=0.09124, pruned_loss=0.01679, audio_tagging_loss=0.02571, over 16261.00 frames. ], tot_loss[loss=0.08811, simple_loss=0.09124, pruned_loss=0.01679, audio_tagging_loss=0.02571, over 16261.00 frames. ], batch size: 62, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:20:22,679 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 22:20:59,039 INFO [train_asr.py:1253] (2/4) Epoch 22, validation: loss=0.05904, simple_loss=0.0519, pruned_loss=0.0051, audio_tagging_loss=0.02799, over 4681554.00 frames. 
2023-11-21 22:20:59,040 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 22:21:03,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252500
2023-11-21 22:21:10,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=1683360.0, ans=0.1
2023-11-21 22:21:16,309 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.182e+01 9.159e+01 9.846e+01 1.292e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-21 22:21:40,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1683493.3333333333, ans=0.125
2023-11-21 22:21:58,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1683560.0, ans=0.0
2023-11-21 22:22:02,755 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 50, loss[loss=0.08127, simple_loss=0.0911, pruned_loss=0.01742, audio_tagging_loss=0.0183, over 15940.00 frames. ], tot_loss[loss=0.08311, simple_loss=0.09642, pruned_loss=0.01652, audio_tagging_loss=0.01838, over 695181.82 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:22:07,694 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252550
2023-11-21 22:22:15,367 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.37 vs. limit=12.0
2023-11-21 22:22:19,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1683693.3333333333, ans=0.2
2023-11-21 22:22:30,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1683760.0, ans=0.0
2023-11-21 22:23:06,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1683960.0, ans=0.0
2023-11-21 22:23:07,476 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 100, loss[loss=0.08997, simple_loss=0.1177, pruned_loss=0.01618, audio_tagging_loss=0.01496, over 15136.00 frames. ], tot_loss[loss=0.08165, simple_loss=0.09564, pruned_loss=0.01619, audio_tagging_loss=0.01764, over 1216491.30 frames. ], batch size: 53, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:23:08,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1683960.0, ans=0.125
2023-11-21 22:23:10,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=15.45 vs. limit=15.0
2023-11-21 22:23:12,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252600
2023-11-21 22:23:16,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=1683960.0, ans=0.02
2023-11-21 22:23:23,301 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:23:25,407 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.481e+01 8.773e+01 9.405e+01 1.016e+02 1.413e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-21 22:23:25,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1684026.6666666667, ans=0.0
2023-11-21 22:23:42,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1684093.3333333333, ans=0.125
2023-11-21 22:23:43,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1684093.3333333333, ans=0.125
2023-11-21 22:23:49,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1684160.0, ans=0.125
2023-11-21 22:23:49,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1684160.0, ans=0.0
2023-11-21 22:23:50,174 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.90 vs. limit=15.0
2023-11-21 22:24:11,245 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 150, loss[loss=0.06766, simple_loss=0.0928, pruned_loss=0.01257, audio_tagging_loss=0.008688, over 13719.00 frames. ], tot_loss[loss=0.07977, simple_loss=0.09609, pruned_loss=0.01596, audio_tagging_loss=0.01577, over 1622909.55 frames. ], batch size: 52, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:24:16,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252650
2023-11-21 22:25:15,846 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 200, loss[loss=0.08701, simple_loss=0.1141, pruned_loss=0.01927, audio_tagging_loss=0.01071, over 14469.00 frames. ], tot_loss[loss=0.07735, simple_loss=0.09534, pruned_loss=0.01567, audio_tagging_loss=0.01401, over 1934403.85 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:25:17,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1684626.6666666667, ans=0.0
2023-11-21 22:25:20,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252700
2023-11-21 22:25:24,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1684626.6666666667, ans=0.2
2023-11-21 22:25:24,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1684626.6666666667, ans=0.125
2023-11-21 22:25:34,378 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.318e+01 8.693e+01 9.716e+01 1.201e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 22:25:36,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1684693.3333333333, ans=0.125
2023-11-21 22:25:50,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1684760.0, ans=0.0
2023-11-21 22:25:57,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1684826.6666666667, ans=0.125
2023-11-21 22:26:16,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1684893.3333333333, ans=0.04949747468305833
2023-11-21 22:26:19,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1684960.0, ans=0.2
2023-11-21 22:26:21,126 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 250, loss[loss=0.08044, simple_loss=0.1057, pruned_loss=0.01957, audio_tagging_loss=0.008047, over 16726.00 frames. ], tot_loss[loss=0.07689, simple_loss=0.09642, pruned_loss=0.01601, audio_tagging_loss=0.01267, over 2185284.36 frames. ], batch size: 62, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:26:26,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252750
2023-11-21 22:26:33,735 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.28 vs. limit=15.0
2023-11-21 22:26:58,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1685160.0, ans=0.0
2023-11-21 22:27:12,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1685226.6666666667, ans=0.125
2023-11-21 22:27:15,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1685226.6666666667, ans=0.125
2023-11-21 22:27:23,269 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.65 vs. limit=15.0
2023-11-21 22:27:24,797 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 300, loss[loss=0.07334, simple_loss=0.0928, pruned_loss=0.01554, audio_tagging_loss=0.0114, over 15185.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09472, pruned_loss=0.01567, audio_tagging_loss=0.01183, over 2377974.26 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:27:30,635 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252800
2023-11-21 22:27:36,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1685293.3333333333, ans=0.125
2023-11-21 22:27:43,812 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.171e+01 8.868e+01 9.880e+01 1.210e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 22:27:49,065 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:27:59,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1685426.6666666667, ans=0.2
2023-11-21 22:28:05,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1685493.3333333333, ans=0.125
2023-11-21 22:28:09,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1685493.3333333333, ans=0.0
2023-11-21 22:28:30,583 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 350, loss[loss=0.06891, simple_loss=0.08604, pruned_loss=0.01534, audio_tagging_loss=0.01055, over 15176.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09658, pruned_loss=0.01612, audio_tagging_loss=0.01102, over 2523114.39 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:28:36,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252850
2023-11-21 22:28:36,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1685626.6666666667, ans=0.125
2023-11-21 22:28:43,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1685693.3333333333, ans=0.0
2023-11-21 22:28:50,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1685693.3333333333, ans=0.2
2023-11-21 22:28:54,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1685693.3333333333, ans=0.125
2023-11-21 22:29:03,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.62 vs. limit=15.0
2023-11-21 22:29:03,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1685760.0, ans=0.0
2023-11-21 22:29:12,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1685826.6666666667, ans=0.2
2023-11-21 22:29:14,108 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.10 vs. limit=15.0
2023-11-21 22:29:22,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.39 vs. limit=15.0
2023-11-21 22:29:28,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1685893.3333333333, ans=0.1
2023-11-21 22:29:35,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1685960.0, ans=0.0
2023-11-21 22:29:36,493 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 400, loss[loss=0.0795, simple_loss=0.105, pruned_loss=0.01877, audio_tagging_loss=0.008208, over 15518.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09572, pruned_loss=0.01606, audio_tagging_loss=0.01057, over 2644208.19 frames. ], batch size: 59, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:29:42,690 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252900
2023-11-21 22:29:45,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1685960.0, ans=0.1
2023-11-21 22:29:49,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1686026.6666666667, ans=0.0
2023-11-21 22:29:55,049 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.919e+01 8.135e+01 8.839e+01 9.372e+01 1.171e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 22:29:59,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.31 vs. limit=22.5
2023-11-21 22:29:59,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1686026.6666666667, ans=0.125
2023-11-21 22:30:10,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1686093.3333333333, ans=0.07
2023-11-21 22:30:14,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1686160.0, ans=0.125
2023-11-21 22:30:39,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1686226.6666666667, ans=0.2
2023-11-21 22:30:40,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1686226.6666666667, ans=0.0
2023-11-21 22:30:40,707 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.20 vs. limit=6.0
2023-11-21 22:30:42,526 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 450, loss[loss=0.06986, simple_loss=0.08309, pruned_loss=0.01537, audio_tagging_loss=0.01295, over 15559.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09649, pruned_loss=0.01626, audio_tagging_loss=0.01026, over 2727072.83 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:30:48,365 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 252950
2023-11-21 22:31:23,296 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.32 vs. limit=15.0
2023-11-21 22:31:25,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1686493.3333333333, ans=0.125
2023-11-21 22:31:30,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1686493.3333333333, ans=0.0
2023-11-21 22:31:31,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1686493.3333333333, ans=0.125
2023-11-21 22:31:34,810 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0
2023-11-21 22:31:48,764 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 500, loss[loss=0.06346, simple_loss=0.0732, pruned_loss=0.01413, audio_tagging_loss=0.01273, over 15571.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09648, pruned_loss=0.01609, audio_tagging_loss=0.01003, over 2808265.44 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:31:51,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1686626.6666666667, ans=0.125
2023-11-21 22:31:53,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253000
2023-11-21 22:31:59,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1686626.6666666667, ans=0.2
2023-11-21 22:32:01,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1686693.3333333333, ans=0.125
2023-11-21 22:32:06,966 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.271e+01 8.753e+01 9.475e+01 1.190e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 22:32:13,962 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.90 vs. limit=22.5
2023-11-21 22:32:43,998 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:32:54,297 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 550, loss[loss=0.0944, simple_loss=0.1225, pruned_loss=0.02396, audio_tagging_loss=0.009177, over 15986.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09527, pruned_loss=0.016, audio_tagging_loss=0.009914, over 2856806.59 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:33:00,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253050
2023-11-21 22:33:07,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1687026.6666666667, ans=0.0
2023-11-21 22:34:00,172 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 600, loss[loss=0.08205, simple_loss=0.1087, pruned_loss=0.01952, audio_tagging_loss=0.008195, over 15502.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09523, pruned_loss=0.01608, audio_tagging_loss=0.009786, over 2903297.47 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:34:05,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253100
2023-11-21 22:34:18,154 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.004e+01 8.758e+01 9.434e+01 1.366e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 22:34:24,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1687426.6666666667, ans=0.95
2023-11-21 22:34:41,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1687493.3333333333, ans=0.125
2023-11-21 22:35:05,435 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 650, loss[loss=0.05707, simple_loss=0.06607, pruned_loss=0.01104, audio_tagging_loss=0.01299, over 15363.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09487, pruned_loss=0.01616, audio_tagging_loss=0.009824, over 2927464.46 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:35:10,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253150
2023-11-21 22:35:11,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1687626.6666666667, ans=0.125
2023-11-21 22:35:16,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1687693.3333333333, ans=0.125
2023-11-21 22:35:44,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1687826.6666666667, ans=0.125
2023-11-21 22:35:56,307 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.87 vs. limit=15.0
2023-11-21 22:35:57,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1687893.3333333333, ans=0.125
2023-11-21 22:36:00,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1687893.3333333333, ans=0.125
2023-11-21 22:36:05,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1687893.3333333333, ans=0.5
2023-11-21 22:36:09,019 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 700, loss[loss=0.05293, simple_loss=0.07208, pruned_loss=0.007827, audio_tagging_loss=0.009063, over 14886.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09544, pruned_loss=0.01611, audio_tagging_loss=0.009735, over 2953009.21 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:36:13,976 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253200
2023-11-21 22:36:24,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1688026.6666666667, ans=0.125
2023-11-21 22:36:28,394 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.099e+01 8.868e+01 9.467e+01 1.457e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 22:36:35,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1688093.3333333333, ans=0.0
2023-11-21 22:36:41,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1688093.3333333333, ans=0.125
2023-11-21 22:37:13,502 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 750, loss[loss=0.08104, simple_loss=0.1112, pruned_loss=0.01748, audio_tagging_loss=0.007972, over 15987.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09532, pruned_loss=0.01618, audio_tagging_loss=0.009706, over 2972072.87 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:37:16,584 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.99 vs. limit=22.5
2023-11-21 22:37:18,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253250
2023-11-21 22:37:23,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1688293.3333333333, ans=0.125
2023-11-21 22:37:33,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1688360.0, ans=0.1
2023-11-21 22:37:42,680 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:37:44,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.98 vs. limit=12.0
2023-11-21 22:37:46,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1688426.6666666667, ans=0.125
2023-11-21 22:37:55,626 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:38:02,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1688493.3333333333, ans=0.0
2023-11-21 22:38:11,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1688560.0, ans=0.125
2023-11-21 22:38:17,557 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 800, loss[loss=0.07985, simple_loss=0.1037, pruned_loss=0.01673, audio_tagging_loss=0.01124, over 14685.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09533, pruned_loss=0.01607, audio_tagging_loss=0.009737, over 2984182.96 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:38:19,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1688626.6666666667, ans=0.125
2023-11-21 22:38:20,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1688626.6666666667, ans=0.125
2023-11-21 22:38:22,647 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253300
2023-11-21 22:38:36,175 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.171e+01 8.697e+01 9.274e+01 1.215e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 22:38:41,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1688760.0, ans=0.125
2023-11-21 22:39:20,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1688960.0, ans=0.125
2023-11-21 22:39:20,957 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 850, loss[loss=0.08233, simple_loss=0.1059, pruned_loss=0.01878, audio_tagging_loss=0.01059, over 16277.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09503, pruned_loss=0.0161, audio_tagging_loss=0.009728, over 3005809.28 frames. ], batch size: 59, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:39:26,001 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253350
2023-11-21 22:40:01,673 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.05 vs. limit=15.0
2023-11-21 22:40:05,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1689160.0, ans=0.125
2023-11-21 22:40:10,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1689226.6666666667, ans=0.125
2023-11-21 22:40:24,931 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 900, loss[loss=0.05081, simple_loss=0.05974, pruned_loss=0.007509, audio_tagging_loss=0.01343, over 15089.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.0946, pruned_loss=0.01599, audio_tagging_loss=0.009939, over 3015094.59 frames. ], batch size: 59, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:40:30,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253400
2023-11-21 22:40:44,864 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.175e+01 9.057e+01 9.659e+01 1.593e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-21 22:40:57,991 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.62 vs. limit=10.0
2023-11-21 22:41:29,245 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:41:30,292 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 950, loss[loss=0.07679, simple_loss=0.1037, pruned_loss=0.01552, audio_tagging_loss=0.009397, over 14497.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09551, pruned_loss=0.01594, audio_tagging_loss=0.009766, over 3021548.82 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:41:35,388 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253450
2023-11-21 22:41:46,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1689693.3333333333, ans=0.035
2023-11-21 22:41:47,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1689693.3333333333, ans=0.0
2023-11-21 22:42:07,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.18 vs. limit=12.0
2023-11-21 22:42:17,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1689826.6666666667, ans=0.1
2023-11-21 22:42:22,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1689893.3333333333, ans=0.0
2023-11-21 22:42:29,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1689893.3333333333, ans=0.125
2023-11-21 22:42:32,964 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1000, loss[loss=0.0885, simple_loss=0.1103, pruned_loss=0.02516, audio_tagging_loss=0.008196, over 14712.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09483, pruned_loss=0.01598, audio_tagging_loss=0.009666, over 3026581.60 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:42:37,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253500
2023-11-21 22:42:42,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1689960.0, ans=0.125
2023-11-21 22:42:51,806 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.522e+01 8.090e+01 8.690e+01 9.714e+01 1.265e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 22:42:53,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1690026.6666666667, ans=0.125
2023-11-21 22:42:55,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1690026.6666666667, ans=0.0
2023-11-21 22:42:59,988 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:43:03,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1690093.3333333333, ans=0.125
2023-11-21 22:43:37,566 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1050, loss[loss=0.0682, simple_loss=0.09924, pruned_loss=0.009331, audio_tagging_loss=0.009249, over 16153.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.094, pruned_loss=0.01569, audio_tagging_loss=0.009504, over 3028806.39 frames. ], batch size: 61, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:43:37,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1690293.3333333333, ans=0.125
2023-11-21 22:43:40,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1690293.3333333333, ans=0.125
2023-11-21 22:43:42,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253550
2023-11-21 22:43:42,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1690293.3333333333, ans=0.1
2023-11-21 22:43:55,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1690360.0, ans=0.0
2023-11-21 22:43:58,525 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.98 vs. limit=22.5
2023-11-21 22:44:08,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1690426.6666666667, ans=0.125
2023-11-21 22:44:12,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1690426.6666666667, ans=0.125
2023-11-21 22:44:12,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=1690426.6666666667, ans=0.02
2023-11-21 22:44:12,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1690426.6666666667, ans=0.125
2023-11-21 22:44:17,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1690493.3333333333, ans=0.125
2023-11-21 22:44:38,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1690560.0, ans=0.125
2023-11-21 22:44:42,865 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1100, loss[loss=0.06755, simple_loss=0.08817, pruned_loss=0.01129, audio_tagging_loss=0.01217, over 15057.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09437, pruned_loss=0.01573, audio_tagging_loss=0.009424, over 3028501.64 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:44:45,369 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:44:45,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1690626.6666666667, ans=0.125
2023-11-21 22:44:47,907 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253600
2023-11-21 22:45:01,572 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.847e+01 8.128e+01 8.851e+01 9.302e+01 1.185e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 22:45:16,807 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.45 vs. limit=15.0
2023-11-21 22:45:22,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1690826.6666666667, ans=0.09899494936611666
2023-11-21 22:45:37,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1690893.3333333333, ans=0.1
2023-11-21 22:45:42,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1690893.3333333333, ans=0.1
2023-11-21 22:45:46,080 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1150, loss[loss=0.06242, simple_loss=0.07221, pruned_loss=0.01291, audio_tagging_loss=0.01341, over 15067.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09534, pruned_loss=0.01592, audio_tagging_loss=0.009331, over 3029513.36 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 8.0
2023-11-21 22:45:48,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1690960.0, ans=0.1
2023-11-21 22:45:49,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1690960.0, ans=0.0
2023-11-21 22:45:51,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253650
2023-11-21 22:46:06,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1691026.6666666667, ans=0.125
2023-11-21 22:46:07,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1691026.6666666667, ans=0.2
2023-11-21 22:46:23,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1691160.0, ans=0.035
2023-11-21 22:46:28,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1691160.0, ans=0.2
2023-11-21 22:46:32,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=1691160.0, ans=0.2
2023-11-21 22:46:50,830 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1200, loss[loss=0.06024, simple_loss=0.07848, pruned_loss=0.01006, audio_tagging_loss=0.01094, over 15526.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.096, pruned_loss=0.01607, audio_tagging_loss=0.009278, over 3039411.27 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:46:55,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253700
2023-11-21 22:47:03,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1691360.0, ans=0.1
2023-11-21 22:47:06,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1691360.0, ans=0.1
2023-11-21 22:47:09,371 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.50 vs. limit=22.5
2023-11-21 22:47:12,906 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.597e+01 8.044e+01 8.657e+01 9.227e+01 1.115e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 22:47:18,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1691426.6666666667, ans=0.125
2023-11-21 22:47:50,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1691560.0, ans=22.5
2023-11-21 22:47:54,564 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1250, loss[loss=0.05824, simple_loss=0.07119, pruned_loss=0.0132, audio_tagging_loss=0.009447, over 14461.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09563, pruned_loss=0.01594, audio_tagging_loss=0.009323, over 3042300.26 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:48:00,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253750
2023-11-21 22:48:25,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1691760.0, ans=0.125
2023-11-21 22:48:42,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1691826.6666666667, ans=0.125
2023-11-21 22:48:45,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1691893.3333333333, ans=0.125
2023-11-21 22:48:59,698 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1300, loss[loss=0.1214, simple_loss=0.1651, pruned_loss=0.03182, audio_tagging_loss=0.006999, over 15096.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09589, pruned_loss=0.01591, audio_tagging_loss=0.009325, over 3039711.49 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:49:01,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1691960.0, ans=0.0
2023-11-21 22:49:04,735 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253800
2023-11-21 22:49:16,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1692026.6666666667, ans=0.125
2023-11-21 22:49:22,319 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 7.957e+01 8.431e+01 9.281e+01 1.105e+02, threshold=1.686e+02, percent-clipped=0.0
2023-11-21 22:49:22,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1692026.6666666667, ans=0.1
2023-11-21 22:49:22,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1692026.6666666667, ans=0.125
2023-11-21 22:49:31,568 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.71 vs. limit=15.0
2023-11-21 22:49:35,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1692093.3333333333, ans=0.0
2023-11-21 22:50:04,311 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1350, loss[loss=0.07215, simple_loss=0.08718, pruned_loss=0.01854, audio_tagging_loss=0.01001, over 16939.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.09547, pruned_loss=0.01597, audio_tagging_loss=0.009306, over 3046165.53 frames. ], batch size: 63, lr: 3.14e-03, grad_scale: 8.0
2023-11-21 22:50:10,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253850
2023-11-21 22:50:23,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1692360.0, ans=0.125
2023-11-21 22:50:31,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1692426.6666666667, ans=0.125
2023-11-21 22:50:51,078 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:51:09,539 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1400, loss[loss=0.08436, simple_loss=0.1124, pruned_loss=0.01925, audio_tagging_loss=0.008889, over 15832.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09535, pruned_loss=0.01599, audio_tagging_loss=0.00942, over 3049121.58 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 8.0
2023-11-21 22:51:15,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253900
2023-11-21 22:51:32,899 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.193e+01 8.723e+01 9.483e+01 1.296e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 22:52:14,996 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1450, loss[loss=0.06938, simple_loss=0.08156, pruned_loss=0.01821, audio_tagging_loss=0.01039, over 15010.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09544, pruned_loss=0.01599, audio_tagging_loss=0.009545, over 3050850.24 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 8.0
2023-11-21 22:52:19,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 253950
2023-11-21 22:52:22,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1692960.0, ans=0.025
2023-11-21 22:52:35,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1693026.6666666667, ans=0.0
2023-11-21 22:52:58,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1693160.0, ans=0.0
2023-11-21 22:53:01,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1693160.0, ans=0.0
2023-11-21 22:53:08,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1693226.6666666667, ans=0.0
2023-11-21 22:53:12,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.59 vs. limit=6.0
2023-11-21 22:53:16,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1693226.6666666667, ans=0.0
2023-11-21 22:53:19,087 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1500, loss[loss=0.05806, simple_loss=0.07177, pruned_loss=0.01226, audio_tagging_loss=0.009923, over 15139.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09504, pruned_loss=0.01604, audio_tagging_loss=0.009668, over 3041274.29 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 8.0
2023-11-21 22:53:24,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254000
2023-11-21 22:53:40,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1693360.0, ans=0.1
2023-11-21 22:53:40,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1693360.0, ans=0.125
2023-11-21 22:53:40,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1693360.0, ans=0.0
2023-11-21 22:53:42,809 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.377e+01 8.863e+01 9.600e+01 1.162e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 22:54:23,596 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1550, loss[loss=0.08819, simple_loss=0.1189, pruned_loss=0.02143, audio_tagging_loss=0.007316, over 15648.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09633, pruned_loss=0.01618, audio_tagging_loss=0.009728, over 3045012.21 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 8.0
2023-11-21 22:54:26,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1693626.6666666667, ans=0.125
2023-11-21 22:54:28,524 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254050
2023-11-21 22:54:58,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1693760.0, ans=0.07
2023-11-21 22:55:05,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1693826.6666666667, ans=0.125
2023-11-21 22:55:08,005 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-21 22:55:15,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1693893.3333333333, ans=0.125
2023-11-21 22:55:16,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1693893.3333333333, ans=0.125
2023-11-21 22:55:18,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1693893.3333333333, ans=0.125
2023-11-21 22:55:22,146 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.07 vs. limit=10.0
2023-11-21 22:55:27,452 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1600, loss[loss=0.05291, simple_loss=0.05755, pruned_loss=0.009914, audio_tagging_loss=0.01421, over 14617.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09568, pruned_loss=0.01597, audio_tagging_loss=0.009823, over 3039602.64 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:55:32,292 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254100
2023-11-21 22:55:38,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1693960.0, ans=0.2
2023-11-21 22:55:39,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten.whitening_limit, batch_count=1694026.6666666667, ans=15.0
2023-11-21 22:55:43,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1694026.6666666667, ans=0.125
2023-11-21 22:55:50,494 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.195e+01 8.712e+01 9.406e+01 1.124e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 22:56:20,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.94 vs. limit=15.0
2023-11-21 22:56:31,736 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1650, loss[loss=0.06776, simple_loss=0.08207, pruned_loss=0.01533, audio_tagging_loss=0.01139, over 14626.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09541, pruned_loss=0.01612, audio_tagging_loss=0.009863, over 3041274.36 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:56:36,631 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254150
2023-11-21 22:56:49,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.15 vs. limit=22.5
2023-11-21 22:56:56,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1694426.6666666667, ans=0.1
2023-11-21 22:57:03,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1694426.6666666667, ans=0.2
2023-11-21 22:57:14,449 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.23 vs. limit=22.5
2023-11-21 22:57:28,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1694560.0, ans=0.125
2023-11-21 22:57:36,322 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1700, loss[loss=0.05823, simple_loss=0.06705, pruned_loss=0.01331, audio_tagging_loss=0.01139, over 14800.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09477, pruned_loss=0.01591, audio_tagging_loss=0.009949, over 3046741.82 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:57:41,421 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254200
2023-11-21 22:57:46,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1694626.6666666667, ans=0.125
2023-11-21 22:57:59,282 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.322e+01 8.804e+01 9.363e+01 1.175e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 22:57:59,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1694693.3333333333, ans=0.09899494936611666
2023-11-21 22:58:33,781 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.47 vs. limit=12.0
2023-11-21 22:58:41,007 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1750, loss[loss=0.07425, simple_loss=0.08675, pruned_loss=0.01521, audio_tagging_loss=0.01566, over 16358.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09466, pruned_loss=0.01581, audio_tagging_loss=0.009845, over 3049362.58 frames. ], batch size: 61, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:58:45,942 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254250
2023-11-21 22:58:56,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1695026.6666666667, ans=0.125
2023-11-21 22:59:17,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=12.0
2023-11-21 22:59:44,767 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1800, loss[loss=0.0817, simple_loss=0.1164, pruned_loss=0.01653, audio_tagging_loss=0.00695, over 15348.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09571, pruned_loss=0.016, audio_tagging_loss=0.009744, over 3049331.41 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:59:50,303 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254300
2023-11-21 22:59:54,325 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.42 vs. limit=15.0
2023-11-21 23:00:08,559 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.157e+01 8.741e+01 9.334e+01 1.416e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 23:00:16,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.59 vs. limit=15.0
2023-11-21 23:00:37,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1695560.0, ans=0.125
2023-11-21 23:00:47,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1695560.0, ans=0.0
2023-11-21 23:00:49,760 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1850, loss[loss=0.07726, simple_loss=0.09874, pruned_loss=0.01894, audio_tagging_loss=0.008946, over 15878.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09594, pruned_loss=0.01603, audio_tagging_loss=0.009649, over 3052400.25 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:00:54,767 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254350
2023-11-21 23:00:57,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1695626.6666666667, ans=0.2
2023-11-21 23:01:09,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1695693.3333333333, ans=0.2
2023-11-21 23:01:26,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1695826.6666666667, ans=0.0
2023-11-21 23:01:43,387 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.45 vs. limit=15.0
2023-11-21 23:01:54,044 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1900, loss[loss=0.05923, simple_loss=0.08112, pruned_loss=0.01133, audio_tagging_loss=0.00734, over 13993.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09404, pruned_loss=0.01573, audio_tagging_loss=0.009621, over 3043774.55 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:01:58,522 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-21 23:01:59,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254400
2023-11-21 23:02:16,779 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 8.079e+01 8.685e+01 9.740e+01 1.437e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 23:02:45,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1696226.6666666667, ans=0.0
2023-11-21 23:02:50,737 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.21 vs. limit=15.0
2023-11-21 23:02:51,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1696226.6666666667, ans=0.125
2023-11-21 23:02:58,598 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 1950, loss[loss=0.08967, simple_loss=0.1135, pruned_loss=0.02623, audio_tagging_loss=0.006716, over 15064.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09392, pruned_loss=0.01581, audio_tagging_loss=0.009488, over 3043228.86 frames. ], batch size: 53, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:03:01,850 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.25 vs. limit=22.5
2023-11-21 23:03:03,552 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254450
2023-11-21 23:03:43,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1696493.3333333333, ans=0.1
2023-11-21 23:03:50,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1696560.0, ans=0.125
2023-11-21 23:03:53,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1696560.0, ans=0.125
2023-11-21 23:04:04,694 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2000, loss[loss=0.07215, simple_loss=0.09159, pruned_loss=0.01901, audio_tagging_loss=0.007347, over 17269.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.0941, pruned_loss=0.01593, audio_tagging_loss=0.009488, over 3045662.12 frames. ], batch size: 65, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:04:09,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254500
2023-11-21 23:04:12,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1696626.6666666667, ans=0.125
2023-11-21 23:04:26,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1696693.3333333333, ans=0.0
2023-11-21 23:04:27,259 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 7.983e+01 8.610e+01 9.428e+01 1.340e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 23:04:28,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1696760.0, ans=0.2
2023-11-21 23:04:42,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1696826.6666666667, ans=0.1
2023-11-21 23:04:46,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1696826.6666666667, ans=0.125
2023-11-21 23:05:07,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1696960.0, ans=0.125
2023-11-21 23:05:09,032 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2050, loss[loss=0.07357, simple_loss=0.0966, pruned_loss=0.01583, audio_tagging_loss=0.009434, over 14977.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09374, pruned_loss=0.01592, audio_tagging_loss=0.009411, over 3038765.99 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:05:13,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1696960.0, ans=0.0
2023-11-21 23:05:14,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254550
2023-11-21 23:05:30,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1697026.6666666667, ans=0.125
2023-11-21 23:05:38,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1697093.3333333333, ans=0.0
2023-11-21 23:05:43,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0
2023-11-21 23:05:48,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1697160.0, ans=0.125
2023-11-21 23:05:49,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1697160.0, ans=0.1
2023-11-21 23:05:56,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1697160.0, ans=10.0
2023-11-21 23:06:13,825 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2100, loss[loss=0.06805, simple_loss=0.09243, pruned_loss=0.0133, audio_tagging_loss=0.008537, over 14927.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09407, pruned_loss=0.01579, audio_tagging_loss=0.009393, over 3042687.66 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:06:16,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1697293.3333333333, ans=0.0
2023-11-21 23:06:18,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254600
2023-11-21 23:06:21,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1697293.3333333333, ans=0.1
2023-11-21 23:06:32,803 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:06:37,855 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.098e+01 8.657e+01 9.385e+01 1.571e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 23:06:48,899 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.20 vs. limit=22.5
2023-11-21 23:06:59,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1697493.3333333333, ans=0.125
2023-11-21 23:07:04,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1697560.0, ans=0.125
2023-11-21 23:07:04,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1697560.0, ans=0.1
2023-11-21 23:07:10,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1697560.0, ans=0.125
2023-11-21 23:07:18,037 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2150, loss[loss=0.06882, simple_loss=0.09184, pruned_loss=0.01412, audio_tagging_loss=0.008779, over 15253.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09481, pruned_loss=0.01596, audio_tagging_loss=0.009368, over 3044034.44 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:07:23,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254650
2023-11-21 23:07:24,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1697626.6666666667, ans=0.125
2023-11-21 23:07:28,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1697626.6666666667, ans=0.0
2023-11-21 23:07:52,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1697760.0, ans=0.125
2023-11-21 23:07:56,835 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:07:59,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1697826.6666666667, ans=0.125
2023-11-21 23:08:07,673 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.67 vs. limit=15.0
2023-11-21 23:08:14,166 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:08:15,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1697893.3333333333, ans=0.2
2023-11-21 23:08:23,091 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2200, loss[loss=0.06995, simple_loss=0.08336, pruned_loss=0.01795, audio_tagging_loss=0.01031, over 15605.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09478, pruned_loss=0.01603, audio_tagging_loss=0.009438, over 3042597.62 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:08:28,723 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254700
2023-11-21 23:08:35,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1698026.6666666667, ans=0.0
2023-11-21 23:08:47,051 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.502e+01 8.027e+01 8.507e+01 9.488e+01 1.436e+02, threshold=1.701e+02, percent-clipped=0.0
2023-11-21 23:08:48,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1698093.3333333333, ans=0.125
2023-11-21 23:08:51,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1698093.3333333333, ans=0.2
2023-11-21 23:08:58,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1698093.3333333333, ans=0.125
2023-11-21 23:09:07,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1698160.0, ans=0.125
2023-11-21 23:09:16,138 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.26 vs. limit=15.0
2023-11-21 23:09:27,605 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2250, loss[loss=0.0709, simple_loss=0.09534, pruned_loss=0.01608, audio_tagging_loss=0.007151, over 14627.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.095, pruned_loss=0.01609, audio_tagging_loss=0.009376, over 3041643.63 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:09:32,672 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254750
2023-11-21 23:10:01,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1698426.6666666667, ans=0.0
2023-11-21 23:10:13,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1698493.3333333333, ans=0.125
2023-11-21 23:10:16,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1698493.3333333333, ans=0.125
2023-11-21 23:10:16,954 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.39 vs. limit=22.5
2023-11-21 23:10:20,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1698560.0, ans=10.0
2023-11-21 23:10:25,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.58 vs. limit=15.0
2023-11-21 23:10:27,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1698560.0, ans=0.125
2023-11-21 23:10:27,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1698560.0, ans=0.125
2023-11-21 23:10:27,669 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.60 vs. limit=12.0
2023-11-21 23:10:32,052 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2300, loss[loss=0.05504, simple_loss=0.07013, pruned_loss=0.01078, audio_tagging_loss=0.009185, over 14407.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09433, pruned_loss=0.01591, audio_tagging_loss=0.009524, over 3045691.24 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:10:38,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254800
2023-11-21 23:10:42,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1698626.6666666667, ans=0.125
2023-11-21 23:10:44,924 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.74 vs. limit=15.0
2023-11-21 23:10:55,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1698693.3333333333, ans=0.0
2023-11-21 23:10:57,741 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.213e+01 8.771e+01 9.442e+01 1.167e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 23:11:17,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1698826.6666666667, ans=0.1
2023-11-21 23:11:21,649 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:11:29,821 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:11:37,148 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2350, loss[loss=0.08036, simple_loss=0.1097, pruned_loss=0.01646, audio_tagging_loss=0.009056, over 14993.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09525, pruned_loss=0.0161, audio_tagging_loss=0.00961, over 3043835.49 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:11:43,421 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254850
2023-11-21 23:11:46,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1698960.0, ans=0.125
2023-11-21 23:11:48,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1698960.0, ans=0.125
2023-11-21 23:12:02,352 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.77 vs. limit=6.0
2023-11-21 23:12:25,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1699160.0, ans=0.125
2023-11-21 23:12:42,280 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2400, loss[loss=0.07053, simple_loss=0.08883, pruned_loss=0.01191, audio_tagging_loss=0.0142, over 14520.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09571, pruned_loss=0.01615, audio_tagging_loss=0.009564, over 3039081.01 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:12:47,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254900
2023-11-21 23:12:56,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1699360.0, ans=0.125
2023-11-21 23:13:02,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1699360.0, ans=0.1
2023-11-21 23:13:06,542 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 7.976e+01 8.691e+01 9.517e+01 1.439e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 23:13:29,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1699493.3333333333, ans=0.125
2023-11-21 23:13:31,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1699560.0, ans=0.1
2023-11-21 23:13:45,573 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2450, loss[loss=0.06937, simple_loss=0.08325, pruned_loss=0.017, audio_tagging_loss=0.01075, over 15962.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09564, pruned_loss=0.01616, audio_tagging_loss=0.009632, over 3044204.74 frames. ], batch size: 61, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:13:51,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 254950
2023-11-21 23:13:59,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1699693.3333333333, ans=0.125
2023-11-21 23:14:00,747 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.20 vs. limit=15.0
2023-11-21 23:14:03,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1699693.3333333333, ans=0.1
2023-11-21 23:14:14,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1699760.0, ans=0.125
2023-11-21 23:14:35,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1699893.3333333333, ans=0.0
2023-11-21 23:14:36,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1699893.3333333333, ans=0.0
2023-11-21 23:14:50,015 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2500, loss[loss=0.06848, simple_loss=0.09538, pruned_loss=0.01345, audio_tagging_loss=0.007339, over 16896.00 frames. ], tot_loss[loss=0.07311, simple_loss=0.09529, pruned_loss=0.0158, audio_tagging_loss=0.009662, over 3044770.54 frames. ], batch size: 65, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:14:54,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255000
2023-11-21 23:15:14,911 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.176e+01 8.833e+01 9.682e+01 1.336e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 23:15:40,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1700160.0, ans=0.125
2023-11-21 23:15:40,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.34 vs. limit=15.0
2023-11-21 23:15:55,691 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2550, loss[loss=0.07186, simple_loss=0.09707, pruned_loss=0.01691, audio_tagging_loss=0.006415, over 14867.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09505, pruned_loss=0.01585, audio_tagging_loss=0.009508, over 3040635.84 frames. ], batch size: 55, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:16:00,737 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255050
2023-11-21 23:16:12,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1700360.0, ans=0.125
2023-11-21 23:16:38,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1700493.3333333333, ans=0.125
2023-11-21 23:16:39,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1700493.3333333333, ans=0.1
2023-11-21 23:16:42,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1700493.3333333333, ans=0.2
2023-11-21 23:16:56,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1700560.0, ans=0.125
2023-11-21 23:16:58,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1700626.6666666667, ans=0.125
2023-11-21 23:17:00,032 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2600, loss[loss=0.06506, simple_loss=0.07948, pruned_loss=0.01231, audio_tagging_loss=0.01302, over 13927.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09479, pruned_loss=0.01582, audio_tagging_loss=0.009331, over 3036729.20 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:17:04,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1700626.6666666667, ans=0.125
2023-11-21 23:17:05,063 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255100
2023-11-21 23:17:18,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1700693.3333333333, ans=0.025
2023-11-21 23:17:24,817 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.035e+01 8.256e+01 8.734e+01 9.365e+01 1.832e+02, threshold=1.747e+02, percent-clipped=1.0
2023-11-21 23:17:49,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1700826.6666666667, ans=0.0
2023-11-21 23:17:55,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1700893.3333333333, ans=0.1
2023-11-21 23:17:57,633 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.656e-03
2023-11-21 23:17:58,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1700893.3333333333, ans=0.1
2023-11-21 23:18:01,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_na.min_abs, batch_count=1700893.3333333333, ans=0.02
2023-11-21 23:18:05,201 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2650, loss[loss=0.07264, simple_loss=0.09381, pruned_loss=0.01557, audio_tagging_loss=0.01017, over 16752.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09428, pruned_loss=0.01571, audio_tagging_loss=0.009252, over 3045124.23 frames. ], batch size: 63, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:18:08,433 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0
2023-11-21 23:18:10,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255150
2023-11-21 23:18:13,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1700960.0, ans=0.125
2023-11-21 23:18:34,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1701093.3333333333, ans=0.1
2023-11-21 23:18:41,435 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=15.03 vs. limit=15.0
2023-11-21 23:19:03,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1701226.6666666667, ans=0.2
2023-11-21 23:19:09,716 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2700, loss[loss=0.06744, simple_loss=0.09181, pruned_loss=0.0151, audio_tagging_loss=0.006433, over 15635.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.0953, pruned_loss=0.01578, audio_tagging_loss=0.009064, over 3049424.83 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:19:12,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1701293.3333333333, ans=0.0
2023-11-21 23:19:14,938 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255200
2023-11-21 23:19:34,273 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.128e+01 8.561e+01 9.314e+01 1.273e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 23:19:40,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1701426.6666666667, ans=0.0
2023-11-21 23:19:43,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.29 vs. limit=12.0
2023-11-21 23:19:46,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.33 vs. limit=15.0
2023-11-21 23:20:15,126 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2750, loss[loss=0.05021, simple_loss=0.06858, pruned_loss=0.009386, audio_tagging_loss=0.006535, over 15097.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09641, pruned_loss=0.01603, audio_tagging_loss=0.009072, over 3053098.76 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:20:19,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1701626.6666666667, ans=0.125
2023-11-21 23:20:20,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255250
2023-11-21 23:20:22,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1701626.6666666667, ans=0.5
2023-11-21 23:20:53,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1701826.6666666667, ans=0.2
2023-11-21 23:20:58,343 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.34 vs. limit=12.0
2023-11-21 23:21:08,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1701893.3333333333, ans=0.0
2023-11-21 23:21:09,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1701893.3333333333, ans=0.1
2023-11-21 23:21:10,829 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:21:11,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1701893.3333333333, ans=0.125
2023-11-21 23:21:20,685 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2800, loss[loss=0.08097, simple_loss=0.1063, pruned_loss=0.01832, audio_tagging_loss=0.009523, over 15203.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.0969, pruned_loss=0.01627, audio_tagging_loss=0.009084, over 3059434.22 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:21:24,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.96 vs. limit=15.0
2023-11-21 23:21:25,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255300
2023-11-21 23:21:27,074 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:21:35,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0
2023-11-21 23:21:41,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1702026.6666666667, ans=0.125
2023-11-21 23:21:44,828 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 7.892e+01 8.472e+01 9.341e+01 1.172e+02, threshold=1.694e+02, percent-clipped=0.0
2023-11-21 23:21:47,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1702093.3333333333, ans=0.125
2023-11-21 23:22:04,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1702160.0, ans=0.125
2023-11-21 23:22:05,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1702160.0, ans=0.125
2023-11-21 23:22:14,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.67 vs. limit=15.0
2023-11-21 23:22:25,123 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2850, loss[loss=0.06679, simple_loss=0.09033, pruned_loss=0.01078, audio_tagging_loss=0.01085, over 14283.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09679, pruned_loss=0.01628, audio_tagging_loss=0.009052, over 3050811.08 frames. ], batch size: 53, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:22:29,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1702293.3333333333, ans=0.1
2023-11-21 23:22:30,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255350
2023-11-21 23:22:34,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.36 vs. limit=15.0
2023-11-21 23:22:43,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1702360.0, ans=0.2
2023-11-21 23:23:16,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1702560.0, ans=0.1
2023-11-21 23:23:28,869 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2900, loss[loss=0.08305, simple_loss=0.1052, pruned_loss=0.01867, audio_tagging_loss=0.01176, over 14736.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09649, pruned_loss=0.01636, audio_tagging_loss=0.009121, over 3048528.21 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:23:31,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1702626.6666666667, ans=0.025
2023-11-21 23:23:33,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1702626.6666666667, ans=0.0
2023-11-21 23:23:34,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255400
2023-11-21 23:23:35,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1702626.6666666667, ans=0.95
2023-11-21 23:23:48,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.25 vs. limit=15.0
2023-11-21 23:23:54,131 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.570e+01 8.449e+01 9.076e+01 9.852e+01 1.265e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-21 23:23:56,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.75 vs. limit=15.0
2023-11-21 23:23:58,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1702760.0, ans=0.2
2023-11-21 23:24:21,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1702893.3333333333, ans=0.125
2023-11-21 23:24:33,954 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 2950, loss[loss=0.09195, simple_loss=0.1196, pruned_loss=0.02311, audio_tagging_loss=0.009018, over 14011.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09643, pruned_loss=0.01635, audio_tagging_loss=0.009187, over 3045509.16 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:24:36,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1702960.0, ans=0.125
2023-11-21 23:24:38,977 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255450
2023-11-21 23:24:41,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1702960.0, ans=0.0
2023-11-21 23:25:23,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1703160.0, ans=0.0
2023-11-21 23:25:37,774 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3000, loss[loss=0.05984, simple_loss=0.07972, pruned_loss=0.008663, audio_tagging_loss=0.01132, over 15251.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.09702, pruned_loss=0.01645, audio_tagging_loss=0.009225, over 3042835.32 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:25:37,775 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-21 23:26:16,409 INFO [train_asr.py:1253] (2/4) Epoch 22, validation: loss=0.05907, simple_loss=0.0519, pruned_loss=0.005126, audio_tagging_loss=0.02799, over 4681554.00 frames. 
2023-11-21 23:26:16,410 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-21 23:26:21,963 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255500
2023-11-21 23:26:30,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1703360.0, ans=0.2
2023-11-21 23:26:34,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.22 vs. limit=15.0
2023-11-21 23:26:41,252 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.989e+01 8.698e+01 9.491e+01 1.260e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 23:26:44,977 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.32 vs. limit=22.5
2023-11-21 23:26:53,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1703426.6666666667, ans=0.1
2023-11-21 23:26:56,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1703493.3333333333, ans=0.0
2023-11-21 23:27:07,408 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.68 vs. limit=22.5
2023-11-21 23:27:08,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0
2023-11-21 23:27:10,968 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.74 vs. limit=15.0
2023-11-21 23:27:19,901 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.05 vs. limit=15.0
2023-11-21 23:27:21,949 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3050, loss[loss=0.06153, simple_loss=0.081, pruned_loss=0.01123, audio_tagging_loss=0.009797, over 13781.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.0962, pruned_loss=0.01631, audio_tagging_loss=0.009346, over 3038148.73 frames. ], batch size: 53, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:27:23,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1703626.6666666667, ans=0.125
2023-11-21 23:27:26,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255550
2023-11-21 23:27:32,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1703626.6666666667, ans=0.125
2023-11-21 23:27:38,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1703693.3333333333, ans=0.125
2023-11-21 23:27:57,688 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:27:57,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1703826.6666666667, ans=0.0
2023-11-21 23:28:14,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1703893.3333333333, ans=0.0
2023-11-21 23:28:15,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1703893.3333333333, ans=0.125
2023-11-21 23:28:25,768 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3100, loss[loss=0.07571, simple_loss=0.09014, pruned_loss=0.01886, audio_tagging_loss=0.01178, over 14918.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09622, pruned_loss=0.01624, audio_tagging_loss=0.009351, over 3043606.70 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:28:26,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1703960.0, ans=0.125
2023-11-21 23:28:30,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255600
2023-11-21 23:28:37,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1704026.6666666667, ans=0.07
2023-11-21 23:28:51,160 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.488e+01 7.990e+01 8.561e+01 9.403e+01 1.205e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 23:29:15,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1704160.0, ans=0.0
2023-11-21 23:29:17,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1704226.6666666667, ans=0.2
2023-11-21 23:29:26,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1704226.6666666667, ans=0.2
2023-11-21 23:29:29,634 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3150, loss[loss=0.06052, simple_loss=0.074, pruned_loss=0.01107, audio_tagging_loss=0.01245, over 15322.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09595, pruned_loss=0.01619, audio_tagging_loss=0.009516, over 3037039.53 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:29:34,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255650
2023-11-21 23:29:48,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1704360.0, ans=0.0
2023-11-21 23:29:53,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.21 vs. limit=15.0
2023-11-21 23:30:12,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1704493.3333333333, ans=0.125
2023-11-21 23:30:35,312 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3200, loss[loss=0.07515, simple_loss=0.1043, pruned_loss=0.01411, audio_tagging_loss=0.008901, over 15433.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.0955, pruned_loss=0.01599, audio_tagging_loss=0.009698, over 3038478.41 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:30:39,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.63 vs. limit=22.5
2023-11-21 23:30:40,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255700
2023-11-21 23:30:53,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1704693.3333333333, ans=0.0
2023-11-21 23:30:54,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1704693.3333333333, ans=0.125
2023-11-21 23:31:01,651 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.275e+01 8.900e+01 9.462e+01 1.368e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 23:31:07,458 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.56 vs. limit=15.0
2023-11-21 23:31:37,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1704893.3333333333, ans=0.125
2023-11-21 23:31:39,788 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3250, loss[loss=0.06027, simple_loss=0.07453, pruned_loss=0.009313, audio_tagging_loss=0.0137, over 16460.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09476, pruned_loss=0.01581, audio_tagging_loss=0.009837, over 3042377.99 frames. ], batch size: 62, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:31:44,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255750
2023-11-21 23:31:47,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1704960.0, ans=0.0
2023-11-21 23:31:51,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1705026.6666666667, ans=0.1
2023-11-21 23:31:58,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1705026.6666666667, ans=0.0
2023-11-21 23:32:08,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1705093.3333333333, ans=0.0
2023-11-21 23:32:10,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1705093.3333333333, ans=0.0
2023-11-21 23:32:10,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1705093.3333333333, ans=0.125
2023-11-21 23:32:14,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1705093.3333333333, ans=0.1
2023-11-21 23:32:17,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1705160.0, ans=0.125
2023-11-21 23:32:30,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1705226.6666666667, ans=0.1
2023-11-21 23:32:36,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1705226.6666666667, ans=0.125
2023-11-21 23:32:40,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1705226.6666666667, ans=0.0
2023-11-21 23:32:40,647 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.93 vs. limit=15.0
2023-11-21 23:32:43,660 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3300, loss[loss=0.07376, simple_loss=0.09582, pruned_loss=0.0163, audio_tagging_loss=0.009546, over 15274.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.09407, pruned_loss=0.01573, audio_tagging_loss=0.009883, over 3047004.81 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:32:48,621 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255800
2023-11-21 23:33:04,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1705360.0, ans=0.2
2023-11-21 23:33:11,148 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.185e+01 8.316e+01 8.844e+01 9.343e+01 1.264e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 23:33:30,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1705493.3333333333, ans=0.125
2023-11-21 23:33:39,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1705560.0, ans=0.0
2023-11-21 23:33:42,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1705560.0, ans=0.025
2023-11-21 23:33:47,417 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3350, loss[loss=0.05615, simple_loss=0.07846, pruned_loss=0.009732, audio_tagging_loss=0.007185, over 14268.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.0954, pruned_loss=0.01593, audio_tagging_loss=0.009664, over 3041338.07 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:33:53,039 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255850
2023-11-21 23:33:53,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1705626.6666666667, ans=0.125
2023-11-21 23:34:52,881 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3400, loss[loss=0.1019, simple_loss=0.1403, pruned_loss=0.02652, audio_tagging_loss=0.005222, over 16190.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09662, pruned_loss=0.01618, audio_tagging_loss=0.009566, over 3043107.66 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:34:57,775 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255900
2023-11-21 23:34:59,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1705960.0, ans=0.0
2023-11-21 23:35:17,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1706093.3333333333, ans=0.125
2023-11-21 23:35:18,273 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 8.132e+01 8.792e+01 9.497e+01 1.160e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 23:35:26,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1706093.3333333333, ans=0.125
2023-11-21 23:35:45,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1706226.6666666667, ans=0.2
2023-11-21 23:35:50,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1706226.6666666667, ans=0.125
2023-11-21 23:35:56,091 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3450, loss[loss=0.07669, simple_loss=0.1024, pruned_loss=0.01806, audio_tagging_loss=0.007444, over 15690.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09724, pruned_loss=0.01613, audio_tagging_loss=0.009445, over 3048861.41 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:35:58,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1706293.3333333333, ans=0.125
2023-11-21 23:36:00,588 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.77 vs. limit=22.5
2023-11-21 23:36:01,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 255950
2023-11-21 23:36:01,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1706293.3333333333, ans=0.125
2023-11-21 23:36:42,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1706493.3333333333, ans=0.1
2023-11-21 23:36:44,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.18 vs. limit=10.0
2023-11-21 23:36:48,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1706560.0, ans=10.0
2023-11-21 23:36:58,661 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.72 vs. limit=15.0
2023-11-21 23:36:59,403 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3500, loss[loss=0.07111, simple_loss=0.09208, pruned_loss=0.01388, audio_tagging_loss=0.01119, over 14751.00 frames. ], tot_loss[loss=0.07363, simple_loss=0.09622, pruned_loss=0.01604, audio_tagging_loss=0.009477, over 3047424.68 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:37:05,530 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256000
2023-11-21 23:37:11,854 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.12 vs. limit=6.0
2023-11-21 23:37:31,121 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 7.990e+01 8.553e+01 9.404e+01 1.412e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 23:37:37,241 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:37:55,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1706893.3333333333, ans=0.2
2023-11-21 23:38:00,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1706893.3333333333, ans=0.125
2023-11-21 23:38:05,958 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.92 vs. limit=15.0
2023-11-21 23:38:08,319 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3550, loss[loss=0.06406, simple_loss=0.07753, pruned_loss=0.01405, audio_tagging_loss=0.01124, over 16057.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09548, pruned_loss=0.01583, audio_tagging_loss=0.009485, over 3048458.88 frames. ], batch size: 62, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:38:13,909 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256050
2023-11-21 23:38:17,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1706960.0, ans=0.1
2023-11-21 23:38:22,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1707026.6666666667, ans=0.0
2023-11-21 23:39:12,518 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3600, loss[loss=0.06509, simple_loss=0.07737, pruned_loss=0.01327, audio_tagging_loss=0.01314, over 15185.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09586, pruned_loss=0.01595, audio_tagging_loss=0.009358, over 3044323.09 frames. ], batch size: 60, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:39:14,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1707293.3333333333, ans=0.0
2023-11-21 23:39:14,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1707293.3333333333, ans=0.0
2023-11-21 23:39:15,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1707293.3333333333, ans=0.0
2023-11-21 23:39:17,487 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256100
2023-11-21 23:39:20,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1707293.3333333333, ans=0.125
2023-11-21 23:39:21,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1707293.3333333333, ans=0.125
2023-11-21 23:39:31,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1707360.0, ans=0.0
2023-11-21 23:39:37,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1707426.6666666667, ans=0.125
2023-11-21 23:39:39,532 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.698e+01 8.079e+01 8.522e+01 9.097e+01 1.276e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 23:39:51,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1707493.3333333333, ans=0.0
2023-11-21 23:39:52,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1707493.3333333333, ans=0.125
2023-11-21 23:39:56,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1707493.3333333333, ans=0.05
2023-11-21 23:39:57,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1707493.3333333333, ans=0.125
2023-11-21 23:40:11,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1707560.0, ans=0.125
2023-11-21 23:40:16,167 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3650, loss[loss=0.08714, simple_loss=0.1095, pruned_loss=0.02265, audio_tagging_loss=0.00972, over 15504.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09548, pruned_loss=0.01594, audio_tagging_loss=0.009364, over 3045667.88 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:40:21,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.26 vs. limit=22.5
2023-11-21 23:40:21,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256150
2023-11-21 23:40:25,307 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.29 vs. limit=15.0
2023-11-21 23:40:37,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1707693.3333333333, ans=0.125
2023-11-21 23:40:54,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1707826.6666666667, ans=0.125
2023-11-21 23:40:57,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1707826.6666666667, ans=0.0
2023-11-21 23:41:08,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1707893.3333333333, ans=0.125
2023-11-21 23:41:21,383 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3700, loss[loss=0.08113, simple_loss=0.09798, pruned_loss=0.02347, audio_tagging_loss=0.008672, over 15654.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09574, pruned_loss=0.01606, audio_tagging_loss=0.009352, over 3046492.56 frames. ], batch size: 60, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:41:26,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1707960.0, ans=0.125
2023-11-21 23:41:27,188 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256200
2023-11-21 23:41:50,114 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.207e+01 8.757e+01 9.528e+01 1.164e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 23:42:00,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1708160.0, ans=0.1
2023-11-21 23:42:22,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.59 vs. limit=22.5
2023-11-21 23:42:26,867 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3750, loss[loss=0.07308, simple_loss=0.09415, pruned_loss=0.01707, audio_tagging_loss=0.008935, over 15728.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.0965, pruned_loss=0.01629, audio_tagging_loss=0.009406, over 3046431.13 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:42:31,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256250
2023-11-21 23:42:51,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1708426.6666666667, ans=0.125
2023-11-21 23:43:11,575 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:43:20,283 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.07 vs. limit=10.0
2023-11-21 23:43:30,482 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3800, loss[loss=0.09673, simple_loss=0.1259, pruned_loss=0.02639, audio_tagging_loss=0.007376, over 15000.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09578, pruned_loss=0.01613, audio_tagging_loss=0.00948, over 3044613.01 frames. ], batch size: 54, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:43:36,261 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256300
2023-11-21 23:43:55,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1708693.3333333333, ans=0.125
2023-11-21 23:43:55,523 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.57 vs. limit=8.0
2023-11-21 23:43:59,423 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.403e+01 8.924e+01 9.481e+01 1.120e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 23:43:59,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1708760.0, ans=0.125
2023-11-21 23:44:00,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1708760.0, ans=0.09899494936611666
2023-11-21 23:44:35,822 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3850, loss[loss=0.07181, simple_loss=0.1001, pruned_loss=0.01438, audio_tagging_loss=0.007373, over 14088.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09536, pruned_loss=0.01598, audio_tagging_loss=0.009518, over 3046292.25 frames. ], batch size: 54, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:44:40,797 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256350
2023-11-21 23:44:48,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1709026.6666666667, ans=0.125
2023-11-21 23:45:02,603 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.61 vs. limit=6.0
2023-11-21 23:45:05,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1709093.3333333333, ans=0.125
2023-11-21 23:45:05,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1709093.3333333333, ans=0.125
2023-11-21 23:45:24,189 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:45:26,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1709226.6666666667, ans=0.0
2023-11-21 23:45:33,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1709226.6666666667, ans=0.0
2023-11-21 23:45:35,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1709226.6666666667, ans=0.0
2023-11-21 23:45:40,176 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3900, loss[loss=0.08114, simple_loss=0.1004, pruned_loss=0.02078, audio_tagging_loss=0.01018, over 15661.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.0955, pruned_loss=0.01602, audio_tagging_loss=0.009493, over 3042709.63 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:45:45,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256400
2023-11-21 23:45:52,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1709360.0, ans=0.1
2023-11-21 23:46:05,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-21 23:46:09,330 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.019e+01 8.673e+01 9.477e+01 1.418e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 23:46:19,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1709493.3333333333, ans=0.2
2023-11-21 23:46:25,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1709493.3333333333, ans=0.0
2023-11-21 23:46:42,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1709560.0, ans=0.125
2023-11-21 23:46:45,104 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 3950, loss[loss=0.08018, simple_loss=0.1002, pruned_loss=0.01802, audio_tagging_loss=0.01206, over 15790.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09509, pruned_loss=0.01585, audio_tagging_loss=0.009447, over 3034023.17 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:46:50,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256450
2023-11-21 23:46:56,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.06 vs. limit=6.0
2023-11-21 23:47:11,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1709760.0, ans=0.0
2023-11-21 23:47:17,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.92 vs. limit=15.0
2023-11-21 23:47:49,833 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4000, loss[loss=0.08574, simple_loss=0.1097, pruned_loss=0.01992, audio_tagging_loss=0.01096, over 14705.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09575, pruned_loss=0.01594, audio_tagging_loss=0.009636, over 3037831.79 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:47:54,743 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256500
2023-11-21 23:47:55,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1709960.0, ans=6.0
2023-11-21 23:47:56,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1709960.0, ans=0.0
2023-11-21 23:47:59,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1709960.0, ans=0.2
2023-11-21 23:48:06,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1710026.6666666667, ans=0.0
2023-11-21 23:48:08,325 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.52 vs. limit=10.0
2023-11-21 23:48:10,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1710026.6666666667, ans=0.2
2023-11-21 23:48:10,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1710026.6666666667, ans=0.125
2023-11-21 23:48:11,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1710026.6666666667, ans=0.1
2023-11-21 23:48:14,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1710093.3333333333, ans=0.1
2023-11-21 23:48:16,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1710093.3333333333, ans=0.0
2023-11-21 23:48:17,178 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.007e+01 8.495e+01 8.892e+01 9.387e+01 1.240e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 23:48:22,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1710093.3333333333, ans=0.2
2023-11-21 23:48:50,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1710226.6666666667, ans=0.04949747468305833
2023-11-21 23:48:51,800 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.02 vs. limit=15.0
2023-11-21 23:48:52,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1710293.3333333333, ans=0.0
2023-11-21 23:48:53,478 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4050, loss[loss=0.07306, simple_loss=0.09463, pruned_loss=0.01507, audio_tagging_loss=0.01067, over 14892.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09591, pruned_loss=0.01606, audio_tagging_loss=0.009639, over 3035674.83 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:48:57,251 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:48:58,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256550
2023-11-21 23:49:04,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1710360.0, ans=0.95
2023-11-21 23:49:24,195 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.58 vs. limit=15.0
2023-11-21 23:49:42,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1710493.3333333333, ans=0.125
2023-11-21 23:49:53,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1710560.0, ans=0.0
2023-11-21 23:49:56,993 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4100, loss[loss=0.07358, simple_loss=0.09494, pruned_loss=0.01483, audio_tagging_loss=0.01128, over 15782.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09596, pruned_loss=0.01607, audio_tagging_loss=0.009611, over 3042554.16 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:50:02,583 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256600
2023-11-21 23:50:08,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1710626.6666666667, ans=0.0
2023-11-21 23:50:14,566 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.72 vs. limit=6.0
2023-11-21 23:50:17,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1710693.3333333333, ans=0.2
2023-11-21 23:50:17,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1710693.3333333333, ans=0.1
2023-11-21 23:50:26,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1710760.0, ans=0.1
2023-11-21 23:50:26,954 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.295e+01 8.154e+01 8.730e+01 9.411e+01 1.441e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 23:50:41,136 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.24 vs. limit=15.0
2023-11-21 23:50:45,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1710826.6666666667, ans=0.0
2023-11-21 23:50:55,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1710893.3333333333, ans=0.2
2023-11-21 23:50:56,330 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.33 vs. limit=10.0
2023-11-21 23:51:03,034 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4150, loss[loss=0.06472, simple_loss=0.08247, pruned_loss=0.01268, audio_tagging_loss=0.0108, over 14956.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09591, pruned_loss=0.01598, audio_tagging_loss=0.009465, over 3036010.85 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:51:08,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256650
2023-11-21 23:51:20,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1711026.6666666667, ans=0.1
2023-11-21 23:51:24,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1711026.6666666667, ans=0.125
2023-11-21 23:51:50,770 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:52:05,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1711226.6666666667, ans=0.04949747468305833
2023-11-21 23:52:08,132 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4200, loss[loss=0.0651, simple_loss=0.07745, pruned_loss=0.01272, audio_tagging_loss=0.01365, over 15039.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09517, pruned_loss=0.01588, audio_tagging_loss=0.0094, over 3041164.12 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:52:11,418 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-21 23:52:13,141 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256700
2023-11-21 23:52:22,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.88 vs. limit=15.0
2023-11-21 23:52:36,095 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 7.996e+01 8.840e+01 9.314e+01 1.225e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 23:52:42,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1711426.6666666667, ans=0.125
2023-11-21 23:52:48,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1711493.3333333333, ans=0.1
2023-11-21 23:52:49,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.77 vs. limit=6.0
2023-11-21 23:53:06,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.33 vs. limit=15.0
2023-11-21 23:53:11,823 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4250, loss[loss=0.06127, simple_loss=0.07903, pruned_loss=0.01282, audio_tagging_loss=0.008931, over 14644.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09524, pruned_loss=0.01581, audio_tagging_loss=0.009308, over 3041936.64 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:53:16,991 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256750
2023-11-21 23:54:01,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1711826.6666666667, ans=0.125
2023-11-21 23:54:07,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1711893.3333333333, ans=0.0
2023-11-21 23:54:10,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1711893.3333333333, ans=0.04949747468305833
2023-11-21 23:54:13,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1711893.3333333333, ans=0.1
2023-11-21 23:54:16,093 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4300, loss[loss=0.05412, simple_loss=0.06474, pruned_loss=0.01365, audio_tagging_loss=0.008094, over 15107.00 frames. ], tot_loss[loss=0.07285, simple_loss=0.09528, pruned_loss=0.01595, audio_tagging_loss=0.00926, over 3045030.02 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:54:22,245 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256800
2023-11-21 23:54:22,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1711960.0, ans=0.0
2023-11-21 23:54:45,105 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.276e+01 8.988e+01 9.758e+01 2.160e+02, threshold=1.798e+02, percent-clipped=1.0
2023-11-21 23:55:21,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1712293.3333333333, ans=10.0
2023-11-21 23:55:22,049 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4350, loss[loss=0.06519, simple_loss=0.08205, pruned_loss=0.01242, audio_tagging_loss=0.01175, over 16301.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09502, pruned_loss=0.01581, audio_tagging_loss=0.009275, over 3038696.30 frames. ], batch size: 65, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:55:26,955 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256850
2023-11-21 23:55:35,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1712360.0, ans=0.0
2023-11-21 23:56:05,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1712493.3333333333, ans=0.0
2023-11-21 23:56:25,329 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4400, loss[loss=0.06725, simple_loss=0.1027, pruned_loss=0.01079, audio_tagging_loss=0.00513, over 15440.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09449, pruned_loss=0.01588, audio_tagging_loss=0.009313, over 3038903.50 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:56:30,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256900
2023-11-21 23:56:36,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1712693.3333333333, ans=0.2
2023-11-21 23:56:55,228 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.141e+01 8.812e+01 9.440e+01 1.228e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 23:56:57,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1712760.0, ans=0.125
2023-11-21 23:57:29,766 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4450, loss[loss=0.07366, simple_loss=0.09828, pruned_loss=0.01775, audio_tagging_loss=0.006776, over 14453.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09527, pruned_loss=0.01606, audio_tagging_loss=0.009338, over 3050261.21 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:57:35,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 256950
2023-11-21 23:58:09,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1713160.0, ans=0.0
2023-11-21 23:58:32,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1713226.6666666667, ans=0.125
2023-11-21 23:58:34,701 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4500, loss[loss=0.07296, simple_loss=0.09976, pruned_loss=0.01596, audio_tagging_loss=0.007127, over 15532.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09574, pruned_loss=0.01614, audio_tagging_loss=0.009283, over 3047036.04 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:58:40,207 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257000
2023-11-21 23:58:48,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1713360.0, ans=0.125
2023-11-21 23:58:51,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1713360.0, ans=0.125
2023-11-21 23:59:01,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1713426.6666666667, ans=0.0
2023-11-21 23:59:03,860 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.360e+01 9.049e+01 9.861e+01 1.318e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-21 23:59:07,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.70 vs. limit=15.0
2023-11-21 23:59:12,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1713493.3333333333, ans=0.05
2023-11-21 23:59:39,438 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4550, loss[loss=0.0714, simple_loss=0.08673, pruned_loss=0.01724, audio_tagging_loss=0.01079, over 15150.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09515, pruned_loss=0.01606, audio_tagging_loss=0.009365, over 3043886.47 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:59:44,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257050
2023-11-21 23:59:51,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1713693.3333333333, ans=0.0
2023-11-22 00:00:10,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1713760.0, ans=0.2
2023-11-22 00:00:10,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn1.whiten.whitening_limit, batch_count=1713760.0, ans=22.5
2023-11-22 00:00:20,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1713826.6666666667, ans=0.125
2023-11-22 00:00:23,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1713826.6666666667, ans=0.1
2023-11-22 00:00:27,879 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:00:37,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1713893.3333333333, ans=0.125
2023-11-22 00:00:42,545 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4600, loss[loss=0.08622, simple_loss=0.1215, pruned_loss=0.01987, audio_tagging_loss=0.00558, over 15681.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09541, pruned_loss=0.0161, audio_tagging_loss=0.009401, over 3050993.87 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 32.0
2023-11-22 00:00:48,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257100
2023-11-22 00:00:58,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1714026.6666666667, ans=15.0
2023-11-22 00:01:13,014 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 7.906e+01 8.473e+01 9.187e+01 1.183e+02, threshold=1.695e+02, percent-clipped=0.0
2023-11-22 00:01:23,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1714160.0, ans=0.125
2023-11-22 00:01:23,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1714160.0, ans=0.125
2023-11-22 00:01:31,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1714160.0, ans=0.0
2023-11-22 00:01:35,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1714226.6666666667, ans=15.0
2023-11-22 00:01:36,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1714226.6666666667, ans=0.0
2023-11-22 00:01:42,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1714226.6666666667, ans=0.0
2023-11-22 00:01:47,033 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4650, loss[loss=0.06746, simple_loss=0.07998, pruned_loss=0.01637, audio_tagging_loss=0.0111, over 15087.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09489, pruned_loss=0.01588, audio_tagging_loss=0.009512, over 3048727.78 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0
2023-11-22 00:01:50,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1714293.3333333333, ans=0.0
2023-11-22 00:01:53,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257150
2023-11-22 00:02:31,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1714493.3333333333, ans=0.0
2023-11-22 00:02:34,201 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.73 vs. limit=10.0
2023-11-22 00:02:35,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.12 vs. limit=15.0
2023-11-22 00:02:45,334 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:02:51,269 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4700, loss[loss=0.07624, simple_loss=0.1049, pruned_loss=0.01418, audio_tagging_loss=0.009599, over 16109.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.09483, pruned_loss=0.0159, audio_tagging_loss=0.009568, over 3049445.95 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 32.0
2023-11-22 00:02:56,383 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257200
2023-11-22 00:03:04,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1714693.3333333333, ans=0.015
2023-11-22 00:03:13,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=1714693.3333333333, ans=10.0
2023-11-22 00:03:13,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.85 vs. limit=15.0
2023-11-22 00:03:18,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.90 vs. limit=15.0
2023-11-22 00:03:21,303 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.209e+01 8.969e+01 9.577e+01 1.305e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 00:03:21,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1714760.0, ans=0.125
2023-11-22 00:03:36,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1714826.6666666667, ans=0.0
2023-11-22 00:03:56,122 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4750, loss[loss=0.07129, simple_loss=0.08383, pruned_loss=0.01937, audio_tagging_loss=0.01001, over 15775.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.0947, pruned_loss=0.01565, audio_tagging_loss=0.009602, over 3047433.02 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:04:01,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257250
2023-11-22 00:04:07,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1714960.0, ans=0.1
2023-11-22 00:04:15,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.21 vs. limit=6.0
2023-11-22 00:04:42,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1715160.0, ans=0.1
2023-11-22 00:04:58,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1715226.6666666667, ans=0.125
2023-11-22 00:05:00,621 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4800, loss[loss=0.07025, simple_loss=0.09571, pruned_loss=0.0144, audio_tagging_loss=0.007987, over 14921.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09565, pruned_loss=0.01591, audio_tagging_loss=0.009751, over 3044410.90 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:05:02,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1715293.3333333333, ans=0.0
2023-11-22 00:05:05,546 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257300
2023-11-22 00:05:27,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1715426.6666666667, ans=0.125
2023-11-22 00:05:27,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1715426.6666666667, ans=0.125
2023-11-22 00:05:30,003 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.329e+01 8.144e+01 8.815e+01 9.467e+01 1.388e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 00:05:53,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1715560.0, ans=0.125
2023-11-22 00:06:05,294 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4850, loss[loss=0.06533, simple_loss=0.07753, pruned_loss=0.01254, audio_tagging_loss=0.01402, over 16258.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09641, pruned_loss=0.01597, audio_tagging_loss=0.009793, over 3045887.05 frames. ], batch size: 61, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:06:10,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257350
2023-11-22 00:06:10,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1715626.6666666667, ans=0.0
2023-11-22 00:06:13,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1715626.6666666667, ans=0.125
2023-11-22 00:06:26,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.48 vs. limit=12.0
2023-11-22 00:06:35,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1715760.0, ans=0.125
2023-11-22 00:06:37,797 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.59 vs. limit=15.0
2023-11-22 00:06:43,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1715826.6666666667, ans=0.0
2023-11-22 00:06:46,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1715826.6666666667, ans=0.0
2023-11-22 00:06:51,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1715826.6666666667, ans=0.0
2023-11-22 00:06:58,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1715893.3333333333, ans=0.0
2023-11-22 00:07:09,099 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4900, loss[loss=0.06545, simple_loss=0.08752, pruned_loss=0.01388, audio_tagging_loss=0.007818, over 15423.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.0963, pruned_loss=0.01598, audio_tagging_loss=0.009637, over 3046783.48 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:07:12,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1715960.0, ans=0.04949747468305833
2023-11-22 00:07:14,208 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257400
2023-11-22 00:07:38,983 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.481e+01 8.110e+01 8.831e+01 9.694e+01 1.184e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 00:07:43,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1716093.3333333333, ans=0.125
2023-11-22 00:07:44,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1716093.3333333333, ans=0.0
2023-11-22 00:07:45,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1716093.3333333333, ans=0.0
2023-11-22 00:07:46,522 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.53 vs. limit=15.0
2023-11-22 00:08:03,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.40 vs. limit=15.0
2023-11-22 00:08:10,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1716226.6666666667, ans=0.125
2023-11-22 00:08:14,172 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 4950, loss[loss=0.1011, simple_loss=0.1312, pruned_loss=0.02812, audio_tagging_loss=0.007353, over 15888.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09559, pruned_loss=0.01575, audio_tagging_loss=0.009434, over 3051699.35 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:08:19,139 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257450
2023-11-22 00:08:24,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1716293.3333333333, ans=0.125
2023-11-22 00:08:56,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1716493.3333333333, ans=0.125
2023-11-22 00:09:05,824 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=6.231e-02
2023-11-22 00:09:13,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1716560.0, ans=0.125
2023-11-22 00:09:18,236 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5000, loss[loss=0.06714, simple_loss=0.09185, pruned_loss=0.01304, audio_tagging_loss=0.008176, over 15738.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09595, pruned_loss=0.01576, audio_tagging_loss=0.009248, over 3048548.30 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:09:22,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1716626.6666666667, ans=0.2
2023-11-22 00:09:23,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257500
2023-11-22 00:09:33,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1716693.3333333333, ans=0.125
2023-11-22 00:09:42,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.58 vs. limit=15.0
2023-11-22 00:09:48,022 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.146e+01 8.929e+01 9.625e+01 1.123e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 00:09:50,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.40 vs. limit=6.0
2023-11-22 00:10:09,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1716893.3333333333, ans=0.125
2023-11-22 00:10:22,343 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5050, loss[loss=0.07851, simple_loss=0.09599, pruned_loss=0.02129, audio_tagging_loss=0.009223, over 13644.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09668, pruned_loss=0.01605, audio_tagging_loss=0.009083, over 3042947.06 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:10:27,507 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257550
2023-11-22 00:10:30,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1716960.0, ans=0.125
2023-11-22 00:10:35,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.28 vs. limit=15.0
2023-11-22 00:10:47,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1717093.3333333333, ans=0.125
2023-11-22 00:10:53,764 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.75 vs. limit=15.0
2023-11-22 00:11:00,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1717160.0, ans=0.125
2023-11-22 00:11:02,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1717160.0, ans=0.125
2023-11-22 00:11:27,999 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5100, loss[loss=0.0696, simple_loss=0.09295, pruned_loss=0.01469, audio_tagging_loss=0.008439, over 15531.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09606, pruned_loss=0.01597, audio_tagging_loss=0.009162, over 3047890.89 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:11:30,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1717293.3333333333, ans=0.035
2023-11-22 00:11:33,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257600
2023-11-22 00:11:50,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1717360.0, ans=0.1
2023-11-22 00:11:55,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1717426.6666666667, ans=0.125
2023-11-22 00:11:58,340 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.309e+01 7.923e+01 8.751e+01 9.756e+01 1.347e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 00:12:00,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1717426.6666666667, ans=0.1
2023-11-22 00:12:26,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1717560.0, ans=0.0
2023-11-22 00:12:34,124 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5150, loss[loss=0.05888, simple_loss=0.0717, pruned_loss=0.009614, audio_tagging_loss=0.01341, over 15839.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09589, pruned_loss=0.01599, audio_tagging_loss=0.009285, over 3051089.33 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:12:39,138 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257650
2023-11-22 00:13:08,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1717760.0, ans=0.1
2023-11-22 00:13:39,407 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5200, loss[loss=0.05879, simple_loss=0.07445, pruned_loss=0.009976, audio_tagging_loss=0.01159, over 15774.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09576, pruned_loss=0.01602, audio_tagging_loss=0.009269, over 3047462.01 frames. ], batch size: 62, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:13:40,824 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:13:44,444 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257700
2023-11-22 00:14:06,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1718093.3333333333, ans=0.125
2023-11-22 00:14:07,111 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.63 vs. limit=15.0
2023-11-22 00:14:09,391 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.143e+01 8.728e+01 9.361e+01 2.351e+02, threshold=1.746e+02, percent-clipped=1.0
2023-11-22 00:14:20,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1718160.0, ans=0.125
2023-11-22 00:14:43,503 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5250, loss[loss=0.07174, simple_loss=0.09608, pruned_loss=0.01461, audio_tagging_loss=0.009081, over 15770.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09613, pruned_loss=0.01613, audio_tagging_loss=0.009175, over 3042085.37 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:14:49,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257750
2023-11-22 00:14:51,098 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.92 vs. limit=15.0
2023-11-22 00:14:52,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1718293.3333333333, ans=0.0
2023-11-22 00:14:52,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1718293.3333333333, ans=0.125
2023-11-22 00:14:54,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1718293.3333333333, ans=0.125
2023-11-22 00:15:20,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1718493.3333333333, ans=0.0
2023-11-22 00:15:25,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1718493.3333333333, ans=0.0
2023-11-22 00:15:36,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1718560.0, ans=0.0
2023-11-22 00:15:44,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1718560.0, ans=0.1
2023-11-22 00:15:48,174 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5300, loss[loss=0.06602, simple_loss=0.0825, pruned_loss=0.01237, audio_tagging_loss=0.0124, over 14752.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09641, pruned_loss=0.01623, audio_tagging_loss=0.009231, over 3038004.66 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:15:52,415 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.78 vs. limit=15.0
2023-11-22 00:15:53,216 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257800
2023-11-22 00:16:17,183 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.246e+01 8.912e+01 9.804e+01 1.159e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 00:16:20,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1718760.0, ans=0.015
2023-11-22 00:16:30,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1718826.6666666667, ans=0.125
2023-11-22 00:16:42,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1718893.3333333333, ans=0.0
2023-11-22 00:16:52,129 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5350, loss[loss=0.07867, simple_loss=0.08507, pruned_loss=0.02389, audio_tagging_loss=0.01225, over 15678.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09724, pruned_loss=0.01642, audio_tagging_loss=0.009241, over 3037892.23 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:16:53,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1718960.0, ans=0.125
2023-11-22 00:16:56,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1718960.0, ans=0.125
2023-11-22 00:16:57,224 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257850
2023-11-22 00:17:34,397 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.51 vs. limit=12.0
2023-11-22 00:17:35,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1719160.0, ans=0.0
2023-11-22 00:17:47,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1719226.6666666667, ans=0.125
2023-11-22 00:17:57,064 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5400, loss[loss=0.09704, simple_loss=0.1267, pruned_loss=0.02401, audio_tagging_loss=0.009695, over 15612.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09627, pruned_loss=0.01607, audio_tagging_loss=0.009355, over 3038602.03 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:18:02,546 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257900
2023-11-22 00:18:10,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1719360.0, ans=0.125
2023-11-22 00:18:19,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1719360.0, ans=0.035
2023-11-22 00:18:22,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1719426.6666666667, ans=0.1
2023-11-22 00:18:27,541 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 7.882e+01 8.565e+01 9.189e+01 1.192e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-22 00:18:29,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1719426.6666666667, ans=0.125
2023-11-22 00:18:54,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1719560.0, ans=0.0
2023-11-22 00:18:55,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1719560.0, ans=0.0
2023-11-22 00:19:01,108 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5450, loss[loss=0.1003, simple_loss=0.1385, pruned_loss=0.02431, audio_tagging_loss=0.006721, over 17164.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09589, pruned_loss=0.01594, audio_tagging_loss=0.009362, over 3047185.59 frames. ], batch size: 62, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:19:06,691 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 257950
2023-11-22 00:19:21,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1719693.3333333333, ans=0.125
2023-11-22 00:19:52,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1719893.3333333333, ans=0.125
2023-11-22 00:19:59,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1719893.3333333333, ans=0.125
2023-11-22 00:20:05,631 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5500, loss[loss=0.0796, simple_loss=0.09932, pruned_loss=0.01985, audio_tagging_loss=0.01009, over 14011.00 frames. ], tot_loss[loss=0.07363, simple_loss=0.09652, pruned_loss=0.016, audio_tagging_loss=0.009372, over 3043831.00 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:20:08,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1719960.0, ans=0.125
2023-11-22 00:20:08,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.61 vs. limit=15.0
2023-11-22 00:20:10,533 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258000
2023-11-22 00:20:11,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1719960.0, ans=0.2
2023-11-22 00:20:21,214 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.14 vs. limit=15.0
2023-11-22 00:20:36,640 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.285e+01 8.796e+01 9.521e+01 1.194e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 00:20:48,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1720160.0, ans=0.125
2023-11-22 00:20:49,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1720160.0, ans=0.05
2023-11-22 00:21:02,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1720226.6666666667, ans=0.0
2023-11-22 00:21:03,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1720226.6666666667, ans=0.1
2023-11-22 00:21:09,030 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5550, loss[loss=0.06107, simple_loss=0.08411, pruned_loss=0.008451, audio_tagging_loss=0.01056, over 14427.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09644, pruned_loss=0.01595, audio_tagging_loss=0.009408, over 3041780.12 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:21:14,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258050
2023-11-22 00:21:18,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1720293.3333333333, ans=0.125
2023-11-22 00:21:44,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1720426.6666666667, ans=0.0
2023-11-22 00:22:02,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1720560.0, ans=0.2
2023-11-22 00:22:09,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1720560.0, ans=0.0
2023-11-22 00:22:13,630 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5600, loss[loss=0.0646, simple_loss=0.0795, pruned_loss=0.01445, audio_tagging_loss=0.0104, over 14818.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09667, pruned_loss=0.01609, audio_tagging_loss=0.009522, over 3039503.25 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:22:16,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1720626.6666666667, ans=0.125
2023-11-22 00:22:19,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258100
2023-11-22 00:22:31,495 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:22:35,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1720693.3333333333, ans=0.125
2023-11-22 00:22:37,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1720760.0, ans=0.125
2023-11-22 00:22:43,446 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 7.788e+01 8.528e+01 9.166e+01 1.353e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-22 00:22:51,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1720826.6666666667, ans=0.125
2023-11-22 00:22:57,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1720826.6666666667, ans=0.125
2023-11-22 00:22:59,859 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:23:00,323 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=12.0
2023-11-22 00:23:05,492 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.96 vs. limit=22.5
2023-11-22 00:23:14,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1720893.3333333333, ans=0.125
2023-11-22 00:23:15,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1720893.3333333333, ans=0.125
2023-11-22 00:23:17,563 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5650, loss[loss=0.05345, simple_loss=0.06665, pruned_loss=0.008162, audio_tagging_loss=0.01196, over 14585.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09508, pruned_loss=0.01579, audio_tagging_loss=0.009612, over 3040533.87 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:23:19,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1720960.0, ans=0.0
2023-11-22 00:23:22,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258150
2023-11-22 00:23:25,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.12 vs. limit=15.0
2023-11-22 00:23:49,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.66 vs. limit=15.0
2023-11-22 00:24:05,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1721160.0, ans=0.1
2023-11-22 00:24:19,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1721293.3333333333, ans=0.125
2023-11-22 00:24:21,011 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5700, loss[loss=0.0822, simple_loss=0.09717, pruned_loss=0.02309, audio_tagging_loss=0.01053, over 15538.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09544, pruned_loss=0.01587, audio_tagging_loss=0.009692, over 3037846.56 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:24:21,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1721293.3333333333, ans=0.0
2023-11-22 00:24:25,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258200
2023-11-22 00:24:28,175 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.60 vs. limit=6.0
2023-11-22 00:24:45,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.71 vs. limit=22.5
2023-11-22 00:24:52,713 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.074e+01 8.821e+01 9.592e+01 1.274e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 00:24:56,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1721426.6666666667, ans=0.1
2023-11-22 00:25:04,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0
2023-11-22 00:25:06,696 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:25:17,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1721560.0, ans=0.0
2023-11-22 00:25:26,451 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5750, loss[loss=0.06691, simple_loss=0.081, pruned_loss=0.01644, audio_tagging_loss=0.009969, over 15599.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09428, pruned_loss=0.01561, audio_tagging_loss=0.00974, over 3040916.55 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:25:31,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258250
2023-11-22 00:25:54,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.86 vs. limit=6.0
2023-11-22 00:26:13,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1721826.6666666667, ans=0.125
2023-11-22 00:26:24,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1721893.3333333333, ans=0.125
2023-11-22 00:26:30,004 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5800, loss[loss=0.05581, simple_loss=0.07881, pruned_loss=0.009713, audio_tagging_loss=0.006695, over 16085.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09463, pruned_loss=0.01577, audio_tagging_loss=0.009502, over 3039542.48 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:26:33,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1721960.0, ans=0.1
2023-11-22 00:26:34,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258300
2023-11-22 00:26:47,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1722026.6666666667, ans=0.0
2023-11-22 00:26:55,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1722093.3333333333, ans=0.07
2023-11-22 00:27:01,814 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.499e+01 8.125e+01 8.820e+01 9.414e+01 1.363e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 00:27:04,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1722093.3333333333, ans=0.2
2023-11-22 00:27:07,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1722160.0, ans=0.125
2023-11-22 00:27:08,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=12.0
2023-11-22 00:27:13,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1722160.0, ans=0.2
2023-11-22 00:27:16,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1722160.0, ans=0.0
2023-11-22 00:27:26,502 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:27:33,614 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5850, loss[loss=0.07519, simple_loss=0.09583, pruned_loss=0.02005, audio_tagging_loss=0.007229, over 14942.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09461, pruned_loss=0.01579, audio_tagging_loss=0.00941, over 3042980.56 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:27:38,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258350
2023-11-22 00:27:38,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1722293.3333333333, ans=0.125
2023-11-22 00:27:45,663 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:27:45,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1722360.0, ans=0.125
2023-11-22 00:27:47,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1722360.0, ans=0.125
2023-11-22 00:28:11,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.44 vs. limit=10.0
2023-11-22 00:28:21,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1722493.3333333333, ans=0.0
2023-11-22 00:28:32,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1722560.0, ans=0.0
2023-11-22 00:28:37,513 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.44 vs. limit=15.0
2023-11-22 00:28:38,097 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5900, loss[loss=0.06451, simple_loss=0.0845, pruned_loss=0.01291, audio_tagging_loss=0.00935, over 14736.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09523, pruned_loss=0.01593, audio_tagging_loss=0.00944, over 3047026.21 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:28:43,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258400
2023-11-22 00:29:02,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=22.5
2023-11-22 00:29:05,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1722760.0, ans=0.125
2023-11-22 00:29:09,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1722760.0, ans=0.95
2023-11-22 00:29:10,521 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.273e+01 8.751e+01 9.457e+01 2.979e+02, threshold=1.750e+02, percent-clipped=1.0
2023-11-22 00:29:16,784 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:29:31,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1722893.3333333333, ans=0.125
2023-11-22 00:29:39,845 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:29:43,332 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 5950, loss[loss=0.05854, simple_loss=0.08355, pruned_loss=0.01021, audio_tagging_loss=0.006563, over 15611.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09412, pruned_loss=0.01553, audio_tagging_loss=0.009512, over 3040766.59 frames. ], batch size: 61, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:29:48,346 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258450
2023-11-22 00:29:52,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1722960.0, ans=10.0
2023-11-22 00:30:11,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1723093.3333333333, ans=0.125
2023-11-22 00:30:24,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1723160.0, ans=0.07
2023-11-22 00:30:46,931 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6000, loss[loss=0.0533, simple_loss=0.06798, pruned_loss=0.007717, audio_tagging_loss=0.0116, over 16834.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09481, pruned_loss=0.01582, audio_tagging_loss=0.009458, over 3047688.00 frames. ], batch size: 65, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:30:46,932 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 00:31:18,930 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2691, 4.1891, 4.4594, 4.4133], device='cuda:2')
2023-11-22 00:31:27,193 INFO [train_asr.py:1253] (2/4) Epoch 22, validation: loss=0.05958, simple_loss=0.05193, pruned_loss=0.005178, audio_tagging_loss=0.02843, over 4681554.00 frames. 
2023-11-22 00:31:27,194 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 00:31:28,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1723293.3333333333, ans=0.125
2023-11-22 00:31:31,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1723293.3333333333, ans=0.2
2023-11-22 00:31:32,812 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258500
2023-11-22 00:31:51,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1723426.6666666667, ans=0.125
2023-11-22 00:31:56,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1723426.6666666667, ans=0.1
2023-11-22 00:31:58,331 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.255e+01 8.695e+01 9.611e+01 1.390e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 00:32:03,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1723493.3333333333, ans=0.09899494936611666
2023-11-22 00:32:08,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1723493.3333333333, ans=0.1
2023-11-22 00:32:10,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1723493.3333333333, ans=0.0
2023-11-22 00:32:14,255 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:32:21,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.83 vs. limit=15.0
2023-11-22 00:32:22,568 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0
2023-11-22 00:32:24,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1723560.0, ans=0.1
2023-11-22 00:32:28,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1723560.0, ans=0.1
2023-11-22 00:32:31,172 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6050, loss[loss=0.06375, simple_loss=0.08956, pruned_loss=0.01229, audio_tagging_loss=0.006679, over 14278.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09319, pruned_loss=0.01545, audio_tagging_loss=0.009483, over 3051718.07 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:32:36,106 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258550
2023-11-22 00:32:46,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1723693.3333333333, ans=0.2
2023-11-22 00:33:03,527 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=12.0
2023-11-22 00:33:33,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1723960.0, ans=0.125
2023-11-22 00:33:34,428 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6100, loss[loss=0.09978, simple_loss=0.1382, pruned_loss=0.02399, audio_tagging_loss=0.006689, over 16337.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09447, pruned_loss=0.01569, audio_tagging_loss=0.009388, over 3042613.16 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:33:38,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1723960.0, ans=0.04949747468305833
2023-11-22 00:33:39,393 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258600
2023-11-22 00:34:06,999 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.187e+01 8.838e+01 9.553e+01 1.618e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 00:34:16,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1724160.0, ans=0.125
2023-11-22 00:34:39,401 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6150, loss[loss=0.09137, simple_loss=0.1208, pruned_loss=0.02317, audio_tagging_loss=0.007769, over 15452.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09414, pruned_loss=0.01562, audio_tagging_loss=0.009424, over 3041749.36 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:34:42,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1724293.3333333333, ans=10.0
2023-11-22 00:34:44,395 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258650
2023-11-22 00:35:03,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1724426.6666666667, ans=0.125
2023-11-22 00:35:04,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1724426.6666666667, ans=0.0
2023-11-22 00:35:22,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1724493.3333333333, ans=22.5
2023-11-22 00:35:24,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1724493.3333333333, ans=0.0
2023-11-22 00:35:30,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1724560.0, ans=0.0
2023-11-22 00:35:43,766 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6200, loss[loss=0.07207, simple_loss=0.09727, pruned_loss=0.01605, audio_tagging_loss=0.00738, over 15534.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09358, pruned_loss=0.01553, audio_tagging_loss=0.009474, over 3041487.69 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:35:48,875 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258700
2023-11-22 00:35:51,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1724626.6666666667, ans=0.2
2023-11-22 00:36:04,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1724693.3333333333, ans=0.0
2023-11-22 00:36:10,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1724760.0, ans=0.1
2023-11-22 00:36:15,753 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.055e+01 8.630e+01 9.332e+01 1.528e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 00:36:41,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1724893.3333333333, ans=0.125
2023-11-22 00:36:48,001 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6250, loss[loss=0.08284, simple_loss=0.1101, pruned_loss=0.01991, audio_tagging_loss=0.007885, over 16088.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09337, pruned_loss=0.01557, audio_tagging_loss=0.009526, over 3044355.23 frames. ], batch size: 61, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:36:51,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1724960.0, ans=0.125
2023-11-22 00:36:53,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258750
2023-11-22 00:37:11,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1725026.6666666667, ans=0.125
2023-11-22 00:37:12,844 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:37:28,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1725160.0, ans=0.2
2023-11-22 00:37:28,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1725160.0, ans=0.125
2023-11-22 00:37:31,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1725160.0, ans=0.125
2023-11-22 00:37:36,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1725160.0, ans=0.0
2023-11-22 00:37:39,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1725226.6666666667, ans=0.0
2023-11-22 00:37:52,623 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6300, loss[loss=0.08767, simple_loss=0.1208, pruned_loss=0.02034, audio_tagging_loss=0.006903, over 15067.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09316, pruned_loss=0.01557, audio_tagging_loss=0.009614, over 3044333.34 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:37:52,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1725293.3333333333, ans=0.125
2023-11-22 00:37:58,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258800
2023-11-22 00:38:02,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1725293.3333333333, ans=0.125
2023-11-22 00:38:16,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1725360.0, ans=0.125
2023-11-22 00:38:20,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1725426.6666666667, ans=0.1
2023-11-22 00:38:25,685 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.059e+01 8.458e+01 8.852e+01 9.578e+01 1.235e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 00:38:36,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.64 vs. limit=22.5
2023-11-22 00:38:54,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1725560.0, ans=0.0
2023-11-22 00:38:58,130 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6350, loss[loss=0.08003, simple_loss=0.1036, pruned_loss=0.01675, audio_tagging_loss=0.01149, over 15549.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09376, pruned_loss=0.01565, audio_tagging_loss=0.009637, over 3048501.70 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:39:03,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258850
2023-11-22 00:39:04,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1725626.6666666667, ans=0.125
2023-11-22 00:39:05,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1725626.6666666667, ans=0.0
2023-11-22 00:39:25,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1725760.0, ans=0.125
2023-11-22 00:39:29,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.00 vs. limit=15.0
2023-11-22 00:39:34,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1725760.0, ans=0.0
2023-11-22 00:39:37,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1725826.6666666667, ans=0.2
2023-11-22 00:39:39,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1725826.6666666667, ans=0.1
2023-11-22 00:39:49,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1725893.3333333333, ans=0.1
2023-11-22 00:39:59,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1725893.3333333333, ans=0.125
2023-11-22 00:40:00,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1725960.0, ans=0.125
2023-11-22 00:40:00,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.68 vs. limit=12.0
2023-11-22 00:40:01,453 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6400, loss[loss=0.07842, simple_loss=0.1047, pruned_loss=0.01896, audio_tagging_loss=0.007119, over 15334.00 frames. ], tot_loss[loss=0.07184, simple_loss=0.09302, pruned_loss=0.01559, audio_tagging_loss=0.009739, over 3041595.24 frames. ], batch size: 61, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:40:02,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1725960.0, ans=0.125
2023-11-22 00:40:07,029 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258900
2023-11-22 00:40:08,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1725960.0, ans=0.0
2023-11-22 00:40:34,299 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.092e+01 8.951e+01 9.607e+01 2.279e+02, threshold=1.790e+02, percent-clipped=1.0
2023-11-22 00:40:39,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1726160.0, ans=0.1
2023-11-22 00:40:41,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1726160.0, ans=0.0
2023-11-22 00:40:52,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1726226.6666666667, ans=0.09899494936611666
2023-11-22 00:41:05,261 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6450, loss[loss=0.08337, simple_loss=0.1081, pruned_loss=0.01797, audio_tagging_loss=0.01137, over 14510.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.0934, pruned_loss=0.01548, audio_tagging_loss=0.00973, over 3040350.01 frames. ], batch size: 54, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:41:11,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 258950
2023-11-22 00:41:15,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1726293.3333333333, ans=0.2
2023-11-22 00:41:18,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1726360.0, ans=0.035
2023-11-22 00:41:19,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1726360.0, ans=0.0
2023-11-22 00:41:33,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1726426.6666666667, ans=0.0
2023-11-22 00:41:35,097 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.00 vs. limit=10.0
2023-11-22 00:41:55,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1726560.0, ans=0.125
2023-11-22 00:41:57,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1726560.0, ans=0.125
2023-11-22 00:42:03,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1726560.0, ans=0.125
2023-11-22 00:42:08,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1726560.0, ans=0.125
2023-11-22 00:42:09,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1726626.6666666667, ans=0.0
2023-11-22 00:42:10,280 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6500, loss[loss=0.08249, simple_loss=0.1155, pruned_loss=0.01688, audio_tagging_loss=0.00788, over 15710.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.09256, pruned_loss=0.01539, audio_tagging_loss=0.009829, over 3033999.14 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:42:15,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259000
2023-11-22 00:42:18,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1726626.6666666667, ans=0.0
2023-11-22 00:42:20,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1726626.6666666667, ans=0.1
2023-11-22 00:42:32,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1726693.3333333333, ans=0.07
2023-11-22 00:42:37,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1726760.0, ans=0.125
2023-11-22 00:42:42,947 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.155e+01 8.144e+01 8.732e+01 9.307e+01 1.366e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 00:42:43,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1726760.0, ans=0.0
2023-11-22 00:42:57,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1726826.6666666667, ans=0.125
2023-11-22 00:43:12,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1726893.3333333333, ans=0.125
2023-11-22 00:43:15,079 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6550, loss[loss=0.05942, simple_loss=0.0792, pruned_loss=0.01175, audio_tagging_loss=0.008064, over 14350.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09436, pruned_loss=0.01583, audio_tagging_loss=0.009598, over 3043767.71 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:43:15,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.32 vs. limit=6.0
2023-11-22 00:43:20,268 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259050
2023-11-22 00:43:46,742 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.99 vs. limit=15.0
2023-11-22 00:44:01,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1727160.0, ans=0.0
2023-11-22 00:44:09,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1727226.6666666667, ans=0.125
2023-11-22 00:44:15,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1727226.6666666667, ans=0.0
2023-11-22 00:44:15,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.93 vs. limit=15.0
2023-11-22 00:44:18,546 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6600, loss[loss=0.08175, simple_loss=0.1104, pruned_loss=0.02055, audio_tagging_loss=0.006007, over 14371.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09456, pruned_loss=0.01581, audio_tagging_loss=0.009455, over 3042565.36 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:44:24,056 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259100
2023-11-22 00:44:52,443 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.221e+01 8.740e+01 9.559e+01 1.333e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 00:45:09,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1727560.0, ans=0.125
2023-11-22 00:45:23,534 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6650, loss[loss=0.07954, simple_loss=0.1079, pruned_loss=0.01494, audio_tagging_loss=0.01063, over 15968.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09457, pruned_loss=0.01564, audio_tagging_loss=0.009363, over 3039776.90 frames. ], batch size: 59, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:45:29,253 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259150
2023-11-22 00:45:47,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1727760.0, ans=0.1
2023-11-22 00:45:50,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1727760.0, ans=10.0
2023-11-22 00:45:51,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1727760.0, ans=0.07
2023-11-22 00:46:06,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1727826.6666666667, ans=0.125
2023-11-22 00:46:15,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1727893.3333333333, ans=0.2
2023-11-22 00:46:16,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1727893.3333333333, ans=0.125
2023-11-22 00:46:23,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1727893.3333333333, ans=0.025
2023-11-22 00:46:27,267 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6700, loss[loss=0.07919, simple_loss=0.1086, pruned_loss=0.01703, audio_tagging_loss=0.007848, over 14663.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09535, pruned_loss=0.01585, audio_tagging_loss=0.009197, over 3036933.64 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 8.0
2023-11-22 00:46:32,147 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259200
2023-11-22 00:46:42,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1728026.6666666667, ans=0.0
2023-11-22 00:46:58,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1728093.3333333333, ans=0.125
2023-11-22 00:47:01,890 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 7.937e+01 8.531e+01 9.262e+01 1.112e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-22 00:47:05,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1728160.0, ans=0.0
2023-11-22 00:47:08,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1728160.0, ans=0.0
2023-11-22 00:47:30,780 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6750, loss[loss=0.0715, simple_loss=0.08593, pruned_loss=0.01801, audio_tagging_loss=0.01052, over 14649.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09578, pruned_loss=0.01603, audio_tagging_loss=0.009157, over 3041756.23 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 8.0
2023-11-22 00:47:32,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1728293.3333333333, ans=0.0
2023-11-22 00:47:35,701 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259250
2023-11-22 00:47:36,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.10 vs. limit=15.0
2023-11-22 00:47:48,086 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.84 vs. limit=22.5
2023-11-22 00:47:48,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1728360.0, ans=0.125
2023-11-22 00:48:00,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1728426.6666666667, ans=0.0
2023-11-22 00:48:10,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1728493.3333333333, ans=0.125
2023-11-22 00:48:17,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1728493.3333333333, ans=0.04949747468305833
2023-11-22 00:48:26,385 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.93 vs. limit=15.0
2023-11-22 00:48:31,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1728560.0, ans=0.1
2023-11-22 00:48:31,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1728560.0, ans=0.1
2023-11-22 00:48:36,234 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6800, loss[loss=0.07907, simple_loss=0.09841, pruned_loss=0.01887, audio_tagging_loss=0.011, over 14807.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09439, pruned_loss=0.01575, audio_tagging_loss=0.009301, over 3030438.61 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:48:41,213 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259300
2023-11-22 00:48:46,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1728626.6666666667, ans=0.07
2023-11-22 00:48:51,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1728693.3333333333, ans=0.0
2023-11-22 00:48:56,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1728693.3333333333, ans=0.125
2023-11-22 00:48:57,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1728693.3333333333, ans=0.125
2023-11-22 00:49:09,126 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.02 vs. limit=15.0
2023-11-22 00:49:09,689 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.092e+01 8.012e+01 8.646e+01 9.288e+01 1.206e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 00:49:11,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1728760.0, ans=0.125
2023-11-22 00:49:12,800 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.99 vs. limit=15.0
2023-11-22 00:49:19,924 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:49:36,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1728893.3333333333, ans=0.0
2023-11-22 00:49:37,332 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.46 vs. limit=15.0
2023-11-22 00:49:40,205 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6850, loss[loss=0.03307, simple_loss=0.03873, pruned_loss=0.004422, audio_tagging_loss=0.009286, over 15507.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09442, pruned_loss=0.01567, audio_tagging_loss=0.009287, over 3037577.52 frames. ], batch size: 61, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:49:40,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1728960.0, ans=0.125
2023-11-22 00:49:45,205 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259350
2023-11-22 00:49:46,765 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:50:05,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1729093.3333333333, ans=0.125
2023-11-22 00:50:12,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1729093.3333333333, ans=0.125
2023-11-22 00:50:13,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1729093.3333333333, ans=0.125
2023-11-22 00:50:15,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1729093.3333333333, ans=0.125
2023-11-22 00:50:17,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1729160.0, ans=0.0
2023-11-22 00:50:20,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1729160.0, ans=0.1
2023-11-22 00:50:36,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1729226.6666666667, ans=0.05
2023-11-22 00:50:40,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1729226.6666666667, ans=0.025
2023-11-22 00:50:43,774 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6900, loss[loss=0.08965, simple_loss=0.131, pruned_loss=0.01798, audio_tagging_loss=0.006168, over 16129.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09461, pruned_loss=0.01564, audio_tagging_loss=0.009221, over 3041066.77 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:50:48,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259400
2023-11-22 00:51:07,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1729360.0, ans=0.125
2023-11-22 00:51:13,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1729426.6666666667, ans=0.125
2023-11-22 00:51:19,569 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.107e+01 8.289e+01 8.755e+01 9.420e+01 1.385e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 00:51:21,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1729426.6666666667, ans=0.1
2023-11-22 00:51:25,260 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.26 vs. limit=15.0
2023-11-22 00:51:35,567 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:51:39,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1729560.0, ans=0.125
2023-11-22 00:51:42,402 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.90 vs. limit=15.0
2023-11-22 00:51:49,207 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 6950, loss[loss=0.06719, simple_loss=0.09279, pruned_loss=0.01285, audio_tagging_loss=0.007946, over 14998.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09473, pruned_loss=0.0157, audio_tagging_loss=0.009219, over 3041115.43 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:51:54,327 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259450
2023-11-22 00:52:09,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1729693.3333333333, ans=0.0
2023-11-22 00:52:31,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1729826.6666666667, ans=0.125
2023-11-22 00:52:31,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1729826.6666666667, ans=0.0
2023-11-22 00:52:41,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1729893.3333333333, ans=0.0
2023-11-22 00:52:53,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1729960.0, ans=0.1
2023-11-22 00:52:54,405 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7000, loss[loss=0.0901, simple_loss=0.1208, pruned_loss=0.02398, audio_tagging_loss=0.005737, over 15875.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09509, pruned_loss=0.0158, audio_tagging_loss=0.009309, over 3043954.89 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:52:55,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1729960.0, ans=0.125
2023-11-22 00:52:59,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259500
2023-11-22 00:53:00,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1729960.0, ans=0.0
2023-11-22 00:53:02,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1729960.0, ans=0.125
2023-11-22 00:53:08,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1730026.6666666667, ans=0.125
2023-11-22 00:53:10,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1730026.6666666667, ans=0.125
2023-11-22 00:53:10,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1730026.6666666667, ans=0.0
2023-11-22 00:53:22,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.31 vs. limit=15.0
2023-11-22 00:53:28,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.29 vs. limit=12.0
2023-11-22 00:53:28,626 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 7.948e+01 8.560e+01 9.274e+01 1.120e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-22 00:53:58,421 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7050, loss[loss=0.06868, simple_loss=0.08695, pruned_loss=0.01571, audio_tagging_loss=0.009487, over 15102.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09483, pruned_loss=0.01583, audio_tagging_loss=0.009458, over 3039876.18 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:54:02,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1730293.3333333333, ans=0.0
2023-11-22 00:54:03,467 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259550
2023-11-22 00:54:08,919 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.92 vs. limit=6.0
2023-11-22 00:54:14,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1730360.0, ans=0.0
2023-11-22 00:54:21,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1730360.0, ans=0.2
2023-11-22 00:54:29,129 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.55 vs. limit=15.0
2023-11-22 00:54:30,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1730426.6666666667, ans=0.0
2023-11-22 00:54:36,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1730493.3333333333, ans=0.125
2023-11-22 00:54:36,545 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0
2023-11-22 00:54:45,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0
2023-11-22 00:55:02,338 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7100, loss[loss=0.06519, simple_loss=0.08062, pruned_loss=0.01401, audio_tagging_loss=0.01086, over 15593.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09414, pruned_loss=0.01561, audio_tagging_loss=0.009589, over 3036453.11 frames. ], batch size: 61, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:55:02,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1730626.6666666667, ans=0.0
2023-11-22 00:55:07,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259600
2023-11-22 00:55:16,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1730693.3333333333, ans=0.0
2023-11-22 00:55:19,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1730693.3333333333, ans=0.0
2023-11-22 00:55:37,156 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.168e+01 8.717e+01 9.423e+01 1.311e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 00:55:40,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.98 vs. limit=6.0
2023-11-22 00:55:57,629 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=15.0
2023-11-22 00:56:07,818 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7150, loss[loss=0.06795, simple_loss=0.09155, pruned_loss=0.01366, audio_tagging_loss=0.008515, over 15347.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09556, pruned_loss=0.01584, audio_tagging_loss=0.009589, over 3044417.27 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:56:12,780 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259650
2023-11-22 00:56:25,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1731026.6666666667, ans=0.125
2023-11-22 00:56:39,955 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.41 vs. limit=22.5
2023-11-22 00:56:42,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1731093.3333333333, ans=0.125
2023-11-22 00:56:43,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1731093.3333333333, ans=0.0
2023-11-22 00:56:43,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1731093.3333333333, ans=0.2
2023-11-22 00:56:52,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.87 vs. limit=12.0
2023-11-22 00:57:03,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1731226.6666666667, ans=0.125
2023-11-22 00:57:11,640 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7200, loss[loss=0.05375, simple_loss=0.07381, pruned_loss=0.007862, audio_tagging_loss=0.008985, over 14857.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09559, pruned_loss=0.01571, audio_tagging_loss=0.009582, over 3044804.88 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:57:14,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1731293.3333333333, ans=0.125
2023-11-22 00:57:16,560 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259700
2023-11-22 00:57:21,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1731293.3333333333, ans=0.125
2023-11-22 00:57:25,369 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:57:42,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1731426.6666666667, ans=0.125
2023-11-22 00:57:45,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1731426.6666666667, ans=0.0
2023-11-22 00:57:46,563 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 8.081e+01 8.874e+01 9.710e+01 1.298e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 00:57:47,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1731426.6666666667, ans=0.1
2023-11-22 00:58:12,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1731560.0, ans=0.1
2023-11-22 00:58:15,111 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7250, loss[loss=0.06834, simple_loss=0.08719, pruned_loss=0.01263, audio_tagging_loss=0.01211, over 14992.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09542, pruned_loss=0.01558, audio_tagging_loss=0.009659, over 3038277.65 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:58:21,207 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259750
2023-11-22 00:58:51,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1731760.0, ans=0.125
2023-11-22 00:58:52,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1731826.6666666667, ans=0.0
2023-11-22 00:59:04,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1731893.3333333333, ans=0.1
2023-11-22 00:59:08,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1731893.3333333333, ans=0.0
2023-11-22 00:59:13,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1731893.3333333333, ans=0.125
2023-11-22 00:59:19,754 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7300, loss[loss=0.07184, simple_loss=0.09443, pruned_loss=0.01587, audio_tagging_loss=0.008758, over 15813.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09434, pruned_loss=0.01547, audio_tagging_loss=0.009569, over 3038816.31 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:59:24,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259800
2023-11-22 00:59:34,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1732026.6666666667, ans=0.125
2023-11-22 00:59:42,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1732026.6666666667, ans=0.07
2023-11-22 00:59:46,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1732093.3333333333, ans=0.0
2023-11-22 00:59:53,085 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.552e+01 8.104e+01 8.588e+01 9.277e+01 1.159e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 00:59:58,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1732160.0, ans=0.125
2023-11-22 01:00:01,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1732160.0, ans=0.07
2023-11-22 01:00:12,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1732226.6666666667, ans=0.0
2023-11-22 01:00:13,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1732226.6666666667, ans=0.0
2023-11-22 01:00:20,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1732226.6666666667, ans=0.0
2023-11-22 01:00:23,337 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7350, loss[loss=0.0746, simple_loss=0.1087, pruned_loss=0.01269, audio_tagging_loss=0.007557, over 15318.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09483, pruned_loss=0.01556, audio_tagging_loss=0.0094, over 3040487.22 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:00:28,268 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259850
2023-11-22 01:00:29,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1732293.3333333333, ans=0.1
2023-11-22 01:00:36,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1732360.0, ans=0.125
2023-11-22 01:01:06,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1732493.3333333333, ans=0.125
2023-11-22 01:01:08,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1732493.3333333333, ans=0.125
2023-11-22 01:01:21,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1732560.0, ans=0.125
2023-11-22 01:01:26,429 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7400, loss[loss=0.08458, simple_loss=0.1102, pruned_loss=0.01909, audio_tagging_loss=0.0104, over 14493.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09465, pruned_loss=0.01542, audio_tagging_loss=0.009188, over 3035600.83 frames. ], batch size: 52, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:01:26,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1732626.6666666667, ans=0.125
2023-11-22 01:01:32,006 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259900
2023-11-22 01:01:53,174 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.13 vs. limit=15.0
2023-11-22 01:01:59,872 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.25 vs. limit=15.0
2023-11-22 01:02:01,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1732760.0, ans=0.125
2023-11-22 01:02:02,679 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.170e+01 8.751e+01 9.458e+01 1.086e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 01:02:07,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1732826.6666666667, ans=0.0
2023-11-22 01:02:31,022 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7450, loss[loss=0.06407, simple_loss=0.07736, pruned_loss=0.01343, audio_tagging_loss=0.01196, over 15326.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09491, pruned_loss=0.01553, audio_tagging_loss=0.009156, over 3045019.03 frames. ], batch size: 59, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:02:31,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1732960.0, ans=0.125
2023-11-22 01:02:36,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 259950
2023-11-22 01:02:57,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1733093.3333333333, ans=0.125
2023-11-22 01:03:16,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=15.0
2023-11-22 01:03:18,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1733160.0, ans=0.125
2023-11-22 01:03:35,339 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7500, loss[loss=0.0658, simple_loss=0.08032, pruned_loss=0.01374, audio_tagging_loss=0.0119, over 13757.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09428, pruned_loss=0.01549, audio_tagging_loss=0.009268, over 3046707.48 frames. ], batch size: 53, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:03:40,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260000
2023-11-22 01:03:42,622 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.71 vs. limit=15.0
2023-11-22 01:03:48,603 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.88 vs. limit=15.0
2023-11-22 01:04:08,817 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.65 vs. limit=15.0
2023-11-22 01:04:14,749 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.000e+01 8.690e+01 9.325e+01 1.229e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 01:04:36,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1733560.0, ans=0.1
2023-11-22 01:04:41,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1733626.6666666667, ans=0.1
2023-11-22 01:04:42,067 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7550, loss[loss=0.06552, simple_loss=0.08331, pruned_loss=0.01495, audio_tagging_loss=0.008916, over 15116.00 frames. ], tot_loss[loss=0.07184, simple_loss=0.09409, pruned_loss=0.01552, audio_tagging_loss=0.009273, over 3048413.10 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:04:42,465 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:04:43,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1733626.6666666667, ans=0.125
2023-11-22 01:04:47,122 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260050
2023-11-22 01:05:33,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1733893.3333333333, ans=0.125
2023-11-22 01:05:45,950 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7600, loss[loss=0.06602, simple_loss=0.08146, pruned_loss=0.01413, audio_tagging_loss=0.01117, over 14646.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09448, pruned_loss=0.01557, audio_tagging_loss=0.009213, over 3054912.69 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:05:51,503 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260100
2023-11-22 01:06:13,993 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.25 vs. limit=5.0
2023-11-22 01:06:17,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1734093.3333333333, ans=0.125
2023-11-22 01:06:18,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1734093.3333333333, ans=0.0
2023-11-22 01:06:21,716 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.674e+01 8.289e+01 8.897e+01 9.780e+01 1.167e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 01:06:22,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1734093.3333333333, ans=0.0
2023-11-22 01:06:33,742 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:06:34,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1734160.0, ans=0.125
2023-11-22 01:06:49,638 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7650, loss[loss=0.06318, simple_loss=0.08497, pruned_loss=0.01128, audio_tagging_loss=0.00942, over 16314.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09445, pruned_loss=0.01563, audio_tagging_loss=0.009235, over 3051252.48 frames. ], batch size: 61, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:06:50,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.28 vs. limit=15.0
2023-11-22 01:06:54,494 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260150
2023-11-22 01:06:57,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1734293.3333333333, ans=0.2
2023-11-22 01:07:11,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=1734360.0, ans=0.1
2023-11-22 01:07:23,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=15.0
2023-11-22 01:07:31,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1734493.3333333333, ans=0.0
2023-11-22 01:07:40,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1734560.0, ans=0.1
2023-11-22 01:07:45,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.59 vs. limit=15.0
2023-11-22 01:07:53,008 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7700, loss[loss=0.06762, simple_loss=0.08853, pruned_loss=0.01369, audio_tagging_loss=0.009667, over 16268.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09509, pruned_loss=0.01556, audio_tagging_loss=0.00923, over 3054061.05 frames. ], batch size: 64, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:07:57,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260200
2023-11-22 01:08:02,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1734626.6666666667, ans=0.125
2023-11-22 01:08:16,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1734693.3333333333, ans=0.2
2023-11-22 01:08:18,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1734760.0, ans=0.07
2023-11-22 01:08:23,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1734760.0, ans=0.125
2023-11-22 01:08:28,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1734760.0, ans=0.125
2023-11-22 01:08:29,047 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 7.976e+01 8.890e+01 9.341e+01 1.175e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 01:08:35,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1734826.6666666667, ans=0.0
2023-11-22 01:08:47,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1734893.3333333333, ans=10.0
2023-11-22 01:08:52,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.68 vs. limit=15.0
2023-11-22 01:08:57,778 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7750, loss[loss=0.07384, simple_loss=0.09439, pruned_loss=0.01621, audio_tagging_loss=0.01044, over 16125.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09556, pruned_loss=0.01557, audio_tagging_loss=0.009262, over 3052308.99 frames. ], batch size: 62, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:09:02,706 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260250
2023-11-22 01:09:06,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1734960.0, ans=0.125
2023-11-22 01:09:07,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1734960.0, ans=0.1
2023-11-22 01:09:38,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1735160.0, ans=0.1
2023-11-22 01:09:47,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1735226.6666666667, ans=0.09899494936611666
2023-11-22 01:10:01,240 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7800, loss[loss=0.08158, simple_loss=0.108, pruned_loss=0.02064, audio_tagging_loss=0.006958, over 15821.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09537, pruned_loss=0.01568, audio_tagging_loss=0.009242, over 3043607.77 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:10:06,224 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260300
2023-11-22 01:10:12,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1735360.0, ans=0.125
2023-11-22 01:10:32,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1735426.6666666667, ans=0.0
2023-11-22 01:10:32,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.66 vs. limit=10.0
2023-11-22 01:10:37,020 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 8.146e+01 9.007e+01 9.749e+01 1.228e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 01:10:43,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=12.0
2023-11-22 01:10:47,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1735493.3333333333, ans=0.125
2023-11-22 01:11:04,751 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7850, loss[loss=0.08134, simple_loss=0.09751, pruned_loss=0.02056, audio_tagging_loss=0.01202, over 14639.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09435, pruned_loss=0.01547, audio_tagging_loss=0.009325, over 3036716.46 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:11:04,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1735626.6666666667, ans=0.1
2023-11-22 01:11:09,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260350
2023-11-22 01:11:39,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1735760.0, ans=0.2
2023-11-22 01:11:54,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.98 vs. limit=15.0
2023-11-22 01:11:58,670 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:12:09,849 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7900, loss[loss=0.06232, simple_loss=0.07506, pruned_loss=0.01511, audio_tagging_loss=0.009679, over 16275.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09365, pruned_loss=0.01546, audio_tagging_loss=0.009467, over 3041641.90 frames. ], batch size: 64, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:12:14,943 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260400
2023-11-22 01:12:29,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1736026.6666666667, ans=0.125
2023-11-22 01:12:32,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1736026.6666666667, ans=0.1
2023-11-22 01:12:46,404 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.401e+01 8.796e+01 9.438e+01 1.194e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 01:12:58,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.52 vs. limit=22.5
2023-11-22 01:13:05,183 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-22 01:13:05,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.02 vs. limit=15.0
2023-11-22 01:13:13,850 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 7950, loss[loss=0.09108, simple_loss=0.1287, pruned_loss=0.02016, audio_tagging_loss=0.006552, over 14668.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.0944, pruned_loss=0.01577, audio_tagging_loss=0.009497, over 3046214.32 frames. ], batch size: 53, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:13:18,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260450
2023-11-22 01:13:23,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1736293.3333333333, ans=0.125
2023-11-22 01:13:29,889 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:13:35,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1736360.0, ans=0.0
2023-11-22 01:14:01,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1736493.3333333333, ans=0.1
2023-11-22 01:14:16,707 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8000, loss[loss=0.06545, simple_loss=0.08439, pruned_loss=0.01343, audio_tagging_loss=0.009829, over 15641.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09379, pruned_loss=0.01563, audio_tagging_loss=0.009588, over 3044480.18 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:14:17,276 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.91 vs. limit=22.5
2023-11-22 01:14:20,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1736626.6666666667, ans=0.2
2023-11-22 01:14:21,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260500
2023-11-22 01:14:25,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1736626.6666666667, ans=0.125
2023-11-22 01:14:38,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1736693.3333333333, ans=0.125
2023-11-22 01:14:38,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1736693.3333333333, ans=0.125
2023-11-22 01:14:42,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1736760.0, ans=0.0
2023-11-22 01:14:46,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.63 vs. limit=6.0
2023-11-22 01:14:54,343 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 7.877e+01 8.568e+01 9.236e+01 1.291e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-22 01:14:58,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1736826.6666666667, ans=0.95
2023-11-22 01:14:58,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1736826.6666666667, ans=0.1
2023-11-22 01:15:14,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1736893.3333333333, ans=0.125
2023-11-22 01:15:20,861 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8050, loss[loss=0.09289, simple_loss=0.1265, pruned_loss=0.02345, audio_tagging_loss=0.006213, over 15056.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09424, pruned_loss=0.01581, audio_tagging_loss=0.009663, over 3045871.20 frames. ], batch size: 53, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:15:26,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260550
2023-11-22 01:15:37,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1737026.6666666667, ans=0.125
2023-11-22 01:15:50,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1737093.3333333333, ans=0.2
2023-11-22 01:15:53,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1737093.3333333333, ans=0.2
2023-11-22 01:16:02,402 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.76 vs. limit=22.5
2023-11-22 01:16:03,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.98 vs. limit=10.0
2023-11-22 01:16:26,004 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8100, loss[loss=0.06546, simple_loss=0.07715, pruned_loss=0.01437, audio_tagging_loss=0.01251, over 15271.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.0943, pruned_loss=0.01586, audio_tagging_loss=0.00967, over 3044390.24 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:16:30,875 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260600
2023-11-22 01:16:47,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1737360.0, ans=0.125
2023-11-22 01:16:47,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1737360.0, ans=0.0
2023-11-22 01:17:04,365 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.436e+01 7.880e+01 8.583e+01 9.266e+01 1.211e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-22 01:17:19,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1737560.0, ans=0.125
2023-11-22 01:17:21,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1737560.0, ans=0.125
2023-11-22 01:17:29,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1737626.6666666667, ans=0.125
2023-11-22 01:17:30,010 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8150, loss[loss=0.09734, simple_loss=0.1398, pruned_loss=0.02161, audio_tagging_loss=0.005804, over 15761.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09466, pruned_loss=0.01575, audio_tagging_loss=0.009524, over 3043489.66 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:17:30,254 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:17:34,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260650
2023-11-22 01:18:09,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1737826.6666666667, ans=0.125
2023-11-22 01:18:33,272 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8200, loss[loss=0.09431, simple_loss=0.1324, pruned_loss=0.01956, audio_tagging_loss=0.008575, over 16722.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09557, pruned_loss=0.01578, audio_tagging_loss=0.009293, over 3042855.33 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:18:36,337 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:18:36,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1737960.0, ans=0.07
2023-11-22 01:18:39,507 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260700
2023-11-22 01:18:44,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1737960.0, ans=0.125
2023-11-22 01:18:50,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1738026.6666666667, ans=0.0
2023-11-22 01:19:00,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1738093.3333333333, ans=0.0
2023-11-22 01:19:06,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1738093.3333333333, ans=0.125
2023-11-22 01:19:10,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1738093.3333333333, ans=0.0
2023-11-22 01:19:10,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.93 vs. limit=15.0
2023-11-22 01:19:12,436 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.958e+01 8.128e+01 8.787e+01 9.499e+01 1.177e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 01:19:19,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.41 vs. limit=10.0
2023-11-22 01:19:38,558 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8250, loss[loss=0.06996, simple_loss=0.09329, pruned_loss=0.01406, audio_tagging_loss=0.009256, over 15700.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09536, pruned_loss=0.01571, audio_tagging_loss=0.009163, over 3044438.92 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:19:43,539 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260750
2023-11-22 01:19:56,147 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.71 vs. limit=15.0
2023-11-22 01:20:16,701 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.19 vs. limit=6.0
2023-11-22 01:20:41,965 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8300, loss[loss=0.07481, simple_loss=0.09581, pruned_loss=0.0163, audio_tagging_loss=0.0106, over 15345.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09479, pruned_loss=0.01563, audio_tagging_loss=0.009119, over 3048717.74 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:20:46,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260800
2023-11-22 01:20:56,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1738693.3333333333, ans=0.95
2023-11-22 01:20:56,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1738693.3333333333, ans=0.1
2023-11-22 01:21:01,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1738693.3333333333, ans=0.0
2023-11-22 01:21:02,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1738693.3333333333, ans=0.125
2023-11-22 01:21:07,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1738760.0, ans=0.0
2023-11-22 01:21:21,281 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.003e+01 8.893e+01 9.644e+01 1.813e+02, threshold=1.779e+02, percent-clipped=1.0
2023-11-22 01:21:32,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1738893.3333333333, ans=0.125
2023-11-22 01:21:46,176 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8350, loss[loss=0.04686, simple_loss=0.0596, pruned_loss=0.007809, audio_tagging_loss=0.009249, over 15623.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09549, pruned_loss=0.01567, audio_tagging_loss=0.009109, over 3057391.22 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:21:51,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260850
2023-11-22 01:22:03,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1739026.6666666667, ans=0.0
2023-11-22 01:22:03,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1739026.6666666667, ans=0.0
2023-11-22 01:22:10,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1739026.6666666667, ans=0.05
2023-11-22 01:22:20,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1739093.3333333333, ans=0.1
2023-11-22 01:22:22,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1739093.3333333333, ans=0.0
2023-11-22 01:22:24,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.93 vs. limit=15.0
2023-11-22 01:22:30,794 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.28 vs. limit=10.0
2023-11-22 01:22:51,175 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8400, loss[loss=0.0869, simple_loss=0.1092, pruned_loss=0.02505, audio_tagging_loss=0.007232, over 15398.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.0937, pruned_loss=0.01537, audio_tagging_loss=0.009196, over 3058694.60 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:22:56,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260900
2023-11-22 01:23:09,565 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.10 vs. limit=15.0
2023-11-22 01:23:19,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1739426.6666666667, ans=0.1
2023-11-22 01:23:19,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1739426.6666666667, ans=0.1
2023-11-22 01:23:27,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1739493.3333333333, ans=0.125
2023-11-22 01:23:28,559 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 7.994e+01 8.596e+01 9.193e+01 1.208e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-22 01:23:38,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1739493.3333333333, ans=0.1
2023-11-22 01:23:39,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1739493.3333333333, ans=0.0
2023-11-22 01:23:41,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1739560.0, ans=0.2
2023-11-22 01:23:50,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1739560.0, ans=0.0
2023-11-22 01:23:50,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.40 vs. limit=22.5
2023-11-22 01:23:54,875 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8450, loss[loss=0.08481, simple_loss=0.1177, pruned_loss=0.01878, audio_tagging_loss=0.007195, over 15973.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09423, pruned_loss=0.01547, audio_tagging_loss=0.009117, over 3059255.65 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:23:59,805 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 260950
2023-11-22 01:24:28,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1739760.0, ans=0.04949747468305833
2023-11-22 01:24:33,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1739826.6666666667, ans=0.125
2023-11-22 01:24:36,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1739826.6666666667, ans=0.125
2023-11-22 01:24:58,322 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8500, loss[loss=0.08449, simple_loss=0.1114, pruned_loss=0.01724, audio_tagging_loss=0.01157, over 15606.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09483, pruned_loss=0.01577, audio_tagging_loss=0.00929, over 3055444.92 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:24:59,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1739960.0, ans=0.125
2023-11-22 01:25:01,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1739960.0, ans=0.2
2023-11-22 01:25:03,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261000
2023-11-22 01:25:16,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1740026.6666666667, ans=0.0
2023-11-22 01:25:36,958 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 8.117e+01 8.646e+01 9.659e+01 1.248e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 01:25:54,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1740226.6666666667, ans=0.2
2023-11-22 01:26:02,750 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8550, loss[loss=0.06978, simple_loss=0.09922, pruned_loss=0.01084, audio_tagging_loss=0.009336, over 16596.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09513, pruned_loss=0.01581, audio_tagging_loss=0.009319, over 3054699.75 frames. ], batch size: 61, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:26:08,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261050
2023-11-22 01:26:27,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1740426.6666666667, ans=0.125
2023-11-22 01:26:51,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1740493.3333333333, ans=0.05
2023-11-22 01:26:54,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1740560.0, ans=0.09899494936611666
2023-11-22 01:26:55,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.20 vs. limit=15.0
2023-11-22 01:27:06,905 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8600, loss[loss=0.07794, simple_loss=0.09467, pruned_loss=0.01994, audio_tagging_loss=0.01067, over 15247.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09486, pruned_loss=0.01583, audio_tagging_loss=0.009392, over 3049544.45 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:27:09,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1740626.6666666667, ans=0.0
2023-11-22 01:27:12,467 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261100
2023-11-22 01:27:34,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1740760.0, ans=0.2
2023-11-22 01:27:45,977 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.115e+01 8.738e+01 9.336e+01 1.223e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 01:27:49,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.09 vs. limit=15.0
2023-11-22 01:27:51,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1740826.6666666667, ans=0.0
2023-11-22 01:27:52,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1740826.6666666667, ans=0.2
2023-11-22 01:27:57,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1740893.3333333333, ans=0.0
2023-11-22 01:28:06,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1740893.3333333333, ans=0.125
2023-11-22 01:28:09,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1740960.0, ans=0.1
2023-11-22 01:28:09,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1740960.0, ans=0.1
2023-11-22 01:28:10,859 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8650, loss[loss=0.08242, simple_loss=0.1202, pruned_loss=0.01493, audio_tagging_loss=0.007389, over 15154.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09442, pruned_loss=0.01574, audio_tagging_loss=0.009434, over 3048904.84 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:28:15,790 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261150
2023-11-22 01:28:26,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1741026.6666666667, ans=0.1
2023-11-22 01:28:28,565 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.01 vs. limit=22.5
2023-11-22 01:28:36,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1741093.3333333333, ans=0.07
2023-11-22 01:28:37,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1741093.3333333333, ans=0.0
2023-11-22 01:29:02,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1741226.6666666667, ans=0.125
2023-11-22 01:29:09,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1741226.6666666667, ans=0.0
2023-11-22 01:29:15,060 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8700, loss[loss=0.09961, simple_loss=0.1349, pruned_loss=0.02239, audio_tagging_loss=0.009778, over 15034.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.0946, pruned_loss=0.01578, audio_tagging_loss=0.009528, over 3051373.59 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:29:16,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1741293.3333333333, ans=0.0
2023-11-22 01:29:16,611 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:29:20,602 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261200
2023-11-22 01:29:23,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1741293.3333333333, ans=0.125
2023-11-22 01:29:25,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1741293.3333333333, ans=10.0
2023-11-22 01:29:54,201 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.323e+01 9.034e+01 9.944e+01 3.984e+02, threshold=1.807e+02, percent-clipped=2.0
2023-11-22 01:29:55,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1741493.3333333333, ans=0.0
2023-11-22 01:29:57,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1741493.3333333333, ans=0.0
2023-11-22 01:30:13,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1741560.0, ans=0.125
2023-11-22 01:30:14,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1741560.0, ans=0.2
2023-11-22 01:30:19,272 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8750, loss[loss=0.05423, simple_loss=0.06446, pruned_loss=0.009539, audio_tagging_loss=0.01246, over 13724.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09423, pruned_loss=0.01567, audio_tagging_loss=0.009622, over 3049400.71 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:30:24,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261250
2023-11-22 01:30:24,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.11 vs. limit=15.0
2023-11-22 01:30:30,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1741693.3333333333, ans=0.125
2023-11-22 01:31:03,852 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.09 vs. limit=15.0
2023-11-22 01:31:19,894 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.89 vs. limit=22.5
2023-11-22 01:31:23,451 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8800, loss[loss=0.07276, simple_loss=0.09216, pruned_loss=0.01616, audio_tagging_loss=0.01052, over 14710.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09562, pruned_loss=0.01599, audio_tagging_loss=0.009662, over 3048394.72 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:31:28,358 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261300
2023-11-22 01:32:00,364 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.49 vs. limit=22.5
2023-11-22 01:32:03,193 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.567e+01 8.285e+01 8.942e+01 9.871e+01 1.431e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 01:32:04,930 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.48 vs. limit=15.0
2023-11-22 01:32:21,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1742226.6666666667, ans=0.125
2023-11-22 01:32:28,146 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8850, loss[loss=0.0571, simple_loss=0.07329, pruned_loss=0.01307, audio_tagging_loss=0.007383, over 14852.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09575, pruned_loss=0.01596, audio_tagging_loss=0.009672, over 3049008.93 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:32:33,064 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261350
2023-11-22 01:32:42,105 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:32:43,795 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.89 vs. limit=12.0
2023-11-22 01:33:03,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1742426.6666666667, ans=0.0
2023-11-22 01:33:03,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.66 vs. limit=15.0
2023-11-22 01:33:08,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1742493.3333333333, ans=0.1
2023-11-22 01:33:11,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1742493.3333333333, ans=0.125
2023-11-22 01:33:12,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.48 vs. limit=22.5
2023-11-22 01:33:28,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1742560.0, ans=0.2
2023-11-22 01:33:31,958 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8900, loss[loss=0.08779, simple_loss=0.1096, pruned_loss=0.02457, audio_tagging_loss=0.008438, over 14878.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.0961, pruned_loss=0.0161, audio_tagging_loss=0.009588, over 3051052.40 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:33:36,917 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261400
2023-11-22 01:33:41,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-22 01:34:13,145 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.357e+01 8.809e+01 9.586e+01 1.233e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 01:34:14,965 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.82 vs. limit=12.0
2023-11-22 01:34:17,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1742826.6666666667, ans=0.125
2023-11-22 01:34:17,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-22 01:34:22,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.06 vs. limit=15.0
2023-11-22 01:34:32,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1742893.3333333333, ans=0.1
2023-11-22 01:34:35,048 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 8950, loss[loss=0.07973, simple_loss=0.1064, pruned_loss=0.01943, audio_tagging_loss=0.00709, over 15334.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09689, pruned_loss=0.01619, audio_tagging_loss=0.009421, over 3049977.26 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:34:37,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1742960.0, ans=0.1
2023-11-22 01:34:40,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261450
2023-11-22 01:34:54,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1743026.6666666667, ans=0.1
2023-11-22 01:35:02,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1743093.3333333333, ans=0.0
2023-11-22 01:35:13,494 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.59 vs. limit=15.0
2023-11-22 01:35:38,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1743293.3333333333, ans=0.125
2023-11-22 01:35:39,542 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9000, loss[loss=0.1053, simple_loss=0.1414, pruned_loss=0.02879, audio_tagging_loss=0.005764, over 15671.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09716, pruned_loss=0.01635, audio_tagging_loss=0.009263, over 3050394.17 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:35:39,543 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 01:36:14,542 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.5920, 3.5720, 3.8566, 3.3096], device='cuda:2')
2023-11-22 01:36:17,277 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7738, 5.8551, 5.8913, 5.9030], device='cuda:2')
2023-11-22 01:36:20,057 INFO [train_asr.py:1253] (2/4) Epoch 22, validation: loss=0.0605, simple_loss=0.05183, pruned_loss=0.005175, audio_tagging_loss=0.02941, over 4681554.00 frames. 
2023-11-22 01:36:20,057 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 01:36:21,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1743293.3333333333, ans=0.1
2023-11-22 01:36:24,862 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261500
2023-11-22 01:36:25,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1743293.3333333333, ans=0.125
2023-11-22 01:36:28,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1743293.3333333333, ans=0.2
2023-11-22 01:36:57,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1743493.3333333333, ans=0.125
2023-11-22 01:37:01,166 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.315e+01 8.919e+01 9.797e+01 1.182e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 01:37:06,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1743493.3333333333, ans=0.125
2023-11-22 01:37:14,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1743560.0, ans=0.125
2023-11-22 01:37:22,990 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9050, loss[loss=0.09012, simple_loss=0.1173, pruned_loss=0.02441, audio_tagging_loss=0.007052, over 14968.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09693, pruned_loss=0.01631, audio_tagging_loss=0.009238, over 3048223.58 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:37:27,911 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261550
2023-11-22 01:37:33,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1743626.6666666667, ans=0.2
2023-11-22 01:37:35,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1743693.3333333333, ans=0.5
2023-11-22 01:37:41,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1743693.3333333333, ans=0.1
2023-11-22 01:37:49,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.72 vs. limit=22.5
2023-11-22 01:37:58,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.85 vs. limit=10.0
2023-11-22 01:38:24,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1743893.3333333333, ans=0.125
2023-11-22 01:38:27,022 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9100, loss[loss=0.07558, simple_loss=0.09737, pruned_loss=0.01417, audio_tagging_loss=0.01273, over 14158.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09741, pruned_loss=0.01641, audio_tagging_loss=0.009184, over 3051301.15 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:38:28,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1743960.0, ans=0.125
2023-11-22 01:38:31,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261600
2023-11-22 01:38:53,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1744093.3333333333, ans=0.125
2023-11-22 01:38:59,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1744093.3333333333, ans=0.125
2023-11-22 01:38:59,874 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.61 vs. limit=12.0
2023-11-22 01:39:06,382 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.063e+01 8.742e+01 9.387e+01 1.203e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 01:39:14,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1744160.0, ans=0.0
2023-11-22 01:39:30,556 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9150, loss[loss=0.07736, simple_loss=0.1019, pruned_loss=0.0174, audio_tagging_loss=0.00902, over 15065.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09625, pruned_loss=0.01604, audio_tagging_loss=0.009183, over 3036572.65 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:39:35,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261650
2023-11-22 01:39:43,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1744360.0, ans=0.125
2023-11-22 01:40:05,195 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.77 vs. limit=15.0
2023-11-22 01:40:33,657 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9200, loss[loss=0.05469, simple_loss=0.06781, pruned_loss=0.01104, audio_tagging_loss=0.00974, over 15351.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09598, pruned_loss=0.01614, audio_tagging_loss=0.009131, over 3046518.60 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:40:38,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261700
2023-11-22 01:40:55,078 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:40:56,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1744693.3333333333, ans=0.125
2023-11-22 01:40:57,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1744693.3333333333, ans=0.125
2023-11-22 01:41:09,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1744760.0, ans=0.125
2023-11-22 01:41:14,626 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.066e+01 8.592e+01 9.287e+01 1.258e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 01:41:17,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1744826.6666666667, ans=0.125
2023-11-22 01:41:22,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1744826.6666666667, ans=0.2
2023-11-22 01:41:29,733 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.48 vs. limit=15.0
2023-11-22 01:41:33,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1744893.3333333333, ans=0.5
2023-11-22 01:41:37,063 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9250, loss[loss=0.06288, simple_loss=0.07862, pruned_loss=0.01323, audio_tagging_loss=0.01035, over 15891.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.09499, pruned_loss=0.01589, audio_tagging_loss=0.009271, over 3049176.66 frames. ], batch size: 61, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:41:37,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1744960.0, ans=0.07
2023-11-22 01:41:43,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261750
2023-11-22 01:41:56,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1745026.6666666667, ans=0.1
2023-11-22 01:42:01,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1745026.6666666667, ans=0.95
2023-11-22 01:42:20,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1745160.0, ans=0.05
2023-11-22 01:42:22,160 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.39 vs. limit=15.0
2023-11-22 01:42:33,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1745226.6666666667, ans=0.04949747468305833
2023-11-22 01:42:41,999 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9300, loss[loss=0.07147, simple_loss=0.09783, pruned_loss=0.01423, audio_tagging_loss=0.008328, over 16105.00 frames. ], tot_loss[loss=0.07351, simple_loss=0.09639, pruned_loss=0.01607, audio_tagging_loss=0.009248, over 3058356.35 frames. ], batch size: 60, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:42:42,757 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.62 vs. limit=10.0
2023-11-22 01:42:43,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1745293.3333333333, ans=0.125
2023-11-22 01:42:46,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261800
2023-11-22 01:43:23,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1745493.3333333333, ans=0.2
2023-11-22 01:43:24,444 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.431e+01 7.861e+01 8.461e+01 9.251e+01 1.268e+02, threshold=1.692e+02, percent-clipped=0.0
2023-11-22 01:43:45,729 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9350, loss[loss=0.08045, simple_loss=0.1026, pruned_loss=0.02227, audio_tagging_loss=0.006888, over 16950.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09638, pruned_loss=0.01605, audio_tagging_loss=0.009295, over 3057646.21 frames. ], batch size: 63, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:43:50,729 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261850
2023-11-22 01:43:51,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1745626.6666666667, ans=0.0
2023-11-22 01:43:59,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.02 vs. limit=15.0
2023-11-22 01:44:04,546 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5
2023-11-22 01:44:05,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1745693.3333333333, ans=0.125
2023-11-22 01:44:13,338 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:44:14,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1745760.0, ans=0.025
2023-11-22 01:44:20,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.86 vs. limit=15.0
2023-11-22 01:44:21,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1745760.0, ans=0.125
2023-11-22 01:44:31,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1745826.6666666667, ans=0.2
2023-11-22 01:44:49,922 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9400, loss[loss=0.07274, simple_loss=0.0998, pruned_loss=0.01522, audio_tagging_loss=0.007616, over 15011.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09557, pruned_loss=0.01584, audio_tagging_loss=0.009369, over 3048728.80 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:44:50,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1745960.0, ans=0.0
2023-11-22 01:44:55,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261900
2023-11-22 01:45:14,141 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:45:32,787 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.919e+01 8.299e+01 8.911e+01 9.594e+01 2.055e+02, threshold=1.782e+02, percent-clipped=1.0
2023-11-22 01:45:38,668 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-22 01:45:56,227 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9450, loss[loss=0.08185, simple_loss=0.1176, pruned_loss=0.0154, audio_tagging_loss=0.007641, over 14160.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09571, pruned_loss=0.01579, audio_tagging_loss=0.00943, over 3054351.76 frames. ], batch size: 52, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:45:56,301 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:45:59,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1746293.3333333333, ans=0.0
2023-11-22 01:46:01,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 261950
2023-11-22 01:46:05,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1746293.3333333333, ans=0.0
2023-11-22 01:46:10,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1746360.0, ans=0.125
2023-11-22 01:46:20,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1746426.6666666667, ans=0.0
2023-11-22 01:46:22,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1746426.6666666667, ans=0.125
2023-11-22 01:46:53,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1746560.0, ans=0.125
2023-11-22 01:46:59,806 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9500, loss[loss=0.07603, simple_loss=0.1009, pruned_loss=0.01752, audio_tagging_loss=0.008076, over 14508.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09631, pruned_loss=0.01578, audio_tagging_loss=0.009419, over 3054026.43 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:47:01,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1746626.6666666667, ans=0.125
2023-11-22 01:47:02,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1746626.6666666667, ans=0.125
2023-11-22 01:47:04,675 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262000
2023-11-22 01:47:08,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.57 vs. limit=22.5
2023-11-22 01:47:12,610 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:47:30,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0
2023-11-22 01:47:42,655 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.239e+01 8.709e+01 9.390e+01 1.179e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 01:47:43,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1746826.6666666667, ans=0.0
2023-11-22 01:48:03,602 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9550, loss[loss=0.06377, simple_loss=0.0805, pruned_loss=0.01373, audio_tagging_loss=0.00979, over 15724.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09668, pruned_loss=0.01588, audio_tagging_loss=0.009426, over 3050462.99 frames. ], batch size: 60, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:48:09,218 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262050
2023-11-22 01:48:18,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1747026.6666666667, ans=0.125
2023-11-22 01:48:21,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1747026.6666666667, ans=0.0
2023-11-22 01:48:27,800 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.28 vs. limit=15.0
2023-11-22 01:49:03,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1747226.6666666667, ans=0.0
2023-11-22 01:49:05,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=1747226.6666666667, ans=0.025
2023-11-22 01:49:08,515 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9600, loss[loss=0.06174, simple_loss=0.07997, pruned_loss=0.01318, audio_tagging_loss=0.00857, over 14240.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.09604, pruned_loss=0.01588, audio_tagging_loss=0.009543, over 3054342.69 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:49:14,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262100
2023-11-22 01:49:23,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1747360.0, ans=0.0
2023-11-22 01:49:25,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1747360.0, ans=0.0
2023-11-22 01:49:27,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.43 vs. limit=15.0
2023-11-22 01:49:30,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1747360.0, ans=0.0
2023-11-22 01:49:34,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1747426.6666666667, ans=0.0
2023-11-22 01:49:51,171 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 8.033e+01 8.722e+01 9.429e+01 2.117e+02, threshold=1.744e+02, percent-clipped=1.0
2023-11-22 01:50:13,334 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9650, loss[loss=0.07073, simple_loss=0.09057, pruned_loss=0.01404, audio_tagging_loss=0.0114, over 14937.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.09628, pruned_loss=0.01612, audio_tagging_loss=0.00955, over 3049632.78 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:50:18,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262150
2023-11-22 01:50:18,309 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:50:19,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1747626.6666666667, ans=0.0
2023-11-22 01:50:26,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1747693.3333333333, ans=0.125
2023-11-22 01:50:29,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1747693.3333333333, ans=0.1
2023-11-22 01:50:33,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1747693.3333333333, ans=0.1
2023-11-22 01:50:44,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1747760.0, ans=0.2
2023-11-22 01:50:50,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1747826.6666666667, ans=0.125
2023-11-22 01:50:53,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1747826.6666666667, ans=0.0
2023-11-22 01:51:16,679 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9700, loss[loss=0.06219, simple_loss=0.08311, pruned_loss=0.01128, audio_tagging_loss=0.009354, over 15661.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09662, pruned_loss=0.01602, audio_tagging_loss=0.009409, over 3051484.95 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:51:21,557 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262200
2023-11-22 01:51:37,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1748026.6666666667, ans=0.125
2023-11-22 01:51:45,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.28 vs. limit=15.0
2023-11-22 01:51:46,401 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.36 vs. limit=12.0
2023-11-22 01:51:47,431 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.44 vs. limit=15.0
2023-11-22 01:52:00,821 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.573e+01 8.169e+01 8.750e+01 9.625e+01 1.305e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 01:52:21,633 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9750, loss[loss=0.06723, simple_loss=0.08201, pruned_loss=0.01538, audio_tagging_loss=0.01085, over 16283.00 frames. ], tot_loss[loss=0.07369, simple_loss=0.09702, pruned_loss=0.01594, audio_tagging_loss=0.009241, over 3053824.51 frames. ], batch size: 63, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:52:27,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262250
2023-11-22 01:52:28,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1748293.3333333333, ans=0.125
2023-11-22 01:52:56,655 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.44 vs. limit=22.5
2023-11-22 01:53:07,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1748493.3333333333, ans=0.0
2023-11-22 01:53:21,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1748560.0, ans=0.0
2023-11-22 01:53:23,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1748560.0, ans=0.125
2023-11-22 01:53:26,075 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9800, loss[loss=0.07129, simple_loss=0.08764, pruned_loss=0.01837, audio_tagging_loss=0.009097, over 14725.00 frames. ], tot_loss[loss=0.07296, simple_loss=0.09587, pruned_loss=0.01574, audio_tagging_loss=0.009286, over 3046638.12 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:53:31,764 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262300
2023-11-22 01:54:10,084 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.436e+01 9.164e+01 9.957e+01 1.259e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-22 01:54:26,556 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:54:30,302 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9850, loss[loss=0.07068, simple_loss=0.09748, pruned_loss=0.0158, audio_tagging_loss=0.006136, over 15180.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09599, pruned_loss=0.01561, audio_tagging_loss=0.009117, over 3051329.64 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:54:35,294 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262350
2023-11-22 01:54:42,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1749026.6666666667, ans=0.125
2023-11-22 01:54:48,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1749026.6666666667, ans=0.95
2023-11-22 01:54:59,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1749093.3333333333, ans=0.0
2023-11-22 01:55:02,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1749093.3333333333, ans=0.2
2023-11-22 01:55:35,346 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9900, loss[loss=0.07043, simple_loss=0.09474, pruned_loss=0.01653, audio_tagging_loss=0.006532, over 14914.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09573, pruned_loss=0.01559, audio_tagging_loss=0.009179, over 3047978.17 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:55:39,280 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:55:40,305 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262400
2023-11-22 01:55:51,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1749360.0, ans=0.125
2023-11-22 01:55:58,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1749360.0, ans=0.1
2023-11-22 01:56:02,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.50 vs. limit=15.0
2023-11-22 01:56:09,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1749426.6666666667, ans=0.125
2023-11-22 01:56:19,014 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.153e+01 9.023e+01 9.668e+01 1.710e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-22 01:56:20,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1749493.3333333333, ans=0.0
2023-11-22 01:56:20,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1749493.3333333333, ans=0.125
2023-11-22 01:56:28,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1749560.0, ans=0.125
2023-11-22 01:56:39,849 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 9950, loss[loss=0.07334, simple_loss=0.09105, pruned_loss=0.01894, audio_tagging_loss=0.008874, over 15574.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09566, pruned_loss=0.01563, audio_tagging_loss=0.00924, over 3044501.47 frames. ], batch size: 59, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:56:41,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1749626.6666666667, ans=0.0
2023-11-22 01:56:43,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1749626.6666666667, ans=0.0
2023-11-22 01:56:44,749 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262450
2023-11-22 01:56:53,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1749693.3333333333, ans=0.2
2023-11-22 01:56:58,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1749693.3333333333, ans=0.125
2023-11-22 01:56:59,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.66 vs. limit=15.0
2023-11-22 01:57:05,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1749760.0, ans=0.125
2023-11-22 01:57:21,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1749826.6666666667, ans=0.0
2023-11-22 01:57:27,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1749826.6666666667, ans=0.125
2023-11-22 01:57:43,622 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10000, loss[loss=0.0867, simple_loss=0.1122, pruned_loss=0.02162, audio_tagging_loss=0.008979, over 15222.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09501, pruned_loss=0.01547, audio_tagging_loss=0.009326, over 3045777.19 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 01:57:48,597 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262500
2023-11-22 01:57:55,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1750026.6666666667, ans=0.0
2023-11-22 01:57:55,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1750026.6666666667, ans=0.1
2023-11-22 01:58:03,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1750026.6666666667, ans=0.125
2023-11-22 01:58:04,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1750026.6666666667, ans=0.125
2023-11-22 01:58:23,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1750160.0, ans=0.0
2023-11-22 01:58:28,343 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.434e+01 8.075e+01 8.690e+01 9.364e+01 1.327e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 01:58:29,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1750160.0, ans=0.0
2023-11-22 01:58:39,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1750226.6666666667, ans=0.2
2023-11-22 01:58:45,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1750226.6666666667, ans=0.0
2023-11-22 01:58:47,753 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10050, loss[loss=0.07493, simple_loss=0.09934, pruned_loss=0.01616, audio_tagging_loss=0.009098, over 15050.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09476, pruned_loss=0.01547, audio_tagging_loss=0.009224, over 3040611.84 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:58:53,424 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262550
2023-11-22 01:59:02,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1750360.0, ans=0.2
2023-11-22 01:59:42,587 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:59:52,876 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10100, loss[loss=0.09843, simple_loss=0.1264, pruned_loss=0.02811, audio_tagging_loss=0.007146, over 15039.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09553, pruned_loss=0.01555, audio_tagging_loss=0.009132, over 3045673.74 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:59:55,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1750626.6666666667, ans=10.0
2023-11-22 01:59:57,994 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262600
2023-11-22 02:00:07,699 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.73 vs. limit=15.0
2023-11-22 02:00:17,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1750760.0, ans=0.125
2023-11-22 02:00:20,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1750760.0, ans=0.0
2023-11-22 02:00:36,293 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.83 vs. limit=22.5
2023-11-22 02:00:38,617 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.193e+01 8.828e+01 9.579e+01 1.146e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 02:00:44,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1750893.3333333333, ans=0.125
2023-11-22 02:00:47,433 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:00:57,224 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10150, loss[loss=0.05555, simple_loss=0.06451, pruned_loss=0.01045, audio_tagging_loss=0.01285, over 16259.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09486, pruned_loss=0.01548, audio_tagging_loss=0.009303, over 3042624.84 frames. ], batch size: 62, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:00:58,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1750960.0, ans=0.125
2023-11-22 02:01:01,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262650
2023-11-22 02:01:02,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1750960.0, ans=0.2
2023-11-22 02:01:16,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1751026.6666666667, ans=0.125
2023-11-22 02:01:21,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1751026.6666666667, ans=0.125
2023-11-22 02:01:30,330 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:01:51,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1751226.6666666667, ans=0.09899494936611666
2023-11-22 02:01:54,520 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-22 02:01:55,290 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:02:01,528 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10200, loss[loss=0.07351, simple_loss=0.1031, pruned_loss=0.01345, audio_tagging_loss=0.0085, over 14459.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09517, pruned_loss=0.01559, audio_tagging_loss=0.00937, over 3045080.78 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:02:01,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1751293.3333333333, ans=0.2
2023-11-22 02:02:07,048 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262700
2023-11-22 02:02:27,331 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:02:30,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1751426.6666666667, ans=0.95
2023-11-22 02:02:46,384 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 7.988e+01 8.675e+01 9.636e+01 1.224e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 02:02:47,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1751493.3333333333, ans=0.125
2023-11-22 02:02:56,846 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.62 vs. limit=10.0
2023-11-22 02:02:57,796 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:03:06,727 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10250, loss[loss=0.08205, simple_loss=0.1004, pruned_loss=0.02239, audio_tagging_loss=0.009442, over 14521.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09459, pruned_loss=0.01554, audio_tagging_loss=0.009397, over 3042659.33 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 8.0
2023-11-22 02:03:07,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1751626.6666666667, ans=0.0
2023-11-22 02:03:11,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262750
2023-11-22 02:03:13,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1751626.6666666667, ans=0.2
2023-11-22 02:03:16,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1751626.6666666667, ans=0.125
2023-11-22 02:03:22,044 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.34 vs. limit=15.0
2023-11-22 02:03:47,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1751826.6666666667, ans=0.125
2023-11-22 02:03:56,005 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0
2023-11-22 02:04:09,763 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10300, loss[loss=0.07132, simple_loss=0.09271, pruned_loss=0.01474, audio_tagging_loss=0.01023, over 15905.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09463, pruned_loss=0.01555, audio_tagging_loss=0.009458, over 3044939.00 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 8.0
2023-11-22 02:04:14,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262800
2023-11-22 02:04:20,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1751960.0, ans=0.1
2023-11-22 02:04:29,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1752026.6666666667, ans=0.0
2023-11-22 02:04:46,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1752093.3333333333, ans=0.125
2023-11-22 02:04:49,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1752160.0, ans=0.0
2023-11-22 02:04:54,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1752160.0, ans=0.0
2023-11-22 02:04:54,828 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.28 vs. limit=6.0
2023-11-22 02:04:55,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0
2023-11-22 02:04:56,375 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.259e+01 8.792e+01 9.429e+01 1.253e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 02:04:59,736 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.86 vs. limit=15.0
2023-11-22 02:05:14,262 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10350, loss[loss=0.08335, simple_loss=0.1185, pruned_loss=0.01806, audio_tagging_loss=0.006024, over 16408.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09562, pruned_loss=0.01582, audio_tagging_loss=0.009613, over 3046198.25 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 8.0
2023-11-22 02:05:14,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1752293.3333333333, ans=0.125
2023-11-22 02:05:16,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1752293.3333333333, ans=0.0
2023-11-22 02:05:20,497 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262850
2023-11-22 02:05:24,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1752293.3333333333, ans=0.0
2023-11-22 02:05:44,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1752426.6666666667, ans=0.05
2023-11-22 02:06:02,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1752493.3333333333, ans=0.125
2023-11-22 02:06:07,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.91 vs. limit=22.5
2023-11-22 02:06:13,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.52 vs. limit=15.0
2023-11-22 02:06:19,257 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10400, loss[loss=0.07226, simple_loss=0.0971, pruned_loss=0.01355, audio_tagging_loss=0.01015, over 14240.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09432, pruned_loss=0.01566, audio_tagging_loss=0.009683, over 3047778.42 frames. ], batch size: 53, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:06:24,723 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262900
2023-11-22 02:06:25,425 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.79 vs. limit=15.0
2023-11-22 02:06:47,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1752760.0, ans=0.07
2023-11-22 02:06:53,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1752760.0, ans=0.125
2023-11-22 02:07:04,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.068e+01 8.726e+01 9.431e+01 1.284e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 02:07:13,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.87 vs. limit=15.0
2023-11-22 02:07:15,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1752893.3333333333, ans=0.125
2023-11-22 02:07:18,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1752893.3333333333, ans=0.125
2023-11-22 02:07:22,667 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10450, loss[loss=0.04416, simple_loss=0.0455, pruned_loss=0.008799, audio_tagging_loss=0.01261, over 15065.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09422, pruned_loss=0.01559, audio_tagging_loss=0.009622, over 3057612.80 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:07:27,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 262950
2023-11-22 02:07:44,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1753026.6666666667, ans=0.1
2023-11-22 02:07:51,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1753093.3333333333, ans=0.2
2023-11-22 02:07:59,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1753093.3333333333, ans=0.125
2023-11-22 02:08:00,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1753160.0, ans=0.125
2023-11-22 02:08:03,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1753160.0, ans=0.125
2023-11-22 02:08:25,952 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10500, loss[loss=0.06834, simple_loss=0.08753, pruned_loss=0.01523, audio_tagging_loss=0.009349, over 15170.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.0947, pruned_loss=0.01566, audio_tagging_loss=0.009478, over 3050920.46 frames. ], batch size: 59, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:08:30,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1753293.3333333333, ans=0.0
2023-11-22 02:08:31,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263000
2023-11-22 02:09:01,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1753426.6666666667, ans=0.125
2023-11-22 02:09:10,806 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.52 vs. limit=15.0
2023-11-22 02:09:12,587 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.420e+01 8.257e+01 8.868e+01 9.626e+01 1.177e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 02:09:31,578 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10550, loss[loss=0.08418, simple_loss=0.1151, pruned_loss=0.01858, audio_tagging_loss=0.00806, over 15148.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09544, pruned_loss=0.01584, audio_tagging_loss=0.009284, over 3054432.35 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:09:33,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.70 vs. limit=22.5
2023-11-22 02:09:37,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263050
2023-11-22 02:09:42,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1753626.6666666667, ans=0.125
2023-11-22 02:10:06,752 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:10:27,882 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.91 vs. limit=15.0
2023-11-22 02:10:31,783 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.58 vs. limit=15.0
2023-11-22 02:10:35,915 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10600, loss[loss=0.06326, simple_loss=0.08315, pruned_loss=0.01303, audio_tagging_loss=0.00865, over 15648.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.0958, pruned_loss=0.01592, audio_tagging_loss=0.009266, over 3055033.27 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:10:38,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1753960.0, ans=0.125
2023-11-22 02:10:40,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263100
2023-11-22 02:10:43,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1753960.0, ans=0.0
2023-11-22 02:10:50,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1754026.6666666667, ans=0.125
2023-11-22 02:11:04,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1754093.3333333333, ans=0.1
2023-11-22 02:11:13,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1754160.0, ans=0.1
2023-11-22 02:11:21,769 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 8.070e+01 8.614e+01 9.337e+01 1.249e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-22 02:11:30,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1754226.6666666667, ans=0.125
2023-11-22 02:11:35,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1754226.6666666667, ans=0.125
2023-11-22 02:11:35,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1754226.6666666667, ans=0.125
2023-11-22 02:11:38,648 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10650, loss[loss=0.08641, simple_loss=0.1136, pruned_loss=0.02005, audio_tagging_loss=0.009563, over 15215.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09596, pruned_loss=0.01608, audio_tagging_loss=0.009258, over 3053193.42 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:11:43,536 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263150
2023-11-22 02:11:53,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1754360.0, ans=0.125
2023-11-22 02:11:54,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1754360.0, ans=0.2
2023-11-22 02:11:57,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1754360.0, ans=0.1
2023-11-22 02:11:59,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1754360.0, ans=0.125
2023-11-22 02:12:00,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1754360.0, ans=0.0
2023-11-22 02:12:35,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1754560.0, ans=0.2
2023-11-22 02:12:42,110 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10700, loss[loss=0.06074, simple_loss=0.08765, pruned_loss=0.008837, audio_tagging_loss=0.008074, over 13996.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09678, pruned_loss=0.01615, audio_tagging_loss=0.009193, over 3048008.15 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:12:43,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1754626.6666666667, ans=0.0
2023-11-22 02:12:47,162 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263200
2023-11-22 02:13:07,333 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:13:12,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1754760.0, ans=0.0
2023-11-22 02:13:22,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1754826.6666666667, ans=0.125
2023-11-22 02:13:26,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1754826.6666666667, ans=0.125
2023-11-22 02:13:29,042 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.759e+01 7.890e+01 8.587e+01 9.339e+01 1.239e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-22 02:13:47,150 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10750, loss[loss=0.07169, simple_loss=0.09367, pruned_loss=0.01539, audio_tagging_loss=0.009467, over 14974.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.0957, pruned_loss=0.01594, audio_tagging_loss=0.009264, over 3045076.86 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:13:48,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1754960.0, ans=0.125
2023-11-22 02:13:52,195 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263250
2023-11-22 02:14:18,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1755093.3333333333, ans=0.125
2023-11-22 02:14:22,977 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.74 vs. limit=22.5
2023-11-22 02:14:29,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1755160.0, ans=0.0
2023-11-22 02:14:44,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1755226.6666666667, ans=0.125
2023-11-22 02:14:49,799 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10800, loss[loss=0.07072, simple_loss=0.0892, pruned_loss=0.0173, audio_tagging_loss=0.008821, over 15026.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09589, pruned_loss=0.01609, audio_tagging_loss=0.009256, over 3047386.59 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:14:52,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1755293.3333333333, ans=0.0
2023-11-22 02:14:54,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.75 vs. limit=6.0
2023-11-22 02:14:54,749 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263300
2023-11-22 02:15:06,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1755360.0, ans=0.125
2023-11-22 02:15:23,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1755426.6666666667, ans=0.0
2023-11-22 02:15:32,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1755493.3333333333, ans=0.125
2023-11-22 02:15:37,523 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 8.311e+01 8.848e+01 9.855e+01 1.832e+02, threshold=1.770e+02, percent-clipped=2.0
2023-11-22 02:15:54,654 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10850, loss[loss=0.04778, simple_loss=0.05125, pruned_loss=0.009466, audio_tagging_loss=0.01269, over 14801.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09492, pruned_loss=0.01603, audio_tagging_loss=0.009378, over 3046255.10 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:15:58,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.79 vs. limit=15.0
2023-11-22 02:15:59,584 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263350
2023-11-22 02:16:05,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1755626.6666666667, ans=0.125
2023-11-22 02:16:20,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1755760.0, ans=0.125
2023-11-22 02:16:26,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1755760.0, ans=0.2
2023-11-22 02:16:57,473 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:16:58,674 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10900, loss[loss=0.06234, simple_loss=0.0934, pruned_loss=0.007684, audio_tagging_loss=0.007956, over 15600.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.0948, pruned_loss=0.0159, audio_tagging_loss=0.009412, over 3043022.36 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:17:04,294 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263400
2023-11-22 02:17:08,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1755960.0, ans=0.125
2023-11-22 02:17:13,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1756026.6666666667, ans=0.1
2023-11-22 02:17:16,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1756026.6666666667, ans=15.0
2023-11-22 02:17:40,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1756160.0, ans=0.0
2023-11-22 02:17:46,993 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 8.172e+01 8.770e+01 9.474e+01 2.328e+02, threshold=1.754e+02, percent-clipped=1.0
2023-11-22 02:18:03,848 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 10950, loss[loss=0.08515, simple_loss=0.1148, pruned_loss=0.02141, audio_tagging_loss=0.006311, over 14664.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09501, pruned_loss=0.01581, audio_tagging_loss=0.009493, over 3045887.39 frames. ], batch size: 53, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:18:08,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1756293.3333333333, ans=0.125
2023-11-22 02:18:08,972 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263450
2023-11-22 02:18:14,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1756293.3333333333, ans=0.0
2023-11-22 02:18:15,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1756360.0, ans=0.125
2023-11-22 02:18:17,581 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=12.91 vs. limit=15.0
2023-11-22 02:18:31,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1756426.6666666667, ans=0.125
2023-11-22 02:18:49,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1756493.3333333333, ans=0.0
2023-11-22 02:18:56,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1756560.0, ans=0.2
2023-11-22 02:18:57,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1756560.0, ans=0.125
2023-11-22 02:19:07,446 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11000, loss[loss=0.07012, simple_loss=0.09971, pruned_loss=0.01049, audio_tagging_loss=0.009775, over 14498.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09434, pruned_loss=0.01568, audio_tagging_loss=0.009518, over 3042408.22 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:19:13,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263500
2023-11-22 02:19:13,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1756626.6666666667, ans=0.125
2023-11-22 02:19:21,172 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:19:35,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1756760.0, ans=0.05
2023-11-22 02:19:51,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1756826.6666666667, ans=0.0
2023-11-22 02:19:53,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1756826.6666666667, ans=0.1
2023-11-22 02:19:55,172 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.467e+01 8.140e+01 8.725e+01 9.449e+01 1.514e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 02:20:12,228 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11050, loss[loss=0.08107, simple_loss=0.1115, pruned_loss=0.01713, audio_tagging_loss=0.00816, over 15052.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09326, pruned_loss=0.0155, audio_tagging_loss=0.009572, over 3030468.29 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:20:16,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1756960.0, ans=0.125
2023-11-22 02:20:17,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263550
2023-11-22 02:21:03,811 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.20 vs. limit=15.0
2023-11-22 02:21:12,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.46 vs. limit=15.0
2023-11-22 02:21:16,227 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11100, loss[loss=0.07233, simple_loss=0.09556, pruned_loss=0.01391, audio_tagging_loss=0.01065, over 16015.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09452, pruned_loss=0.01567, audio_tagging_loss=0.00961, over 3043599.63 frames. ], batch size: 59, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:21:21,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263600
2023-11-22 02:21:50,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1757426.6666666667, ans=0.125
2023-11-22 02:21:53,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1757426.6666666667, ans=0.04949747468305833
2023-11-22 02:21:55,658 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.66 vs. limit=15.0
2023-11-22 02:22:03,459 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.113e+01 8.819e+01 9.747e+01 1.374e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 02:22:07,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1757560.0, ans=0.0
2023-11-22 02:22:08,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.71 vs. limit=15.0
2023-11-22 02:22:20,506 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11150, loss[loss=0.0644, simple_loss=0.08306, pruned_loss=0.01262, audio_tagging_loss=0.01025, over 14562.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09329, pruned_loss=0.01549, audio_tagging_loss=0.009799, over 3043856.45 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:22:25,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263650
2023-11-22 02:22:38,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1757693.3333333333, ans=0.125
2023-11-22 02:22:41,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1757693.3333333333, ans=0.0
2023-11-22 02:23:10,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1757893.3333333333, ans=0.125
2023-11-22 02:23:25,255 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11200, loss[loss=0.06297, simple_loss=0.07764, pruned_loss=0.01518, audio_tagging_loss=0.008971, over 16688.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09246, pruned_loss=0.01531, audio_tagging_loss=0.009973, over 3043412.56 frames. ], batch size: 64, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:23:27,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1757960.0, ans=0.125
2023-11-22 02:23:30,082 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263700
2023-11-22 02:23:54,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1758093.3333333333, ans=0.09899494936611666
2023-11-22 02:24:11,977 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.183e+01 8.908e+01 9.614e+01 1.304e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 02:24:27,700 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11250, loss[loss=0.05765, simple_loss=0.07605, pruned_loss=0.01089, audio_tagging_loss=0.008738, over 14482.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09219, pruned_loss=0.01517, audio_tagging_loss=0.009929, over 3038965.04 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:24:32,659 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263750
2023-11-22 02:24:38,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1758293.3333333333, ans=0.0
2023-11-22 02:25:00,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.70 vs. limit=22.5
2023-11-22 02:25:01,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.49 vs. limit=22.5
2023-11-22 02:25:31,820 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11300, loss[loss=0.06227, simple_loss=0.0816, pruned_loss=0.01257, audio_tagging_loss=0.0089, over 15016.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09279, pruned_loss=0.01524, audio_tagging_loss=0.009789, over 3040183.05 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:25:36,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263800
2023-11-22 02:25:41,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1758626.6666666667, ans=0.125
2023-11-22 02:26:08,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1758760.0, ans=0.2
2023-11-22 02:26:18,464 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:26:19,792 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.464e+01 8.064e+01 8.666e+01 9.719e+01 1.287e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 02:26:26,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1758893.3333333333, ans=0.2
2023-11-22 02:26:36,260 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11350, loss[loss=0.07866, simple_loss=0.104, pruned_loss=0.01673, audio_tagging_loss=0.00992, over 15422.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09411, pruned_loss=0.01551, audio_tagging_loss=0.009654, over 3046233.78 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:26:41,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263850
2023-11-22 02:27:14,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1759160.0, ans=0.0
2023-11-22 02:27:39,809 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11400, loss[loss=0.07362, simple_loss=0.1004, pruned_loss=0.01705, audio_tagging_loss=0.006347, over 14773.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09408, pruned_loss=0.01563, audio_tagging_loss=0.009585, over 3049153.60 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:27:44,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263900
2023-11-22 02:27:52,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1759360.0, ans=0.0
2023-11-22 02:28:10,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1759426.6666666667, ans=0.1
2023-11-22 02:28:26,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1759493.3333333333, ans=0.125
2023-11-22 02:28:27,241 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 8.049e+01 8.601e+01 9.494e+01 1.206e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-22 02:28:27,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1759493.3333333333, ans=0.0
2023-11-22 02:28:27,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1759493.3333333333, ans=0.125
2023-11-22 02:28:34,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1759560.0, ans=0.0
2023-11-22 02:28:43,098 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11450, loss[loss=0.07843, simple_loss=0.1045, pruned_loss=0.019, audio_tagging_loss=0.007174, over 14321.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09431, pruned_loss=0.01585, audio_tagging_loss=0.009539, over 3047781.63 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:28:48,627 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 263950
2023-11-22 02:28:48,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1759626.6666666667, ans=0.125
2023-11-22 02:29:04,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1759693.3333333333, ans=0.1
2023-11-22 02:29:38,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1759893.3333333333, ans=0.125
2023-11-22 02:29:47,462 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11500, loss[loss=0.07368, simple_loss=0.09938, pruned_loss=0.0154, audio_tagging_loss=0.008597, over 15786.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.0935, pruned_loss=0.01552, audio_tagging_loss=0.009522, over 3039997.37 frames. ], batch size: 57, lr: 3.07e-03, grad_scale: 32.0
2023-11-22 02:29:53,154 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264000
2023-11-22 02:30:06,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1760026.6666666667, ans=0.125
2023-11-22 02:30:10,413 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.88 vs. limit=15.0
2023-11-22 02:30:34,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1760160.0, ans=0.125
2023-11-22 02:30:38,029 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.114e+01 8.860e+01 9.560e+01 1.133e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 02:30:54,559 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11550, loss[loss=0.05855, simple_loss=0.07274, pruned_loss=0.01021, audio_tagging_loss=0.01196, over 13333.00 frames. ], tot_loss[loss=0.07184, simple_loss=0.09348, pruned_loss=0.01554, audio_tagging_loss=0.009563, over 3041240.09 frames. ], batch size: 51, lr: 3.07e-03, grad_scale: 32.0
2023-11-22 02:30:59,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264050
2023-11-22 02:31:11,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.43 vs. limit=15.0
2023-11-22 02:31:11,198 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.57 vs. limit=22.5
2023-11-22 02:31:16,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1760360.0, ans=0.125
2023-11-22 02:31:35,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1760493.3333333333, ans=0.5
2023-11-22 02:31:37,435 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:31:57,970 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11600, loss[loss=0.08694, simple_loss=0.1127, pruned_loss=0.02269, audio_tagging_loss=0.007922, over 16068.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09456, pruned_loss=0.0157, audio_tagging_loss=0.009518, over 3042462.58 frames. ], batch size: 59, lr: 3.07e-03, grad_scale: 32.0
2023-11-22 02:32:02,833 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=12.0
2023-11-22 02:32:03,527 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264100
2023-11-22 02:32:10,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1760693.3333333333, ans=0.125
2023-11-22 02:32:36,575 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0
2023-11-22 02:32:47,755 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.153e+01 8.705e+01 9.370e+01 1.172e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 02:32:57,422 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.78 vs. limit=15.0
2023-11-22 02:33:02,434 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11650, loss[loss=0.06125, simple_loss=0.08259, pruned_loss=0.01106, audio_tagging_loss=0.008899, over 15813.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09371, pruned_loss=0.01567, audio_tagging_loss=0.009505, over 3038381.26 frames. ], batch size: 60, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:33:05,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1760960.0, ans=0.5
2023-11-22 02:33:08,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264150
2023-11-22 02:33:55,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1761226.6666666667, ans=0.0
2023-11-22 02:34:07,587 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11700, loss[loss=0.05252, simple_loss=0.06296, pruned_loss=0.01186, audio_tagging_loss=0.009173, over 15443.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09339, pruned_loss=0.01561, audio_tagging_loss=0.009504, over 3043838.77 frames. ], batch size: 60, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:34:12,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264200
2023-11-22 02:34:15,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1761293.3333333333, ans=0.5
2023-11-22 02:34:20,823 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.31 vs. limit=6.0
2023-11-22 02:34:45,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1761493.3333333333, ans=0.0
2023-11-22 02:34:49,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=1761493.3333333333, ans=0.05
2023-11-22 02:34:52,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1761493.3333333333, ans=0.1
2023-11-22 02:34:57,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1761493.3333333333, ans=0.0
2023-11-22 02:34:57,767 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.071e+01 8.694e+01 9.444e+01 1.226e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 02:35:01,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1761560.0, ans=0.0
2023-11-22 02:35:09,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1761560.0, ans=0.07
2023-11-22 02:35:11,127 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11750, loss[loss=0.09422, simple_loss=0.1376, pruned_loss=0.01653, audio_tagging_loss=0.00889, over 15760.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.0946, pruned_loss=0.01575, audio_tagging_loss=0.009434, over 3045792.03 frames. ], batch size: 56, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:35:11,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.54 vs. limit=15.0
2023-11-22 02:35:16,093 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264250
2023-11-22 02:35:18,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1761626.6666666667, ans=0.0
2023-11-22 02:35:33,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1761693.3333333333, ans=0.2
2023-11-22 02:35:40,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1761760.0, ans=0.125
2023-11-22 02:35:41,528 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=14.24 vs. limit=22.5
2023-11-22 02:35:58,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1761826.6666666667, ans=0.125
2023-11-22 02:35:59,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.79 vs. limit=10.0
2023-11-22 02:36:10,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1761893.3333333333, ans=0.1
2023-11-22 02:36:15,725 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11800, loss[loss=0.05531, simple_loss=0.06935, pruned_loss=0.01038, audio_tagging_loss=0.01026, over 15025.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09415, pruned_loss=0.01568, audio_tagging_loss=0.009409, over 3042408.39 frames. ], batch size: 58, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:36:18,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1761960.0, ans=0.125
2023-11-22 02:36:21,285 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264300
2023-11-22 02:36:44,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.03 vs. limit=12.0
2023-11-22 02:36:45,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1762093.3333333333, ans=0.125
2023-11-22 02:36:46,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1762093.3333333333, ans=0.0
2023-11-22 02:37:06,213 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.285e+01 8.657e+01 9.300e+01 1.154e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 02:37:07,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1762226.6666666667, ans=0.125
2023-11-22 02:37:15,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1762226.6666666667, ans=0.125
2023-11-22 02:37:21,183 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11850, loss[loss=0.08767, simple_loss=0.1181, pruned_loss=0.01857, audio_tagging_loss=0.01004, over 15443.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09438, pruned_loss=0.01566, audio_tagging_loss=0.009562, over 3036700.57 frames. ], batch size: 55, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:37:26,196 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264350
2023-11-22 02:37:39,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1762360.0, ans=0.1
2023-11-22 02:37:41,188 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.57 vs. limit=22.5
2023-11-22 02:38:10,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1762560.0, ans=0.2
2023-11-22 02:38:16,080 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.10 vs. limit=22.5
2023-11-22 02:38:22,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1762626.6666666667, ans=0.125
2023-11-22 02:38:23,845 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11900, loss[loss=0.05966, simple_loss=0.08429, pruned_loss=0.007189, audio_tagging_loss=0.01033, over 15005.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.0943, pruned_loss=0.01561, audio_tagging_loss=0.009557, over 3037409.61 frames. ], batch size: 57, lr: 3.07e-03, grad_scale: 8.0
2023-11-22 02:38:28,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264400
2023-11-22 02:38:54,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1762760.0, ans=10.0
2023-11-22 02:39:01,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1762826.6666666667, ans=0.125
2023-11-22 02:39:07,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.80 vs. limit=10.0
2023-11-22 02:39:07,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1762826.6666666667, ans=0.125
2023-11-22 02:39:11,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1762826.6666666667, ans=0.0
2023-11-22 02:39:12,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1762826.6666666667, ans=0.1
2023-11-22 02:39:15,107 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.906e+01 8.188e+01 8.826e+01 9.362e+01 1.336e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 02:39:17,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.38 vs. limit=10.0
2023-11-22 02:39:28,034 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 11950, loss[loss=0.07144, simple_loss=0.1012, pruned_loss=0.01236, audio_tagging_loss=0.008482, over 14646.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09438, pruned_loss=0.01551, audio_tagging_loss=0.00963, over 3035509.60 frames. ], batch size: 54, lr: 3.07e-03, grad_scale: 8.0
2023-11-22 02:39:28,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1762960.0, ans=0.125
2023-11-22 02:39:33,695 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264450
2023-11-22 02:39:39,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1762960.0, ans=0.125
2023-11-22 02:39:54,123 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.21 vs. limit=15.0
2023-11-22 02:40:04,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1763093.3333333333, ans=0.2
2023-11-22 02:40:16,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1763160.0, ans=0.125
2023-11-22 02:40:20,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1763226.6666666667, ans=0.0
2023-11-22 02:40:26,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1763226.6666666667, ans=0.125
2023-11-22 02:40:28,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1763226.6666666667, ans=0.0
2023-11-22 02:40:31,119 INFO [train_asr.py:1221] (2/4) Epoch 22, batch 12000, loss[loss=0.07574, simple_loss=0.1092, pruned_loss=0.01282, audio_tagging_loss=0.008337, over 15269.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09431, pruned_loss=0.01554, audio_tagging_loss=0.009758, over 3036595.41 frames. ], batch size: 57, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:40:31,119 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 02:41:12,956 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.8734, 4.0666, 3.7209, 3.1734], device='cuda:2')
2023-11-22 02:41:14,445 INFO [train_asr.py:1253] (2/4) Epoch 22, validation: loss=0.05922, simple_loss=0.05191, pruned_loss=0.005254, audio_tagging_loss=0.02801, over 4681554.00 frames. 
2023-11-22 02:41:14,446 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 02:41:14,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1763293.3333333333, ans=0.125
2023-11-22 02:41:19,297 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264500
2023-11-22 02:41:33,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1763360.0, ans=0.0
2023-11-22 02:41:39,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1763426.6666666667, ans=0.1
2023-11-22 02:42:19,442 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.06 vs. limit=15.0
2023-11-22 02:42:19,905 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 0, loss[loss=0.0822, simple_loss=0.0995, pruned_loss=0.01029, audio_tagging_loss=0.02217, over 15171.00 frames. ], tot_loss[loss=0.0822, simple_loss=0.0995, pruned_loss=0.01029, audio_tagging_loss=0.02217, over 15171.00 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 32.0
2023-11-22 02:42:19,906 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 02:42:55,363 INFO [train_asr.py:1253] (2/4) Epoch 23, validation: loss=0.05874, simple_loss=0.05183, pruned_loss=0.005194, audio_tagging_loss=0.02763, over 4681554.00 frames. 
2023-11-22 02:42:55,364 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 02:43:05,480 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.70 vs. limit=15.0
2023-11-22 02:43:06,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.73 vs. limit=15.0
2023-11-22 02:43:13,344 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.370e+01 9.150e+01 9.856e+01 1.621e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-22 02:43:18,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1763540.0, ans=0.125
2023-11-22 02:43:22,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1763606.6666666667, ans=0.1
2023-11-22 02:43:23,598 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.15 vs. limit=10.0
2023-11-22 02:43:30,154 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264550
2023-11-22 02:43:43,655 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.94 vs. limit=12.0
2023-11-22 02:43:47,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1763740.0, ans=0.0
2023-11-22 02:43:55,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1763740.0, ans=0.1
2023-11-22 02:43:57,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1763740.0, ans=0.2
2023-11-22 02:43:59,730 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 50, loss[loss=0.07668, simple_loss=0.09437, pruned_loss=0.01344, audio_tagging_loss=0.01606, over 14936.00 frames. ], tot_loss[loss=0.08129, simple_loss=0.09502, pruned_loss=0.01569, audio_tagging_loss=0.01809, over 684095.85 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:44:03,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1763806.6666666667, ans=0.0
2023-11-22 02:44:35,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264600
2023-11-22 02:45:00,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1764073.3333333333, ans=0.125
2023-11-22 02:45:01,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1764073.3333333333, ans=0.125
2023-11-22 02:45:05,755 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 100, loss[loss=0.08202, simple_loss=0.101, pruned_loss=0.01536, audio_tagging_loss=0.01616, over 15056.00 frames. ], tot_loss[loss=0.08123, simple_loss=0.09602, pruned_loss=0.01603, audio_tagging_loss=0.01719, over 1204363.78 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:45:08,951 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.32 vs. limit=6.0
2023-11-22 02:45:23,746 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 9.017e+01 9.520e+01 1.030e+02 1.259e+02, threshold=1.904e+02, percent-clipped=0.0
2023-11-22 02:45:24,524 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.06 vs. limit=15.0
2023-11-22 02:45:40,573 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264650
2023-11-22 02:46:00,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1764406.6666666667, ans=0.2
2023-11-22 02:46:03,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1764406.6666666667, ans=0.1
2023-11-22 02:46:11,424 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 150, loss[loss=0.07713, simple_loss=0.09995, pruned_loss=0.0177, audio_tagging_loss=0.009463, over 14444.00 frames. ], tot_loss[loss=0.0783, simple_loss=0.09449, pruned_loss=0.01549, audio_tagging_loss=0.01556, over 1619736.77 frames. ], batch size: 54, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:46:39,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1764606.6666666667, ans=0.0
2023-11-22 02:46:46,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264700
2023-11-22 02:47:16,090 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 200, loss[loss=0.0856, simple_loss=0.1161, pruned_loss=0.02116, audio_tagging_loss=0.006396, over 14748.00 frames. ], tot_loss[loss=0.07687, simple_loss=0.0951, pruned_loss=0.01566, audio_tagging_loss=0.01365, over 1938296.23 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:47:30,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1764873.3333333333, ans=0.125
2023-11-22 02:47:33,138 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0
2023-11-22 02:47:33,579 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.877e+01 8.445e+01 8.941e+01 9.618e+01 1.553e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 02:47:38,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1764873.3333333333, ans=0.0
2023-11-22 02:47:50,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264750
2023-11-22 02:47:53,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1765006.6666666667, ans=0.2
2023-11-22 02:47:56,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1765006.6666666667, ans=0.125
2023-11-22 02:48:07,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1765073.3333333333, ans=0.125
2023-11-22 02:48:11,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1765073.3333333333, ans=0.125
2023-11-22 02:48:20,510 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 250, loss[loss=0.06633, simple_loss=0.08827, pruned_loss=0.01296, audio_tagging_loss=0.009241, over 15180.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09471, pruned_loss=0.01557, audio_tagging_loss=0.01244, over 2178369.09 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:48:22,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1765140.0, ans=0.07
2023-11-22 02:48:39,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1765206.6666666667, ans=0.035
2023-11-22 02:48:40,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1765206.6666666667, ans=0.125
2023-11-22 02:48:40,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1765206.6666666667, ans=0.2
2023-11-22 02:48:43,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1765206.6666666667, ans=0.1
2023-11-22 02:48:46,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1765273.3333333333, ans=0.0
2023-11-22 02:48:50,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1765273.3333333333, ans=0.125
2023-11-22 02:48:55,788 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264800
2023-11-22 02:48:57,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1765273.3333333333, ans=0.125
2023-11-22 02:49:08,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1765340.0, ans=0.125
2023-11-22 02:49:17,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1765406.6666666667, ans=0.0
2023-11-22 02:49:26,647 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 300, loss[loss=0.07925, simple_loss=0.1092, pruned_loss=0.01486, audio_tagging_loss=0.00978, over 16146.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09458, pruned_loss=0.01556, audio_tagging_loss=0.01161, over 2375417.95 frames. ], batch size: 59, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:49:44,447 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.292e+01 9.030e+01 9.528e+01 1.689e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 02:50:01,694 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264850
2023-11-22 02:50:10,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1765673.3333333333, ans=0.05
2023-11-22 02:50:17,862 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:50:26,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1765740.0, ans=0.0
2023-11-22 02:50:31,851 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 350, loss[loss=0.06361, simple_loss=0.07554, pruned_loss=0.01561, audio_tagging_loss=0.01023, over 15237.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09386, pruned_loss=0.01535, audio_tagging_loss=0.01099, over 2522441.79 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:50:34,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1765806.6666666667, ans=0.125
2023-11-22 02:50:52,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1765873.3333333333, ans=0.125
2023-11-22 02:50:56,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1765940.0, ans=0.1
2023-11-22 02:51:07,171 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264900
2023-11-22 02:51:18,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1766006.6666666667, ans=0.125
2023-11-22 02:51:23,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1766073.3333333333, ans=0.0
2023-11-22 02:51:36,574 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 400, loss[loss=0.08164, simple_loss=0.106, pruned_loss=0.018, audio_tagging_loss=0.01065, over 15499.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09423, pruned_loss=0.01544, audio_tagging_loss=0.01054, over 2638348.28 frames. ], batch size: 60, lr: 3.00e-03, grad_scale: 32.0
2023-11-22 02:51:40,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1766140.0, ans=0.0
2023-11-22 02:51:55,102 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.193e+01 8.089e+01 8.847e+01 9.636e+01 1.262e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 02:52:11,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 264950
2023-11-22 02:52:16,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1766340.0, ans=0.125
2023-11-22 02:52:26,431 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.39 vs. limit=22.5
2023-11-22 02:52:34,303 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.84 vs. limit=15.0
2023-11-22 02:52:41,684 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 450, loss[loss=0.09954, simple_loss=0.1364, pruned_loss=0.02296, audio_tagging_loss=0.008393, over 15212.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09469, pruned_loss=0.01567, audio_tagging_loss=0.0102, over 2734915.24 frames. ], batch size: 54, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:52:45,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1766473.3333333333, ans=0.09899494936611666
2023-11-22 02:53:00,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1766540.0, ans=0.125
2023-11-22 02:53:00,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1766540.0, ans=0.125
2023-11-22 02:53:16,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1766606.6666666667, ans=0.1
2023-11-22 02:53:17,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265000
2023-11-22 02:53:28,078 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.09 vs. limit=15.0
2023-11-22 02:53:44,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1766740.0, ans=0.125
2023-11-22 02:53:46,802 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 500, loss[loss=0.06429, simple_loss=0.08337, pruned_loss=0.01339, audio_tagging_loss=0.009215, over 14948.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09414, pruned_loss=0.01561, audio_tagging_loss=0.01012, over 2798800.28 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:54:06,418 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.266e+01 9.127e+01 9.856e+01 1.336e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-22 02:54:11,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1766940.0, ans=0.125
2023-11-22 02:54:15,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1766940.0, ans=0.0
2023-11-22 02:54:21,506 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265050
2023-11-22 02:54:51,570 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 550, loss[loss=0.08021, simple_loss=0.111, pruned_loss=0.01666, audio_tagging_loss=0.008026, over 15658.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09415, pruned_loss=0.01548, audio_tagging_loss=0.00996, over 2848745.04 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:55:06,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1767206.6666666667, ans=0.125
2023-11-22 02:55:24,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1767273.3333333333, ans=0.2
2023-11-22 02:55:26,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265100
2023-11-22 02:55:40,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1767340.0, ans=0.0
2023-11-22 02:55:44,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1767406.6666666667, ans=0.5
2023-11-22 02:55:45,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.61 vs. limit=15.0
2023-11-22 02:55:55,658 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 600, loss[loss=0.05005, simple_loss=0.06624, pruned_loss=0.006302, audio_tagging_loss=0.01063, over 15220.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09464, pruned_loss=0.01555, audio_tagging_loss=0.009869, over 2891081.92 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:55:57,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1767473.3333333333, ans=0.0
2023-11-22 02:56:08,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.23 vs. limit=12.0
2023-11-22 02:56:08,682 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.29 vs. limit=15.0
2023-11-22 02:56:15,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.111e+01 8.755e+01 9.380e+01 1.139e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 02:56:20,191 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.87 vs. limit=15.0
2023-11-22 02:56:24,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1767606.6666666667, ans=0.125
2023-11-22 02:56:31,596 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265150
2023-11-22 02:56:45,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1767673.3333333333, ans=0.0
2023-11-22 02:57:01,423 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 650, loss[loss=0.07587, simple_loss=0.1072, pruned_loss=0.01501, audio_tagging_loss=0.007266, over 16122.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09478, pruned_loss=0.01559, audio_tagging_loss=0.009822, over 2925112.00 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:57:25,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1767873.3333333333, ans=0.0
2023-11-22 02:57:34,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1767940.0, ans=0.1
2023-11-22 02:57:36,018 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265200
2023-11-22 02:57:45,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1768006.6666666667, ans=0.0
2023-11-22 02:57:48,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1768006.6666666667, ans=0.2
2023-11-22 02:57:50,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1768006.6666666667, ans=0.125
2023-11-22 02:58:06,628 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 700, loss[loss=0.07607, simple_loss=0.1057, pruned_loss=0.01582, audio_tagging_loss=0.007401, over 15886.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09463, pruned_loss=0.01566, audio_tagging_loss=0.009851, over 2952907.00 frames. ], batch size: 59, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:58:15,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1768140.0, ans=0.125
2023-11-22 02:58:23,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1768206.6666666667, ans=0.2
2023-11-22 02:58:26,076 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.011e+01 8.675e+01 9.413e+01 1.282e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 02:58:42,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265250
2023-11-22 02:58:46,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1768340.0, ans=0.125
2023-11-22 02:59:11,954 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 750, loss[loss=0.1049, simple_loss=0.1357, pruned_loss=0.02897, audio_tagging_loss=0.008052, over 16474.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09541, pruned_loss=0.01583, audio_tagging_loss=0.009661, over 2986128.33 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 02:59:47,604 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265300
2023-11-22 03:00:03,633 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:00:17,346 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 800, loss[loss=0.07429, simple_loss=0.09216, pruned_loss=0.01737, audio_tagging_loss=0.01084, over 14908.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09502, pruned_loss=0.01566, audio_tagging_loss=0.009669, over 2999923.43 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:00:39,219 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.565e+01 9.192e+01 1.025e+02 1.525e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-22 03:00:48,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1768940.0, ans=0.125
2023-11-22 03:00:50,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1768940.0, ans=0.0
2023-11-22 03:00:53,123 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265350
2023-11-22 03:00:53,567 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.39 vs. limit=15.0
2023-11-22 03:00:55,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1769006.6666666667, ans=0.125
2023-11-22 03:01:03,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1769006.6666666667, ans=0.025
2023-11-22 03:01:07,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1769006.6666666667, ans=0.125
2023-11-22 03:01:23,370 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 850, loss[loss=0.07299, simple_loss=0.09549, pruned_loss=0.01726, audio_tagging_loss=0.00799, over 15163.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09472, pruned_loss=0.01563, audio_tagging_loss=0.009762, over 3009388.32 frames. ], batch size: 54, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:01:43,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1769206.6666666667, ans=0.0
2023-11-22 03:01:58,363 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265400
2023-11-22 03:02:28,367 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 900, loss[loss=0.077, simple_loss=0.09742, pruned_loss=0.01665, audio_tagging_loss=0.01164, over 15518.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.0952, pruned_loss=0.01586, audio_tagging_loss=0.009819, over 3024662.53 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:02:50,274 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.073e+01 8.740e+01 9.361e+01 1.142e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 03:02:51,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1769540.0, ans=0.125
2023-11-22 03:02:53,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1769606.6666666667, ans=0.1
2023-11-22 03:02:53,956 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0
2023-11-22 03:02:58,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1769606.6666666667, ans=0.95
2023-11-22 03:03:04,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265450
2023-11-22 03:03:16,278 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:03:17,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1769673.3333333333, ans=0.1
2023-11-22 03:03:33,891 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 950, loss[loss=0.07497, simple_loss=0.1114, pruned_loss=0.01487, audio_tagging_loss=0.004416, over 15813.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09409, pruned_loss=0.0157, audio_tagging_loss=0.009673, over 3028455.71 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:04:04,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1769940.0, ans=0.0
2023-11-22 03:04:08,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265500
2023-11-22 03:04:39,193 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1000, loss[loss=0.1013, simple_loss=0.1336, pruned_loss=0.02724, audio_tagging_loss=0.007247, over 16577.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09519, pruned_loss=0.01596, audio_tagging_loss=0.009543, over 3036946.23 frames. ], batch size: 59, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:04:40,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1770140.0, ans=0.09899494936611666
2023-11-22 03:04:54,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1770206.6666666667, ans=0.1
2023-11-22 03:04:57,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1770206.6666666667, ans=0.125
2023-11-22 03:04:57,702 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.69 vs. limit=15.0
2023-11-22 03:04:59,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1770206.6666666667, ans=0.0
2023-11-22 03:05:00,672 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.104e+01 8.710e+01 9.492e+01 1.362e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 03:05:02,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1770206.6666666667, ans=0.125
2023-11-22 03:05:05,587 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:05:12,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.60 vs. limit=15.0
2023-11-22 03:05:13,775 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265550
2023-11-22 03:05:16,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1770340.0, ans=0.0
2023-11-22 03:05:28,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1770340.0, ans=0.125
2023-11-22 03:05:43,705 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1050, loss[loss=0.06908, simple_loss=0.09194, pruned_loss=0.01071, audio_tagging_loss=0.0124, over 16043.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09448, pruned_loss=0.01604, audio_tagging_loss=0.009449, over 3037271.33 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:06:15,293 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.62 vs. limit=22.5
2023-11-22 03:06:19,141 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265600
2023-11-22 03:06:36,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1770740.0, ans=0.125
2023-11-22 03:06:49,353 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1100, loss[loss=0.1075, simple_loss=0.1371, pruned_loss=0.02988, audio_tagging_loss=0.009049, over 14997.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.0942, pruned_loss=0.01581, audio_tagging_loss=0.009443, over 3035237.18 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:06:51,763 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:06:53,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1770806.6666666667, ans=0.0
2023-11-22 03:07:05,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.05 vs. limit=15.0
2023-11-22 03:07:11,641 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 7.983e+01 8.697e+01 9.396e+01 1.258e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 03:07:13,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1770873.3333333333, ans=0.5
2023-11-22 03:07:15,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1770940.0, ans=0.125
2023-11-22 03:07:24,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265650
2023-11-22 03:07:37,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1771006.6666666667, ans=0.125
2023-11-22 03:07:37,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1771006.6666666667, ans=0.0
2023-11-22 03:07:48,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1771073.3333333333, ans=0.0
2023-11-22 03:07:54,473 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1150, loss[loss=0.07462, simple_loss=0.104, pruned_loss=0.01525, audio_tagging_loss=0.00738, over 15074.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09374, pruned_loss=0.0157, audio_tagging_loss=0.009469, over 3037121.77 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:07:54,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1771140.0, ans=0.125
2023-11-22 03:07:57,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1771140.0, ans=0.07
2023-11-22 03:08:04,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1771140.0, ans=0.125
2023-11-22 03:08:29,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265700
2023-11-22 03:08:41,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1771340.0, ans=0.1
2023-11-22 03:08:48,020 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:08:59,595 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.86 vs. limit=15.0
2023-11-22 03:08:59,955 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1200, loss[loss=0.07667, simple_loss=0.106, pruned_loss=0.01695, audio_tagging_loss=0.006716, over 15361.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09375, pruned_loss=0.01567, audio_tagging_loss=0.00952, over 3036616.40 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:09:00,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.54 vs. limit=15.0
2023-11-22 03:09:20,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1771540.0, ans=0.05
2023-11-22 03:09:21,603 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.059e+01 8.626e+01 9.376e+01 1.132e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 03:09:32,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1771606.6666666667, ans=0.05
2023-11-22 03:09:34,727 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265750
2023-11-22 03:09:44,827 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.31 vs. limit=15.0
2023-11-22 03:09:51,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1771740.0, ans=0.0
2023-11-22 03:10:01,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1771740.0, ans=0.05
2023-11-22 03:10:04,602 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1250, loss[loss=0.08373, simple_loss=0.1167, pruned_loss=0.01834, audio_tagging_loss=0.00706, over 15743.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09379, pruned_loss=0.0156, audio_tagging_loss=0.009569, over 3038797.89 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:10:04,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1771806.6666666667, ans=0.125
2023-11-22 03:10:11,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1771806.6666666667, ans=0.0
2023-11-22 03:10:21,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1771873.3333333333, ans=0.125
2023-11-22 03:10:24,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1771873.3333333333, ans=0.0
2023-11-22 03:10:26,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1771873.3333333333, ans=0.125
2023-11-22 03:10:29,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1771940.0, ans=0.125
2023-11-22 03:10:39,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265800
2023-11-22 03:11:04,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.39 vs. limit=12.0
2023-11-22 03:11:10,108 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1300, loss[loss=0.08596, simple_loss=0.1172, pruned_loss=0.01896, audio_tagging_loss=0.008395, over 15198.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09361, pruned_loss=0.01554, audio_tagging_loss=0.009516, over 3039691.35 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:11:18,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.67 vs. limit=22.5
2023-11-22 03:11:32,788 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.081e+01 8.746e+01 9.574e+01 1.188e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 03:11:40,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1772273.3333333333, ans=0.125
2023-11-22 03:11:45,786 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265850
2023-11-22 03:11:57,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.12 vs. limit=15.0
2023-11-22 03:11:58,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1772340.0, ans=0.5
2023-11-22 03:12:08,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=1772406.6666666667, ans=0.02
2023-11-22 03:12:14,915 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1350, loss[loss=0.07844, simple_loss=0.1042, pruned_loss=0.02019, audio_tagging_loss=0.006159, over 15648.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09411, pruned_loss=0.01554, audio_tagging_loss=0.009355, over 3044227.85 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:12:20,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff2.min_abs, batch_count=1772473.3333333333, ans=0.1
2023-11-22 03:12:49,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1772606.6666666667, ans=0.2
2023-11-22 03:12:49,963 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265900
2023-11-22 03:12:51,629 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.38 vs. limit=12.0
2023-11-22 03:12:52,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1772673.3333333333, ans=0.125
2023-11-22 03:12:53,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1772673.3333333333, ans=0.125
2023-11-22 03:12:53,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1772673.3333333333, ans=10.0
2023-11-22 03:12:56,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1772673.3333333333, ans=0.125
2023-11-22 03:13:01,471 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:13:10,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.37 vs. limit=6.0
2023-11-22 03:13:13,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1772740.0, ans=0.1
2023-11-22 03:13:19,373 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1400, loss[loss=0.07456, simple_loss=0.09116, pruned_loss=0.01817, audio_tagging_loss=0.01081, over 14970.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09396, pruned_loss=0.01536, audio_tagging_loss=0.009464, over 3047779.31 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:13:41,397 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.105e+01 8.866e+01 9.820e+01 1.190e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 03:13:49,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1772940.0, ans=0.1
2023-11-22 03:13:54,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 265950
2023-11-22 03:14:08,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1773006.6666666667, ans=0.0
2023-11-22 03:14:09,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1773073.3333333333, ans=10.0
2023-11-22 03:14:16,979 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.22 vs. limit=22.5
2023-11-22 03:14:19,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1773073.3333333333, ans=0.125
2023-11-22 03:14:21,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1773073.3333333333, ans=0.125
2023-11-22 03:14:22,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1773140.0, ans=0.125
2023-11-22 03:14:23,828 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1450, loss[loss=0.06482, simple_loss=0.08198, pruned_loss=0.0141, audio_tagging_loss=0.00973, over 15160.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09386, pruned_loss=0.0155, audio_tagging_loss=0.00953, over 3041765.94 frames. ], batch size: 60, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:14:29,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1773140.0, ans=0.125
2023-11-22 03:14:35,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1773206.6666666667, ans=0.125
2023-11-22 03:14:44,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1773206.6666666667, ans=0.125
2023-11-22 03:14:58,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266000
2023-11-22 03:14:58,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1773273.3333333333, ans=0.5
2023-11-22 03:15:02,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1773340.0, ans=0.1
2023-11-22 03:15:03,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1773340.0, ans=0.1
2023-11-22 03:15:26,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1773406.6666666667, ans=0.0
2023-11-22 03:15:28,974 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1500, loss[loss=0.05996, simple_loss=0.07485, pruned_loss=0.01387, audio_tagging_loss=0.008662, over 15983.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09442, pruned_loss=0.0156, audio_tagging_loss=0.00944, over 3045872.80 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:15:33,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.88 vs. limit=15.0
2023-11-22 03:15:41,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1773540.0, ans=0.125
2023-11-22 03:15:50,682 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.23 vs. limit=15.0
2023-11-22 03:15:51,114 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.300e+01 8.986e+01 9.912e+01 1.319e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 03:15:56,668 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.18 vs. limit=6.0
2023-11-22 03:16:04,213 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266050
2023-11-22 03:16:20,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1773740.0, ans=0.0
2023-11-22 03:16:22,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1773740.0, ans=0.1
2023-11-22 03:16:23,387 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.66 vs. limit=6.0
2023-11-22 03:16:32,112 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:16:34,340 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1550, loss[loss=0.06151, simple_loss=0.08212, pruned_loss=0.01281, audio_tagging_loss=0.007641, over 15152.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09553, pruned_loss=0.01596, audio_tagging_loss=0.009536, over 3042694.98 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:16:42,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1773806.6666666667, ans=0.125
2023-11-22 03:17:07,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1773940.0, ans=0.125
2023-11-22 03:17:09,207 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266100
2023-11-22 03:17:13,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1774006.6666666667, ans=0.2
2023-11-22 03:17:19,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1774006.6666666667, ans=0.0
2023-11-22 03:17:38,628 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1600, loss[loss=0.08205, simple_loss=0.1177, pruned_loss=0.0159, audio_tagging_loss=0.007325, over 15042.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09562, pruned_loss=0.01587, audio_tagging_loss=0.009597, over 3043650.13 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:17:40,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1774140.0, ans=0.1
2023-11-22 03:17:54,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1774206.6666666667, ans=0.125
2023-11-22 03:17:56,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1774206.6666666667, ans=0.1
2023-11-22 03:18:00,642 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.210e+01 8.853e+01 9.521e+01 1.233e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 03:18:11,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1774273.3333333333, ans=10.0
2023-11-22 03:18:13,982 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266150
2023-11-22 03:18:19,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1774340.0, ans=0.125
2023-11-22 03:18:41,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=1774406.6666666667, ans=0.02
2023-11-22 03:18:43,459 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1650, loss[loss=0.09249, simple_loss=0.1275, pruned_loss=0.02087, audio_tagging_loss=0.007877, over 15145.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09538, pruned_loss=0.01588, audio_tagging_loss=0.00968, over 3043041.49 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:19:04,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1774540.0, ans=0.0
2023-11-22 03:19:15,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1774606.6666666667, ans=0.125
2023-11-22 03:19:18,958 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266200
2023-11-22 03:19:20,868 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.41 vs. limit=22.5
2023-11-22 03:19:32,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1774673.3333333333, ans=0.1
2023-11-22 03:19:48,719 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1700, loss[loss=0.06711, simple_loss=0.08541, pruned_loss=0.01413, audio_tagging_loss=0.01028, over 15100.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09504, pruned_loss=0.01579, audio_tagging_loss=0.009685, over 3045687.26 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:19:50,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1774806.6666666667, ans=0.125
2023-11-22 03:19:58,188 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.49 vs. limit=15.0
2023-11-22 03:19:58,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1774806.6666666667, ans=0.125
2023-11-22 03:20:05,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.29 vs. limit=15.0
2023-11-22 03:20:11,471 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.074e+01 8.655e+01 9.245e+01 1.270e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 03:20:15,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1774940.0, ans=0.125
2023-11-22 03:20:19,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1774940.0, ans=0.0
2023-11-22 03:20:21,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1774940.0, ans=0.1
2023-11-22 03:20:24,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266250
2023-11-22 03:20:31,403 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0
2023-11-22 03:20:36,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1775006.6666666667, ans=0.0
2023-11-22 03:20:47,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.92 vs. limit=15.0
2023-11-22 03:20:54,026 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1750, loss[loss=0.06015, simple_loss=0.07933, pruned_loss=0.01323, audio_tagging_loss=0.007255, over 16173.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09443, pruned_loss=0.0157, audio_tagging_loss=0.009574, over 3046144.43 frames. ], batch size: 60, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:21:28,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266300
2023-11-22 03:21:39,554 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.31 vs. limit=15.0
2023-11-22 03:21:49,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1775406.6666666667, ans=0.125
2023-11-22 03:21:51,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1775406.6666666667, ans=0.1
2023-11-22 03:21:58,586 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1800, loss[loss=0.06273, simple_loss=0.08431, pruned_loss=0.01313, audio_tagging_loss=0.007444, over 14748.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09398, pruned_loss=0.01563, audio_tagging_loss=0.009521, over 3049541.88 frames. ], batch size: 54, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:22:21,025 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 7.973e+01 8.651e+01 9.483e+01 1.357e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 03:22:24,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1775606.6666666667, ans=0.125
2023-11-22 03:22:33,279 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266350
2023-11-22 03:22:38,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1775673.3333333333, ans=0.125
2023-11-22 03:22:43,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1775673.3333333333, ans=0.125
2023-11-22 03:22:53,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1775740.0, ans=0.1
2023-11-22 03:22:56,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1775740.0, ans=0.125
2023-11-22 03:23:01,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1775806.6666666667, ans=0.2
2023-11-22 03:23:02,250 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1850, loss[loss=0.07015, simple_loss=0.09405, pruned_loss=0.01699, audio_tagging_loss=0.006131, over 15313.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09494, pruned_loss=0.01582, audio_tagging_loss=0.009412, over 3048612.25 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:23:26,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1775873.3333333333, ans=0.0
2023-11-22 03:23:37,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266400
2023-11-22 03:23:46,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1776006.6666666667, ans=0.125
2023-11-22 03:23:49,452 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.52 vs. limit=15.0
2023-11-22 03:24:06,613 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1900, loss[loss=0.08547, simple_loss=0.112, pruned_loss=0.02173, audio_tagging_loss=0.007729, over 14896.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09488, pruned_loss=0.01588, audio_tagging_loss=0.009415, over 3047635.80 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:24:16,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1776140.0, ans=0.125
2023-11-22 03:24:25,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1776206.6666666667, ans=0.0
2023-11-22 03:24:26,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.15 vs. limit=22.5
2023-11-22 03:24:30,459 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.182e+01 8.329e+01 9.047e+01 9.606e+01 1.236e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 03:24:34,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1776273.3333333333, ans=0.125
2023-11-22 03:24:41,906 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266450
2023-11-22 03:25:01,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1776406.6666666667, ans=0.2
2023-11-22 03:25:09,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1776406.6666666667, ans=0.0
2023-11-22 03:25:11,712 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 1950, loss[loss=0.05852, simple_loss=0.0769, pruned_loss=0.01172, audio_tagging_loss=0.008349, over 16459.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09512, pruned_loss=0.01581, audio_tagging_loss=0.009322, over 3048299.81 frames. ], batch size: 63, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:25:14,434 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:25:20,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.14 vs. limit=22.5
2023-11-22 03:25:23,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1776540.0, ans=0.125
2023-11-22 03:25:27,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1776540.0, ans=0.0
2023-11-22 03:25:28,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1776540.0, ans=0.125
2023-11-22 03:25:30,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.whiten.whitening_limit, batch_count=1776540.0, ans=12.0
2023-11-22 03:25:41,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1776606.6666666667, ans=0.0
2023-11-22 03:25:45,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=1776606.6666666667, ans=0.02
2023-11-22 03:25:46,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266500
2023-11-22 03:25:49,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1776673.3333333333, ans=0.125
2023-11-22 03:26:16,492 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2000, loss[loss=0.0777, simple_loss=0.1054, pruned_loss=0.01726, audio_tagging_loss=0.007735, over 14762.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.0943, pruned_loss=0.01555, audio_tagging_loss=0.009275, over 3044787.24 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:26:39,145 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.529e+01 7.854e+01 8.461e+01 9.076e+01 1.159e+02, threshold=1.692e+02, percent-clipped=0.0
2023-11-22 03:26:51,401 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266550
2023-11-22 03:26:51,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1776940.0, ans=0.0
2023-11-22 03:27:02,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1777006.6666666667, ans=10.0
2023-11-22 03:27:06,362 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.97 vs. limit=22.5
2023-11-22 03:27:07,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1777073.3333333333, ans=0.125
2023-11-22 03:27:18,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1777073.3333333333, ans=0.0
2023-11-22 03:27:21,307 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2050, loss[loss=0.08233, simple_loss=0.1033, pruned_loss=0.02034, audio_tagging_loss=0.01032, over 14468.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09385, pruned_loss=0.01548, audio_tagging_loss=0.00934, over 3042893.91 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:27:56,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266600
2023-11-22 03:28:17,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1777406.6666666667, ans=0.2
2023-11-22 03:28:27,074 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2100, loss[loss=0.0602, simple_loss=0.0782, pruned_loss=0.01067, audio_tagging_loss=0.01042, over 15572.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09462, pruned_loss=0.01567, audio_tagging_loss=0.00932, over 3046230.36 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:28:40,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1777540.0, ans=0.125
2023-11-22 03:28:47,833 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.319e-02
2023-11-22 03:28:49,896 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.317e+01 8.958e+01 9.611e+01 1.215e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 03:28:59,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1777606.6666666667, ans=0.125
2023-11-22 03:29:01,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266650
2023-11-22 03:29:20,758 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.51 vs. limit=22.5
2023-11-22 03:29:25,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1777740.0, ans=0.125
2023-11-22 03:29:31,480 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2150, loss[loss=0.05563, simple_loss=0.0646, pruned_loss=0.01237, audio_tagging_loss=0.01097, over 15092.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09367, pruned_loss=0.01559, audio_tagging_loss=0.009349, over 3046999.84 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:30:07,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266700
2023-11-22 03:30:10,783 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:30:15,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1778006.6666666667, ans=0.1
2023-11-22 03:30:36,532 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2200, loss[loss=0.07011, simple_loss=0.09186, pruned_loss=0.01638, audio_tagging_loss=0.007801, over 15252.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09421, pruned_loss=0.01566, audio_tagging_loss=0.009322, over 3045952.88 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:30:49,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.45 vs. limit=15.0
2023-11-22 03:30:52,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.08 vs. limit=15.0
2023-11-22 03:31:01,201 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.015e+01 8.446e+01 9.352e+01 1.016e+02 2.574e+02, threshold=1.870e+02, percent-clipped=1.0
2023-11-22 03:31:08,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1778273.3333333333, ans=0.0
2023-11-22 03:31:12,200 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266750
2023-11-22 03:31:19,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.whiten.whitening_limit, batch_count=1778340.0, ans=12.0
2023-11-22 03:31:36,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1778406.6666666667, ans=0.0
2023-11-22 03:31:41,879 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2250, loss[loss=0.07267, simple_loss=0.09193, pruned_loss=0.01783, audio_tagging_loss=0.008867, over 14623.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.0945, pruned_loss=0.0155, audio_tagging_loss=0.009355, over 3040079.16 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:31:56,003 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.12 vs. limit=6.0
2023-11-22 03:32:17,387 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266800
2023-11-22 03:32:22,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1778673.3333333333, ans=0.1
2023-11-22 03:32:47,669 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2300, loss[loss=0.06868, simple_loss=0.08385, pruned_loss=0.01713, audio_tagging_loss=0.009623, over 13607.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09491, pruned_loss=0.01555, audio_tagging_loss=0.009444, over 3037810.69 frames. ], batch size: 54, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:33:02,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1778873.3333333333, ans=0.125
2023-11-22 03:33:11,809 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.550e+01 8.208e+01 8.733e+01 9.600e+01 1.238e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 03:33:22,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266850
2023-11-22 03:33:26,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1779006.6666666667, ans=0.0
2023-11-22 03:33:43,866 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:33:52,758 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2350, loss[loss=0.07255, simple_loss=0.08764, pruned_loss=0.01725, audio_tagging_loss=0.01148, over 14976.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09473, pruned_loss=0.0155, audio_tagging_loss=0.009578, over 3040109.89 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:34:19,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1779273.3333333333, ans=0.125
2023-11-22 03:34:24,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1779273.3333333333, ans=0.07
2023-11-22 03:34:28,371 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266900
2023-11-22 03:34:52,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1779406.6666666667, ans=0.1
2023-11-22 03:34:57,448 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2400, loss[loss=0.05915, simple_loss=0.07752, pruned_loss=0.01037, audio_tagging_loss=0.01002, over 14840.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09433, pruned_loss=0.01553, audio_tagging_loss=0.009711, over 3047582.31 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:35:02,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1779473.3333333333, ans=0.125
2023-11-22 03:35:22,843 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 7.925e+01 8.589e+01 9.306e+01 1.063e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 03:35:31,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1779606.6666666667, ans=0.125
2023-11-22 03:35:33,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 266950
2023-11-22 03:35:40,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1779673.3333333333, ans=0.0
2023-11-22 03:36:00,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1779740.0, ans=0.0
2023-11-22 03:36:02,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1779806.6666666667, ans=0.0
2023-11-22 03:36:03,426 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2450, loss[loss=0.04727, simple_loss=0.05261, pruned_loss=0.007978, audio_tagging_loss=0.01299, over 15011.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09425, pruned_loss=0.01544, audio_tagging_loss=0.009724, over 3049555.50 frames. ], batch size: 60, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:36:05,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1779806.6666666667, ans=0.0
2023-11-22 03:36:18,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1779873.3333333333, ans=0.07
2023-11-22 03:36:35,724 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.61 vs. limit=15.0
2023-11-22 03:36:39,258 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267000
2023-11-22 03:36:44,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1780006.6666666667, ans=0.0
2023-11-22 03:36:46,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1780006.6666666667, ans=0.07
2023-11-22 03:36:47,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1780006.6666666667, ans=0.1
2023-11-22 03:36:50,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1780006.6666666667, ans=0.125
2023-11-22 03:37:03,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1780073.3333333333, ans=0.0
2023-11-22 03:37:09,370 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2500, loss[loss=0.07435, simple_loss=0.0972, pruned_loss=0.01639, audio_tagging_loss=0.009357, over 16792.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09436, pruned_loss=0.01542, audio_tagging_loss=0.009752, over 3045600.47 frames. ], batch size: 64, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:37:34,051 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.102e+01 8.757e+01 9.446e+01 1.143e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 03:37:45,129 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267050
2023-11-22 03:38:06,846 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.82 vs. limit=15.0
2023-11-22 03:38:07,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1780406.6666666667, ans=0.125
2023-11-22 03:38:14,929 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2550, loss[loss=0.05075, simple_loss=0.06863, pruned_loss=0.00918, audio_tagging_loss=0.007254, over 14838.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09475, pruned_loss=0.01573, audio_tagging_loss=0.009609, over 3042438.56 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:38:15,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1780473.3333333333, ans=0.125
2023-11-22 03:38:27,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1780540.0, ans=0.2
2023-11-22 03:38:32,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1780540.0, ans=0.125
2023-11-22 03:38:44,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1780606.6666666667, ans=0.125
2023-11-22 03:38:50,685 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267100
2023-11-22 03:39:18,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.45 vs. limit=6.0
2023-11-22 03:39:20,360 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2600, loss[loss=0.0557, simple_loss=0.07091, pruned_loss=0.009752, audio_tagging_loss=0.01049, over 14693.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09493, pruned_loss=0.0159, audio_tagging_loss=0.009473, over 3042701.14 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:39:45,356 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.768e+01 8.100e+01 8.795e+01 9.749e+01 1.306e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 03:39:49,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1780940.0, ans=0.0
2023-11-22 03:39:55,278 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267150
2023-11-22 03:40:20,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1781073.3333333333, ans=0.0
2023-11-22 03:40:25,441 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2650, loss[loss=0.07608, simple_loss=0.1003, pruned_loss=0.01628, audio_tagging_loss=0.009644, over 15062.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09411, pruned_loss=0.01574, audio_tagging_loss=0.00952, over 3041937.20 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:40:49,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1781206.6666666667, ans=0.125
2023-11-22 03:41:00,650 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267200
2023-11-22 03:41:27,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1781406.6666666667, ans=0.1
2023-11-22 03:41:30,833 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2700, loss[loss=0.08558, simple_loss=0.111, pruned_loss=0.0228, audio_tagging_loss=0.00728, over 16260.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09399, pruned_loss=0.01552, audio_tagging_loss=0.009449, over 3050719.08 frames. ], batch size: 61, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:41:31,791 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.33 vs. limit=15.0
2023-11-22 03:41:34,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1781473.3333333333, ans=0.125
2023-11-22 03:41:57,246 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.172e+01 8.019e+01 8.519e+01 9.203e+01 1.202e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-22 03:42:06,786 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267250
2023-11-22 03:42:20,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1781673.3333333333, ans=0.1
2023-11-22 03:42:22,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1781740.0, ans=0.05
2023-11-22 03:42:36,528 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2750, loss[loss=0.07263, simple_loss=0.09978, pruned_loss=0.01511, audio_tagging_loss=0.007632, over 15118.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09378, pruned_loss=0.01537, audio_tagging_loss=0.009397, over 3045651.75 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:42:43,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1781806.6666666667, ans=0.125
2023-11-22 03:42:51,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.85 vs. limit=15.0
2023-11-22 03:42:53,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1781873.3333333333, ans=0.1
2023-11-22 03:43:03,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1781940.0, ans=0.1
2023-11-22 03:43:11,859 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267300
2023-11-22 03:43:17,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1782006.6666666667, ans=0.125
2023-11-22 03:43:32,639 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:43:38,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1782073.3333333333, ans=0.125
2023-11-22 03:43:41,243 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2800, loss[loss=0.05434, simple_loss=0.066, pruned_loss=0.007374, audio_tagging_loss=0.01396, over 15116.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.09321, pruned_loss=0.01541, audio_tagging_loss=0.009466, over 3040996.91 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:43:49,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1782140.0, ans=0.1
2023-11-22 03:43:51,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1782140.0, ans=0.0
2023-11-22 03:43:54,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1782206.6666666667, ans=0.125
2023-11-22 03:43:56,921 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.34 vs. limit=15.0
2023-11-22 03:43:58,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1782206.6666666667, ans=0.2
2023-11-22 03:44:05,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1782206.6666666667, ans=0.1
2023-11-22 03:44:07,806 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.173e+01 8.676e+01 9.379e+01 1.430e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 03:44:16,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267350
2023-11-22 03:44:27,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.59 vs. limit=22.5
2023-11-22 03:44:47,371 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2850, loss[loss=0.07507, simple_loss=0.1027, pruned_loss=0.01427, audio_tagging_loss=0.009454, over 14809.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09421, pruned_loss=0.01549, audio_tagging_loss=0.009445, over 3041289.16 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:45:15,764 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-22 03:45:22,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267400
2023-11-22 03:45:25,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1782673.3333333333, ans=0.125
2023-11-22 03:45:35,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1782673.3333333333, ans=0.0
2023-11-22 03:45:52,361 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2900, loss[loss=0.0642, simple_loss=0.07637, pruned_loss=0.01455, audio_tagging_loss=0.01147, over 14720.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09429, pruned_loss=0.01562, audio_tagging_loss=0.009332, over 3045989.54 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:46:00,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1782806.6666666667, ans=0.125
2023-11-22 03:46:01,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1782806.6666666667, ans=0.125
2023-11-22 03:46:06,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1782873.3333333333, ans=0.0
2023-11-22 03:46:10,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1782873.3333333333, ans=0.125
2023-11-22 03:46:10,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1782873.3333333333, ans=0.0
2023-11-22 03:46:19,361 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.173e+01 8.778e+01 9.450e+01 1.328e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 03:46:27,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267450
2023-11-22 03:46:27,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1782940.0, ans=0.125
2023-11-22 03:46:30,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1783006.6666666667, ans=0.1
2023-11-22 03:46:31,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.90 vs. limit=6.0
2023-11-22 03:46:32,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1783006.6666666667, ans=0.0
2023-11-22 03:46:51,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1783073.3333333333, ans=0.0
2023-11-22 03:46:52,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.49 vs. limit=22.5
2023-11-22 03:46:56,576 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 2950, loss[loss=0.05718, simple_loss=0.07828, pruned_loss=0.00699, audio_tagging_loss=0.01105, over 15590.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.0947, pruned_loss=0.01556, audio_tagging_loss=0.009361, over 3048787.54 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:47:00,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1783140.0, ans=0.0
2023-11-22 03:47:04,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-22 03:47:05,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1783140.0, ans=0.1
2023-11-22 03:47:20,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.whiten.whitening_limit, batch_count=1783206.6666666667, ans=12.0
2023-11-22 03:47:31,619 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267500
2023-11-22 03:47:41,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1783340.0, ans=0.125
2023-11-22 03:47:50,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1783406.6666666667, ans=0.125
2023-11-22 03:47:54,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1783406.6666666667, ans=0.1
2023-11-22 03:47:55,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1783406.6666666667, ans=0.2
2023-11-22 03:48:01,465 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3000, loss[loss=0.06298, simple_loss=0.07989, pruned_loss=0.01177, audio_tagging_loss=0.01126, over 14663.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09537, pruned_loss=0.0157, audio_tagging_loss=0.009356, over 3047300.86 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:48:01,466 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 03:48:41,400 INFO [train_asr.py:1253] (2/4) Epoch 23, validation: loss=0.05946, simple_loss=0.05181, pruned_loss=0.005129, audio_tagging_loss=0.02843, over 4681554.00 frames. 
2023-11-22 03:48:41,401 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 03:49:01,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.12 vs. limit=15.0
2023-11-22 03:49:07,729 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.252e+01 8.878e+01 9.473e+01 1.263e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-22 03:49:10,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1783606.6666666667, ans=0.125
2023-11-22 03:49:15,831 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267550
2023-11-22 03:49:30,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1783673.3333333333, ans=0.125
2023-11-22 03:49:34,573 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.03 vs. limit=6.0
2023-11-22 03:49:44,979 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3050, loss[loss=0.08063, simple_loss=0.1073, pruned_loss=0.01806, audio_tagging_loss=0.008942, over 15765.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09554, pruned_loss=0.01577, audio_tagging_loss=0.009292, over 3045757.92 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 8.0
2023-11-22 03:50:20,640 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267600
2023-11-22 03:50:23,299 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:50:23,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1784006.6666666667, ans=0.0
2023-11-22 03:50:28,569 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:50:31,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1784006.6666666667, ans=0.2
2023-11-22 03:50:34,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1784006.6666666667, ans=0.07
2023-11-22 03:50:49,185 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3100, loss[loss=0.07358, simple_loss=0.09481, pruned_loss=0.01632, audio_tagging_loss=0.009855, over 14502.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09557, pruned_loss=0.01574, audio_tagging_loss=0.009378, over 3043623.42 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 8.0
2023-11-22 03:50:49,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1784140.0, ans=0.0
2023-11-22 03:50:59,027 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.28 vs. limit=15.0
2023-11-22 03:51:15,924 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.11 vs. limit=15.0
2023-11-22 03:51:17,604 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.039e+01 8.127e+01 8.791e+01 9.439e+01 1.386e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 03:51:18,195 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.76 vs. limit=15.0
2023-11-22 03:51:23,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267650
2023-11-22 03:51:34,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1784340.0, ans=0.125
2023-11-22 03:51:35,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1784340.0, ans=0.1
2023-11-22 03:51:44,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1784406.6666666667, ans=0.125
2023-11-22 03:51:53,225 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3150, loss[loss=0.07478, simple_loss=0.1071, pruned_loss=0.01331, audio_tagging_loss=0.007908, over 15871.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09582, pruned_loss=0.01558, audio_tagging_loss=0.009449, over 3041371.43 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 8.0
2023-11-22 03:52:04,216 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.89 vs. limit=15.0
2023-11-22 03:52:06,573 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.88 vs. limit=15.0
2023-11-22 03:52:11,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1784540.0, ans=0.025
2023-11-22 03:52:14,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1784540.0, ans=0.2
2023-11-22 03:52:17,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1784606.6666666667, ans=0.125
2023-11-22 03:52:27,538 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267700
2023-11-22 03:52:37,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1784673.3333333333, ans=0.04949747468305833
2023-11-22 03:52:37,887 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=26.63 vs. limit=22.5
2023-11-22 03:52:41,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1784673.3333333333, ans=0.125
2023-11-22 03:52:49,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1784740.0, ans=0.05
2023-11-22 03:52:52,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1784740.0, ans=0.1
2023-11-22 03:52:57,571 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3200, loss[loss=0.05331, simple_loss=0.06365, pruned_loss=0.006943, audio_tagging_loss=0.01454, over 15747.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09575, pruned_loss=0.01576, audio_tagging_loss=0.009545, over 3040855.09 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:53:21,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1784940.0, ans=0.1
2023-11-22 03:53:23,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.29 vs. limit=10.0
2023-11-22 03:53:26,394 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.278e+01 8.640e+01 9.610e+01 1.233e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-22 03:53:33,298 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267750
2023-11-22 03:54:01,748 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3250, loss[loss=0.06938, simple_loss=0.08391, pruned_loss=0.01581, audio_tagging_loss=0.01162, over 14869.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09593, pruned_loss=0.01574, audio_tagging_loss=0.009567, over 3047094.79 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:54:18,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1785206.6666666667, ans=0.1
2023-11-22 03:54:18,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.84 vs. limit=10.0
2023-11-22 03:54:34,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1785273.3333333333, ans=0.1
2023-11-22 03:54:35,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267800
2023-11-22 03:55:06,140 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3300, loss[loss=0.08537, simple_loss=0.1135, pruned_loss=0.02125, audio_tagging_loss=0.007349, over 16181.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09524, pruned_loss=0.01564, audio_tagging_loss=0.009595, over 3049869.46 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:55:18,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1785540.0, ans=0.125
2023-11-22 03:55:20,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.23 vs. limit=15.0
2023-11-22 03:55:24,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1785540.0, ans=10.0
2023-11-22 03:55:34,163 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.246e+01 8.906e+01 9.569e+01 1.172e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 03:55:40,361 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267850
2023-11-22 03:55:42,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1785673.3333333333, ans=0.125
2023-11-22 03:55:42,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1785673.3333333333, ans=0.04949747468305833
2023-11-22 03:55:44,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.07 vs. limit=10.0
2023-11-22 03:55:45,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1785673.3333333333, ans=0.025
2023-11-22 03:56:01,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1785740.0, ans=0.125
2023-11-22 03:56:04,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1785740.0, ans=0.1
2023-11-22 03:56:08,025 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.66 vs. limit=12.0
2023-11-22 03:56:09,886 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3350, loss[loss=0.07434, simple_loss=0.1009, pruned_loss=0.0143, audio_tagging_loss=0.009611, over 14965.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09497, pruned_loss=0.0156, audio_tagging_loss=0.00963, over 3041889.26 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:56:12,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1785806.6666666667, ans=0.0
2023-11-22 03:56:16,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1785806.6666666667, ans=0.1
2023-11-22 03:56:25,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1785873.3333333333, ans=0.125
2023-11-22 03:56:42,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.47 vs. limit=15.0
2023-11-22 03:56:44,828 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267900
2023-11-22 03:56:54,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786006.6666666667, ans=0.1
2023-11-22 03:57:00,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1786073.3333333333, ans=0.07
2023-11-22 03:57:13,699 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3400, loss[loss=0.05798, simple_loss=0.08034, pruned_loss=0.006972, audio_tagging_loss=0.01084, over 14926.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09619, pruned_loss=0.01584, audio_tagging_loss=0.009426, over 3050853.21 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:57:22,958 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.78 vs. limit=15.0
2023-11-22 03:57:32,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1786206.6666666667, ans=0.125
2023-11-22 03:57:42,474 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.335e+01 8.871e+01 9.703e+01 1.398e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 03:57:48,666 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 267950
2023-11-22 03:57:48,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1786273.3333333333, ans=0.0
2023-11-22 03:58:00,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1786340.0, ans=0.1
2023-11-22 03:58:06,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1786406.6666666667, ans=0.0
2023-11-22 03:58:10,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1786406.6666666667, ans=0.05
2023-11-22 03:58:13,457 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.98 vs. limit=22.5
2023-11-22 03:58:18,182 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3450, loss[loss=0.063, simple_loss=0.08141, pruned_loss=0.01127, audio_tagging_loss=0.01103, over 14154.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09618, pruned_loss=0.01584, audio_tagging_loss=0.009298, over 3043623.94 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 03:58:19,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1786473.3333333333, ans=0.125
2023-11-22 03:58:23,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1786473.3333333333, ans=0.125
2023-11-22 03:58:30,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786540.0, ans=0.1
2023-11-22 03:58:34,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1786540.0, ans=0.125
2023-11-22 03:58:40,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1786540.0, ans=0.2
2023-11-22 03:58:41,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1786540.0, ans=0.125
2023-11-22 03:58:47,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1786606.6666666667, ans=0.2
2023-11-22 03:58:53,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268000
2023-11-22 03:59:08,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786673.3333333333, ans=0.1
2023-11-22 03:59:13,395 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0
2023-11-22 03:59:17,979 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.97 vs. limit=22.5
2023-11-22 03:59:25,609 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3500, loss[loss=0.1231, simple_loss=0.165, pruned_loss=0.03623, audio_tagging_loss=0.004366, over 14545.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09559, pruned_loss=0.01578, audio_tagging_loss=0.009262, over 3037945.76 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 03:59:49,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1786940.0, ans=0.0
2023-11-22 03:59:54,062 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.643e+01 7.936e+01 8.648e+01 9.422e+01 1.502e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 03:59:58,397 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:59:59,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268050
2023-11-22 04:00:24,098 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.63 vs. limit=15.0
2023-11-22 04:00:29,140 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3550, loss[loss=0.05992, simple_loss=0.0796, pruned_loss=0.01227, audio_tagging_loss=0.007844, over 14833.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09456, pruned_loss=0.01571, audio_tagging_loss=0.009213, over 3038396.84 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:00:47,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1787206.6666666667, ans=0.125
2023-11-22 04:00:52,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1787206.6666666667, ans=0.125
2023-11-22 04:01:03,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268100
2023-11-22 04:01:04,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1787273.3333333333, ans=0.125
2023-11-22 04:01:23,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1787406.6666666667, ans=0.0
2023-11-22 04:01:27,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1787406.6666666667, ans=0.04949747468305833
2023-11-22 04:01:31,987 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3600, loss[loss=0.06229, simple_loss=0.07828, pruned_loss=0.0123, audio_tagging_loss=0.01086, over 15938.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09442, pruned_loss=0.01567, audio_tagging_loss=0.009321, over 3037490.85 frames. ], batch size: 63, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:01:33,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1787473.3333333333, ans=0.125
2023-11-22 04:01:37,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1787473.3333333333, ans=0.125
2023-11-22 04:01:58,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1787606.6666666667, ans=0.0
2023-11-22 04:02:01,923 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.327e+01 8.799e+01 9.625e+01 1.533e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 04:02:06,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1787606.6666666667, ans=0.125
2023-11-22 04:02:07,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268150
2023-11-22 04:02:13,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1787673.3333333333, ans=0.0
2023-11-22 04:02:22,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1787740.0, ans=0.125
2023-11-22 04:02:36,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1787806.6666666667, ans=0.125
2023-11-22 04:02:37,501 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3650, loss[loss=0.07161, simple_loss=0.08716, pruned_loss=0.02026, audio_tagging_loss=0.007776, over 14706.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09556, pruned_loss=0.01597, audio_tagging_loss=0.009266, over 3044352.75 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:02:41,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1787806.6666666667, ans=0.1
2023-11-22 04:02:51,568 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.94 vs. limit=15.0
2023-11-22 04:02:56,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1787873.3333333333, ans=0.125
2023-11-22 04:03:00,706 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=23.19 vs. limit=22.5
2023-11-22 04:03:09,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=15.0
2023-11-22 04:03:10,906 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268200
2023-11-22 04:03:18,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1788006.6666666667, ans=0.125
2023-11-22 04:03:36,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1788073.3333333333, ans=0.125
2023-11-22 04:03:39,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1788140.0, ans=0.0
2023-11-22 04:03:40,415 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3700, loss[loss=0.07587, simple_loss=0.09734, pruned_loss=0.01819, audio_tagging_loss=0.009007, over 15135.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09458, pruned_loss=0.01581, audio_tagging_loss=0.009332, over 3038828.20 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:04:09,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.85 vs. limit=22.5
2023-11-22 04:04:09,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.401e+01 8.835e+01 9.477e+01 1.225e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 04:04:15,638 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268250
2023-11-22 04:04:43,990 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3750, loss[loss=0.05227, simple_loss=0.06663, pruned_loss=0.008939, audio_tagging_loss=0.01001, over 15760.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09529, pruned_loss=0.01585, audio_tagging_loss=0.00929, over 3046684.21 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:04:52,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1788473.3333333333, ans=0.1
2023-11-22 04:05:18,426 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268300
2023-11-22 04:05:18,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1788606.6666666667, ans=0.125
2023-11-22 04:05:28,615 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:05:29,180 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.74 vs. limit=22.5
2023-11-22 04:05:48,208 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3800, loss[loss=0.06885, simple_loss=0.0898, pruned_loss=0.01397, audio_tagging_loss=0.00998, over 14891.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09534, pruned_loss=0.01569, audio_tagging_loss=0.00937, over 3050119.00 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:05:54,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1788806.6666666667, ans=0.0
2023-11-22 04:06:01,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1788873.3333333333, ans=0.125
2023-11-22 04:06:17,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1788940.0, ans=0.125
2023-11-22 04:06:18,647 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.152e+01 8.764e+01 9.512e+01 1.350e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 04:06:22,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268350
2023-11-22 04:06:23,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1788940.0, ans=0.125
2023-11-22 04:06:25,104 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:06:38,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1789073.3333333333, ans=0.0
2023-11-22 04:06:43,629 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.39 vs. limit=12.0
2023-11-22 04:06:47,924 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.58 vs. limit=22.5
2023-11-22 04:06:52,251 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3850, loss[loss=0.08895, simple_loss=0.1202, pruned_loss=0.02042, audio_tagging_loss=0.008442, over 15447.00 frames. ], tot_loss[loss=0.07285, simple_loss=0.09548, pruned_loss=0.01573, audio_tagging_loss=0.009384, over 3046516.18 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:06:53,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1789140.0, ans=0.09899494936611666
2023-11-22 04:06:56,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1789140.0, ans=0.125
2023-11-22 04:06:57,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1789140.0, ans=0.0
2023-11-22 04:07:07,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1789206.6666666667, ans=0.2
2023-11-22 04:07:09,007 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:07:18,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1789273.3333333333, ans=0.95
2023-11-22 04:07:25,518 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.51 vs. limit=15.0
2023-11-22 04:07:27,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268400
2023-11-22 04:07:52,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1789406.6666666667, ans=0.0
2023-11-22 04:07:57,336 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3900, loss[loss=0.09028, simple_loss=0.1161, pruned_loss=0.02232, audio_tagging_loss=0.009886, over 15360.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09591, pruned_loss=0.01584, audio_tagging_loss=0.009454, over 3043417.61 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:08:01,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.80 vs. limit=22.5
2023-11-22 04:08:03,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1789473.3333333333, ans=0.125
2023-11-22 04:08:15,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.41 vs. limit=15.0
2023-11-22 04:08:16,460 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.87 vs. limit=15.0
2023-11-22 04:08:28,732 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 8.044e+01 8.875e+01 9.539e+01 1.176e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 04:08:32,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268450
2023-11-22 04:08:32,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1789606.6666666667, ans=0.125
2023-11-22 04:08:33,085 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0
2023-11-22 04:08:33,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1789606.6666666667, ans=0.125
2023-11-22 04:08:40,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1789673.3333333333, ans=0.125
2023-11-22 04:08:55,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1789740.0, ans=0.125
2023-11-22 04:08:55,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1789740.0, ans=0.125
2023-11-22 04:09:01,972 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 3950, loss[loss=0.08818, simple_loss=0.1105, pruned_loss=0.02603, audio_tagging_loss=0.006895, over 14809.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09523, pruned_loss=0.01574, audio_tagging_loss=0.009619, over 3035725.50 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:09:02,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1789806.6666666667, ans=0.035
2023-11-22 04:09:22,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1789873.3333333333, ans=0.125
2023-11-22 04:09:25,779 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=15.0
2023-11-22 04:09:35,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1789940.0, ans=0.1
2023-11-22 04:09:35,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268500
2023-11-22 04:09:36,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1789940.0, ans=0.125
2023-11-22 04:10:04,737 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4000, loss[loss=0.08058, simple_loss=0.1092, pruned_loss=0.01398, audio_tagging_loss=0.01202, over 15308.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09572, pruned_loss=0.01596, audio_tagging_loss=0.009654, over 3040969.07 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:10:13,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1790140.0, ans=0.125
2023-11-22 04:10:13,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1790140.0, ans=0.07
2023-11-22 04:10:23,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1790206.6666666667, ans=0.1
2023-11-22 04:10:27,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1790206.6666666667, ans=0.0
2023-11-22 04:10:36,040 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.201e+01 8.416e+01 8.758e+01 9.509e+01 1.185e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 04:10:40,283 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268550
2023-11-22 04:11:06,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1790406.6666666667, ans=0.0
2023-11-22 04:11:09,602 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4050, loss[loss=0.07157, simple_loss=0.08861, pruned_loss=0.01667, audio_tagging_loss=0.01059, over 14287.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09618, pruned_loss=0.01609, audio_tagging_loss=0.009584, over 3035563.50 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:11:12,173 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:11:29,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1790540.0, ans=0.2
2023-11-22 04:11:30,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1790540.0, ans=0.125
2023-11-22 04:11:43,670 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268600
2023-11-22 04:12:08,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1790740.0, ans=0.125
2023-11-22 04:12:13,285 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4100, loss[loss=0.0737, simple_loss=0.09938, pruned_loss=0.01557, audio_tagging_loss=0.008439, over 14963.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09627, pruned_loss=0.01603, audio_tagging_loss=0.009504, over 3031758.27 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:12:16,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1790806.6666666667, ans=0.125
2023-11-22 04:12:31,169 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.45 vs. limit=10.0
2023-11-22 04:12:36,451 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.17 vs. limit=12.0
2023-11-22 04:12:39,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.34 vs. limit=22.5
2023-11-22 04:12:44,093 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.196e+01 8.304e+01 9.033e+01 9.578e+01 1.202e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-22 04:12:44,909 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.36 vs. limit=10.0
2023-11-22 04:12:47,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268650
2023-11-22 04:12:48,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0
2023-11-22 04:13:13,780 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:13:17,307 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4150, loss[loss=0.06119, simple_loss=0.07574, pruned_loss=0.01206, audio_tagging_loss=0.01126, over 14064.00 frames. ], tot_loss[loss=0.07369, simple_loss=0.09658, pruned_loss=0.01606, audio_tagging_loss=0.009343, over 3037518.18 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:13:17,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1791140.0, ans=0.125
2023-11-22 04:13:29,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1791206.6666666667, ans=0.2
2023-11-22 04:13:37,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1791206.6666666667, ans=0.0
2023-11-22 04:13:41,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.91 vs. limit=10.0
2023-11-22 04:13:52,787 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268700
2023-11-22 04:14:00,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.17 vs. limit=15.0
2023-11-22 04:14:03,565 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:14:09,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1791406.6666666667, ans=0.125
2023-11-22 04:14:12,906 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.30 vs. limit=15.0
2023-11-22 04:14:15,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1791406.6666666667, ans=0.0
2023-11-22 04:14:17,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1791406.6666666667, ans=0.125
2023-11-22 04:14:21,876 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4200, loss[loss=0.05485, simple_loss=0.06527, pruned_loss=0.01038, audio_tagging_loss=0.01183, over 15154.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09542, pruned_loss=0.01587, audio_tagging_loss=0.009346, over 3030981.44 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:14:31,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1791473.3333333333, ans=0.0
2023-11-22 04:14:32,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.12 vs. limit=6.0
2023-11-22 04:14:52,145 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.143e+01 8.798e+01 9.723e+01 1.732e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 04:14:55,952 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268750
2023-11-22 04:15:05,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1791673.3333333333, ans=0.05
2023-11-22 04:15:14,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.15 vs. limit=15.0
2023-11-22 04:15:25,749 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4250, loss[loss=0.07454, simple_loss=0.1054, pruned_loss=0.01587, audio_tagging_loss=0.005963, over 14901.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09512, pruned_loss=0.01571, audio_tagging_loss=0.00932, over 3040672.59 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:15:30,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.15 vs. limit=15.0
2023-11-22 04:15:36,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1791806.6666666667, ans=0.1
2023-11-22 04:15:42,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1791873.3333333333, ans=0.2
2023-11-22 04:15:50,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1791940.0, ans=0.125
2023-11-22 04:15:54,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1791940.0, ans=0.0
2023-11-22 04:16:00,502 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268800
2023-11-22 04:16:07,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1792006.6666666667, ans=0.0
2023-11-22 04:16:22,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1792073.3333333333, ans=0.125
2023-11-22 04:16:27,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1792073.3333333333, ans=10.0
2023-11-22 04:16:30,662 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4300, loss[loss=0.06791, simple_loss=0.09494, pruned_loss=0.01143, audio_tagging_loss=0.009013, over 15420.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09509, pruned_loss=0.01565, audio_tagging_loss=0.009321, over 3049842.07 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:17:01,429 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.380e+01 9.006e+01 9.846e+01 1.214e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 04:17:04,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1792273.3333333333, ans=0.125
2023-11-22 04:17:05,857 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268850
2023-11-22 04:17:11,506 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:17:35,221 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4350, loss[loss=0.0598, simple_loss=0.07896, pruned_loss=0.01173, audio_tagging_loss=0.008583, over 14323.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09469, pruned_loss=0.01552, audio_tagging_loss=0.009272, over 3043096.30 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:17:52,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1792540.0, ans=0.125
2023-11-22 04:17:56,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1792540.0, ans=0.0
2023-11-22 04:18:00,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1792606.6666666667, ans=0.0
2023-11-22 04:18:07,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.65 vs. limit=6.0
2023-11-22 04:18:10,659 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268900
2023-11-22 04:18:28,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.72 vs. limit=15.0
2023-11-22 04:18:30,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.94 vs. limit=12.0
2023-11-22 04:18:39,822 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4400, loss[loss=0.06318, simple_loss=0.07316, pruned_loss=0.01448, audio_tagging_loss=0.01213, over 15046.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09502, pruned_loss=0.01556, audio_tagging_loss=0.009253, over 3049607.75 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 32.0
2023-11-22 04:18:55,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1792873.3333333333, ans=0.0
2023-11-22 04:18:57,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1792873.3333333333, ans=0.125
2023-11-22 04:19:10,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 7.984e+01 8.605e+01 9.395e+01 1.319e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-22 04:19:14,629 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 268950
2023-11-22 04:19:29,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1793006.6666666667, ans=0.125
2023-11-22 04:19:33,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1793073.3333333333, ans=0.125
2023-11-22 04:19:34,802 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:19:45,070 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4450, loss[loss=0.0618, simple_loss=0.07454, pruned_loss=0.01277, audio_tagging_loss=0.01176, over 15872.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09533, pruned_loss=0.01573, audio_tagging_loss=0.00919, over 3049809.53 frames. ], batch size: 61, lr: 2.98e-03, grad_scale: 32.0
2023-11-22 04:19:52,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1793140.0, ans=0.1
2023-11-22 04:19:57,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1793206.6666666667, ans=0.2
2023-11-22 04:19:57,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1793206.6666666667, ans=0.09899494936611666
2023-11-22 04:20:11,658 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.022e-02
2023-11-22 04:20:19,917 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269000
2023-11-22 04:20:26,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1793340.0, ans=0.1
2023-11-22 04:20:49,467 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4500, loss[loss=0.09343, simple_loss=0.1322, pruned_loss=0.02029, audio_tagging_loss=0.007066, over 16264.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09582, pruned_loss=0.01583, audio_tagging_loss=0.009157, over 3049893.52 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:20:53,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1793473.3333333333, ans=0.2
2023-11-22 04:21:05,005 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.30 vs. limit=15.0
2023-11-22 04:21:19,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1793606.6666666667, ans=0.125
2023-11-22 04:21:23,203 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.138e+01 8.864e+01 9.705e+01 1.342e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 04:21:24,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269050
2023-11-22 04:21:44,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1793740.0, ans=0.125
2023-11-22 04:21:50,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1793740.0, ans=0.2
2023-11-22 04:21:52,982 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4550, loss[loss=0.0734, simple_loss=0.1014, pruned_loss=0.01545, audio_tagging_loss=0.007249, over 15943.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09577, pruned_loss=0.01581, audio_tagging_loss=0.009213, over 3043786.88 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:21:57,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1793806.6666666667, ans=0.125
2023-11-22 04:22:18,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.24 vs. limit=15.0
2023-11-22 04:22:28,056 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269100
2023-11-22 04:22:41,454 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:22:42,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.37 vs. limit=6.0
2023-11-22 04:22:43,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1794073.3333333333, ans=0.04949747468305833
2023-11-22 04:22:50,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1794073.3333333333, ans=0.125
2023-11-22 04:22:57,690 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4600, loss[loss=0.07241, simple_loss=0.09191, pruned_loss=0.01673, audio_tagging_loss=0.009726, over 14010.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09455, pruned_loss=0.01572, audio_tagging_loss=0.009423, over 3043510.88 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:23:12,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1794206.6666666667, ans=0.1
2023-11-22 04:23:18,470 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.62 vs. limit=22.5
2023-11-22 04:23:30,453 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.306e+01 8.737e+01 9.544e+01 1.284e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 04:23:31,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269150
2023-11-22 04:24:00,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=22.5
2023-11-22 04:24:02,025 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4650, loss[loss=0.06736, simple_loss=0.08226, pruned_loss=0.01475, audio_tagging_loss=0.01148, over 14996.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09418, pruned_loss=0.01571, audio_tagging_loss=0.00953, over 3038525.91 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:24:10,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1794473.3333333333, ans=0.125
2023-11-22 04:24:22,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1794540.0, ans=0.05
2023-11-22 04:24:27,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1794606.6666666667, ans=0.0
2023-11-22 04:24:30,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1794606.6666666667, ans=0.125
2023-11-22 04:24:37,461 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269200
2023-11-22 04:24:46,840 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.29 vs. limit=15.0
2023-11-22 04:24:52,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1794740.0, ans=0.2
2023-11-22 04:25:03,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1794740.0, ans=0.125
2023-11-22 04:25:06,242 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4700, loss[loss=0.08072, simple_loss=0.1085, pruned_loss=0.01948, audio_tagging_loss=0.007012, over 14707.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09461, pruned_loss=0.01584, audio_tagging_loss=0.009567, over 3037845.55 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:25:31,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1794940.0, ans=0.1
2023-11-22 04:25:39,201 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.243e+01 8.677e+01 9.667e+01 1.211e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 04:25:40,546 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269250
2023-11-22 04:25:41,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1794940.0, ans=0.0
2023-11-22 04:26:03,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1795073.3333333333, ans=0.125
2023-11-22 04:26:11,053 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4750, loss[loss=0.07585, simple_loss=0.09949, pruned_loss=0.01728, audio_tagging_loss=0.00883, over 16483.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09445, pruned_loss=0.01575, audio_tagging_loss=0.009616, over 3041939.80 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:26:12,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1795140.0, ans=0.2
2023-11-22 04:26:12,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1795140.0, ans=0.0
2023-11-22 04:26:15,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-22 04:26:19,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1795140.0, ans=0.2
2023-11-22 04:26:40,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1795273.3333333333, ans=0.125
2023-11-22 04:26:44,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269300
2023-11-22 04:26:53,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1795340.0, ans=0.2
2023-11-22 04:27:10,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1795406.6666666667, ans=0.125
2023-11-22 04:27:11,622 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.34 vs. limit=15.0
2023-11-22 04:27:14,473 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4800, loss[loss=0.07095, simple_loss=0.09402, pruned_loss=0.01304, audio_tagging_loss=0.0109, over 15195.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.0951, pruned_loss=0.01562, audio_tagging_loss=0.009638, over 3044875.96 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:27:16,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1795473.3333333333, ans=0.0
2023-11-22 04:27:18,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1795473.3333333333, ans=0.0
2023-11-22 04:27:21,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1795473.3333333333, ans=0.125
2023-11-22 04:27:36,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1795540.0, ans=0.0
2023-11-22 04:27:46,870 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:27:47,741 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.253e+01 9.179e+01 1.004e+02 1.459e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-22 04:27:49,049 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269350
2023-11-22 04:27:52,738 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.70 vs. limit=15.0
2023-11-22 04:27:56,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1795673.3333333333, ans=0.2
2023-11-22 04:27:59,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.83 vs. limit=15.0
2023-11-22 04:28:17,847 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4850, loss[loss=0.09773, simple_loss=0.1304, pruned_loss=0.02562, audio_tagging_loss=0.006912, over 16000.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09499, pruned_loss=0.0156, audio_tagging_loss=0.009816, over 3046934.06 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:28:25,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1795806.6666666667, ans=0.0
2023-11-22 04:28:28,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1795806.6666666667, ans=0.0
2023-11-22 04:28:38,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1795873.3333333333, ans=0.0
2023-11-22 04:28:42,973 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:28:47,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1795940.0, ans=0.015
2023-11-22 04:28:52,316 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269400
2023-11-22 04:28:52,790 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.57 vs. limit=12.0
2023-11-22 04:29:21,635 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4900, loss[loss=0.09177, simple_loss=0.1262, pruned_loss=0.02089, audio_tagging_loss=0.007776, over 16941.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09547, pruned_loss=0.01572, audio_tagging_loss=0.009845, over 3048375.23 frames. ], batch size: 61, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:29:31,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1796140.0, ans=0.125
2023-11-22 04:29:35,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1796206.6666666667, ans=0.125
2023-11-22 04:29:55,268 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 7.981e+01 8.746e+01 9.394e+01 1.160e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 04:29:56,605 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269450
2023-11-22 04:29:59,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1796340.0, ans=0.125
2023-11-22 04:30:09,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1796340.0, ans=0.125
2023-11-22 04:30:26,658 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 4950, loss[loss=0.07467, simple_loss=0.0949, pruned_loss=0.01753, audio_tagging_loss=0.009686, over 15017.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.0957, pruned_loss=0.01587, audio_tagging_loss=0.009705, over 3039357.80 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:30:29,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1796473.3333333333, ans=0.125
2023-11-22 04:30:33,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1796473.3333333333, ans=0.125
2023-11-22 04:30:58,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1796606.6666666667, ans=0.125
2023-11-22 04:31:01,475 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269500
2023-11-22 04:31:09,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1796673.3333333333, ans=0.125
2023-11-22 04:31:18,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1796740.0, ans=0.125
2023-11-22 04:31:18,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1796740.0, ans=0.125
2023-11-22 04:31:30,675 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5000, loss[loss=0.06487, simple_loss=0.09218, pruned_loss=0.0131, audio_tagging_loss=0.005682, over 15704.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09506, pruned_loss=0.01575, audio_tagging_loss=0.009549, over 3039562.08 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:31:34,606 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:31:43,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1796873.3333333333, ans=0.125
2023-11-22 04:31:55,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1796940.0, ans=0.1
2023-11-22 04:32:04,494 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.014e+01 8.549e+01 9.215e+01 1.274e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 04:32:05,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269550
2023-11-22 04:32:12,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1797006.6666666667, ans=0.04949747468305833
2023-11-22 04:32:28,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.14 vs. limit=15.0
2023-11-22 04:32:30,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1797073.3333333333, ans=0.125
2023-11-22 04:32:35,460 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5050, loss[loss=0.05186, simple_loss=0.06894, pruned_loss=0.01016, audio_tagging_loss=0.007231, over 14687.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09483, pruned_loss=0.01561, audio_tagging_loss=0.009459, over 3038944.01 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:32:44,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.97 vs. limit=12.0
2023-11-22 04:32:46,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1797140.0, ans=0.125
2023-11-22 04:32:59,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1797206.6666666667, ans=0.0
2023-11-22 04:33:00,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1797273.3333333333, ans=0.025
2023-11-22 04:33:04,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1797273.3333333333, ans=0.0
2023-11-22 04:33:10,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269600
2023-11-22 04:33:29,057 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.66 vs. limit=10.0
2023-11-22 04:33:40,604 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5100, loss[loss=0.07971, simple_loss=0.1105, pruned_loss=0.01786, audio_tagging_loss=0.006592, over 15268.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09513, pruned_loss=0.01575, audio_tagging_loss=0.009356, over 3033084.28 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:33:49,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1797473.3333333333, ans=0.2
2023-11-22 04:34:08,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1797606.6666666667, ans=0.125
2023-11-22 04:34:14,417 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.728e+01 7.951e+01 8.483e+01 9.214e+01 1.214e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-22 04:34:15,796 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269650
2023-11-22 04:34:35,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1797740.0, ans=0.125
2023-11-22 04:34:43,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1797740.0, ans=0.2
2023-11-22 04:34:45,989 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5150, loss[loss=0.07758, simple_loss=0.09599, pruned_loss=0.02092, audio_tagging_loss=0.008666, over 14703.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09584, pruned_loss=0.01584, audio_tagging_loss=0.009285, over 3031179.02 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:34:48,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1797806.6666666667, ans=0.125
2023-11-22 04:34:51,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.63 vs. limit=15.0
2023-11-22 04:34:56,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.98 vs. limit=22.5
2023-11-22 04:35:04,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1797873.3333333333, ans=0.05
2023-11-22 04:35:19,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1797940.0, ans=0.2
2023-11-22 04:35:20,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269700
2023-11-22 04:35:37,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.25 vs. limit=22.5
2023-11-22 04:35:51,083 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5200, loss[loss=0.06356, simple_loss=0.07739, pruned_loss=0.01291, audio_tagging_loss=0.01195, over 15530.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09506, pruned_loss=0.01571, audio_tagging_loss=0.009281, over 3036292.16 frames. ], batch size: 62, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:36:00,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1798140.0, ans=0.1
2023-11-22 04:36:03,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.15 vs. limit=22.5
2023-11-22 04:36:24,908 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.234e+01 8.647e+01 9.394e+01 1.220e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 04:36:26,255 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269750
2023-11-22 04:36:27,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1798273.3333333333, ans=0.2
2023-11-22 04:36:31,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1798340.0, ans=0.125
2023-11-22 04:36:55,987 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5250, loss[loss=0.07244, simple_loss=0.09853, pruned_loss=0.01301, audio_tagging_loss=0.01017, over 14566.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09642, pruned_loss=0.01583, audio_tagging_loss=0.009239, over 3037432.96 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:36:59,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1798473.3333333333, ans=0.1
2023-11-22 04:37:12,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1798540.0, ans=0.125
2023-11-22 04:37:22,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1798606.6666666667, ans=0.0
2023-11-22 04:37:22,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1798606.6666666667, ans=0.125
2023-11-22 04:37:30,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269800
2023-11-22 04:38:00,443 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5300, loss[loss=0.04934, simple_loss=0.0532, pruned_loss=0.01126, audio_tagging_loss=0.01148, over 15754.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09595, pruned_loss=0.01565, audio_tagging_loss=0.009167, over 3044347.35 frames. ], batch size: 62, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:38:34,771 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.362e+01 8.825e+01 9.313e+01 1.444e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 04:38:34,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269850
2023-11-22 04:38:39,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.12 vs. limit=15.0
2023-11-22 04:38:46,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.98 vs. limit=8.0
2023-11-22 04:38:52,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1799073.3333333333, ans=0.125
2023-11-22 04:39:04,226 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5350, loss[loss=0.07834, simple_loss=0.1042, pruned_loss=0.01827, audio_tagging_loss=0.007962, over 15271.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09491, pruned_loss=0.01554, audio_tagging_loss=0.009267, over 3047143.23 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:39:08,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1799140.0, ans=0.125
2023-11-22 04:39:14,005 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.73 vs. limit=15.0
2023-11-22 04:39:29,516 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0
2023-11-22 04:39:39,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269900
2023-11-22 04:39:46,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1799340.0, ans=0.0
2023-11-22 04:40:09,598 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5400, loss[loss=0.08511, simple_loss=0.1178, pruned_loss=0.01778, audio_tagging_loss=0.008405, over 15663.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.0951, pruned_loss=0.0155, audio_tagging_loss=0.009275, over 3046836.86 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:40:13,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1799473.3333333333, ans=0.1
2023-11-22 04:40:13,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1799473.3333333333, ans=0.1
2023-11-22 04:40:19,982 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.91 vs. limit=22.5
2023-11-22 04:40:27,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1799540.0, ans=0.125
2023-11-22 04:40:43,902 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.520e+01 8.148e+01 8.900e+01 9.649e+01 1.296e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 04:40:44,041 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 269950
2023-11-22 04:40:55,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.18 vs. limit=15.0
2023-11-22 04:40:57,215 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.28 vs. limit=22.5
2023-11-22 04:40:59,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.54 vs. limit=6.0
2023-11-22 04:41:02,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1799740.0, ans=10.0
2023-11-22 04:41:08,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1799740.0, ans=0.0
2023-11-22 04:41:13,901 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5450, loss[loss=0.065, simple_loss=0.08731, pruned_loss=0.01032, audio_tagging_loss=0.01103, over 13959.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09605, pruned_loss=0.01576, audio_tagging_loss=0.009272, over 3051736.25 frames. ], batch size: 52, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:41:43,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1799940.0, ans=0.125
2023-11-22 04:41:49,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270000
2023-11-22 04:41:51,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.41 vs. limit=22.5
2023-11-22 04:41:59,220 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.99 vs. limit=10.0
2023-11-22 04:42:00,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1800006.6666666667, ans=0.0
2023-11-22 04:42:08,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1800073.3333333333, ans=0.0
2023-11-22 04:42:19,702 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5500, loss[loss=0.104, simple_loss=0.1382, pruned_loss=0.02516, audio_tagging_loss=0.009692, over 15407.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09535, pruned_loss=0.0156, audio_tagging_loss=0.009383, over 3046331.13 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:42:26,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1800140.0, ans=0.1
2023-11-22 04:42:27,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1800140.0, ans=0.0
2023-11-22 04:42:38,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1800206.6666666667, ans=0.0
2023-11-22 04:42:42,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1800206.6666666667, ans=0.0
2023-11-22 04:42:47,188 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.18 vs. limit=22.5
2023-11-22 04:42:49,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1800273.3333333333, ans=0.2
2023-11-22 04:42:53,708 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.241e+01 8.879e+01 9.729e+01 1.307e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-22 04:42:53,840 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270050
2023-11-22 04:43:18,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1800406.6666666667, ans=0.1
2023-11-22 04:43:23,737 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.43 vs. limit=22.5
2023-11-22 04:43:24,370 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5550, loss[loss=0.09308, simple_loss=0.1202, pruned_loss=0.0225, audio_tagging_loss=0.0105, over 15751.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09604, pruned_loss=0.01584, audio_tagging_loss=0.00936, over 3050041.91 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:43:30,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1800473.3333333333, ans=0.0
2023-11-22 04:43:35,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1800540.0, ans=0.1
2023-11-22 04:43:50,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1800606.6666666667, ans=0.0
2023-11-22 04:43:59,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270100
2023-11-22 04:44:21,348 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:44:28,576 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5600, loss[loss=0.07563, simple_loss=0.1074, pruned_loss=0.01445, audio_tagging_loss=0.007468, over 15095.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09612, pruned_loss=0.01575, audio_tagging_loss=0.009509, over 3043697.23 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:44:32,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1800806.6666666667, ans=0.125
2023-11-22 04:44:40,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1800873.3333333333, ans=0.125
2023-11-22 04:44:54,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1800940.0, ans=0.125
2023-11-22 04:45:01,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1800940.0, ans=0.2
2023-11-22 04:45:04,388 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.061e+01 8.801e+01 9.492e+01 1.565e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 04:45:04,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270150
2023-11-22 04:45:15,353 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:45:33,021 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5650, loss[loss=0.05673, simple_loss=0.06442, pruned_loss=0.01116, audio_tagging_loss=0.01336, over 14886.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09594, pruned_loss=0.01567, audio_tagging_loss=0.009572, over 3043474.15 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:45:54,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1801206.6666666667, ans=0.0
2023-11-22 04:45:55,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1801206.6666666667, ans=0.125
2023-11-22 04:45:59,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1801273.3333333333, ans=0.125
2023-11-22 04:45:59,668 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.97 vs. limit=15.0
2023-11-22 04:46:07,672 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270200
2023-11-22 04:46:17,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0
2023-11-22 04:46:37,705 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5700, loss[loss=0.0651, simple_loss=0.09015, pruned_loss=0.01082, audio_tagging_loss=0.009199, over 15262.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09506, pruned_loss=0.01547, audio_tagging_loss=0.009672, over 3040226.93 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:46:51,104 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.60 vs. limit=15.0
2023-11-22 04:46:54,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1801540.0, ans=0.125
2023-11-22 04:47:12,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270250
2023-11-22 04:47:12,701 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.18 vs. limit=15.0
2023-11-22 04:47:13,228 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.286e+01 8.958e+01 9.545e+01 1.493e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 04:47:27,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.05 vs. limit=15.0
2023-11-22 04:47:34,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1801740.0, ans=0.125
2023-11-22 04:47:34,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1801740.0, ans=0.125
2023-11-22 04:47:38,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1801740.0, ans=0.1
2023-11-22 04:47:40,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1801806.6666666667, ans=0.125
2023-11-22 04:47:41,462 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5750, loss[loss=0.07335, simple_loss=0.09451, pruned_loss=0.01779, audio_tagging_loss=0.008303, over 14462.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09529, pruned_loss=0.01566, audio_tagging_loss=0.009519, over 3042610.91 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:47:44,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1801806.6666666667, ans=0.125
2023-11-22 04:47:45,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1801806.6666666667, ans=0.125
2023-11-22 04:48:11,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1801940.0, ans=0.125
2023-11-22 04:48:14,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1801940.0, ans=0.04949747468305833
2023-11-22 04:48:16,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270300
2023-11-22 04:48:36,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1802073.3333333333, ans=0.125
2023-11-22 04:48:45,857 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5800, loss[loss=0.09196, simple_loss=0.1239, pruned_loss=0.02288, audio_tagging_loss=0.007157, over 15014.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.0956, pruned_loss=0.01586, audio_tagging_loss=0.009347, over 3039263.82 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:48:46,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1802140.0, ans=0.125
2023-11-22 04:49:13,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1802273.3333333333, ans=0.125
2023-11-22 04:49:21,059 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270350
2023-11-22 04:49:23,358 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.122e+01 8.117e+01 8.783e+01 9.431e+01 1.385e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 04:49:30,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1802340.0, ans=0.0
2023-11-22 04:49:45,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1802406.6666666667, ans=0.0
2023-11-22 04:49:45,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1802406.6666666667, ans=0.0
2023-11-22 04:49:50,746 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5850, loss[loss=0.07821, simple_loss=0.107, pruned_loss=0.01429, audio_tagging_loss=0.01041, over 14877.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09541, pruned_loss=0.01573, audio_tagging_loss=0.009336, over 3036826.25 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:49:55,591 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.16 vs. limit=10.0
2023-11-22 04:50:08,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1802540.0, ans=0.125
2023-11-22 04:50:10,587 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.42 vs. limit=22.5
2023-11-22 04:50:19,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1802606.6666666667, ans=0.0
2023-11-22 04:50:25,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270400
2023-11-22 04:50:42,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1802740.0, ans=0.0
2023-11-22 04:50:53,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1802740.0, ans=0.125
2023-11-22 04:50:55,529 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5900, loss[loss=0.07072, simple_loss=0.09244, pruned_loss=0.01313, audio_tagging_loss=0.01138, over 15254.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09517, pruned_loss=0.01573, audio_tagging_loss=0.009316, over 3032175.39 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:51:13,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1802873.3333333333, ans=0.0
2023-11-22 04:51:16,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1802873.3333333333, ans=15.0
2023-11-22 04:51:30,449 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270450
2023-11-22 04:51:32,857 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.407e+01 8.248e+01 8.937e+01 9.600e+01 1.150e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 04:51:38,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1803006.6666666667, ans=0.125
2023-11-22 04:51:56,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1803073.3333333333, ans=0.0
2023-11-22 04:52:00,265 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 5950, loss[loss=0.08884, simple_loss=0.1102, pruned_loss=0.02398, audio_tagging_loss=0.009759, over 15309.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09529, pruned_loss=0.01584, audio_tagging_loss=0.009326, over 3040367.52 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:52:01,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1803140.0, ans=0.2
2023-11-22 04:52:36,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270500
2023-11-22 04:52:46,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1803340.0, ans=0.125
2023-11-22 04:52:48,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1803340.0, ans=0.0
2023-11-22 04:52:49,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1803340.0, ans=0.0
2023-11-22 04:53:05,910 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6000, loss[loss=0.05808, simple_loss=0.06597, pruned_loss=0.01306, audio_tagging_loss=0.01203, over 16100.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09528, pruned_loss=0.01581, audio_tagging_loss=0.009359, over 3048990.21 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:53:05,911 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 04:53:45,805 INFO [train_asr.py:1253] (2/4) Epoch 23, validation: loss=0.05955, simple_loss=0.05175, pruned_loss=0.005139, audio_tagging_loss=0.02853, over 4681554.00 frames. 
2023-11-22 04:53:45,806 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 04:53:46,107 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:54:07,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.05 vs. limit=15.0
2023-11-22 04:54:20,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270550
2023-11-22 04:54:23,986 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.322e+01 8.050e+01 8.627e+01 9.368e+01 1.674e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 04:54:25,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1803673.3333333333, ans=0.125
2023-11-22 04:54:33,789 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:54:50,470 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6050, loss[loss=0.08142, simple_loss=0.1062, pruned_loss=0.01953, audio_tagging_loss=0.008783, over 14484.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09535, pruned_loss=0.01584, audio_tagging_loss=0.009324, over 3044341.10 frames. ], batch size: 53, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:55:03,853 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.18 vs. limit=15.0
2023-11-22 04:55:08,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1803873.3333333333, ans=0.0
2023-11-22 04:55:15,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1803940.0, ans=0.125
2023-11-22 04:55:17,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1803940.0, ans=0.125
2023-11-22 04:55:24,547 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270600
2023-11-22 04:55:40,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1804073.3333333333, ans=0.125
2023-11-22 04:55:54,002 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6100, loss[loss=0.06158, simple_loss=0.07847, pruned_loss=0.01253, audio_tagging_loss=0.009811, over 14349.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09553, pruned_loss=0.01587, audio_tagging_loss=0.009299, over 3049747.67 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:55:54,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1804140.0, ans=0.125
2023-11-22 04:56:29,367 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270650
2023-11-22 04:56:31,744 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.256e+01 8.259e+01 8.763e+01 9.618e+01 1.406e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 04:56:32,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.05 vs. limit=22.5
2023-11-22 04:56:43,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1804340.0, ans=0.125
2023-11-22 04:56:59,487 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6150, loss[loss=0.07615, simple_loss=0.09812, pruned_loss=0.01779, audio_tagging_loss=0.009298, over 15226.00 frames. ], tot_loss[loss=0.07285, simple_loss=0.09533, pruned_loss=0.01585, audio_tagging_loss=0.009331, over 3051295.35 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:57:01,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.28 vs. limit=22.5
2023-11-22 04:57:02,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1804473.3333333333, ans=0.125
2023-11-22 04:57:34,155 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270700
2023-11-22 04:57:45,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1804673.3333333333, ans=10.0
2023-11-22 04:57:49,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1804673.3333333333, ans=0.0
2023-11-22 04:58:03,970 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6200, loss[loss=0.06384, simple_loss=0.07565, pruned_loss=0.01569, audio_tagging_loss=0.01032, over 15971.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09442, pruned_loss=0.01567, audio_tagging_loss=0.009495, over 3051314.99 frames. ], batch size: 63, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:58:17,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.84 vs. limit=15.0
2023-11-22 04:58:36,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1804940.0, ans=0.1
2023-11-22 04:58:38,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270750
2023-11-22 04:58:40,806 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.162e+01 9.001e+01 9.547e+01 1.182e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 04:59:02,284 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.90 vs. limit=15.0
2023-11-22 04:59:02,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1805073.3333333333, ans=0.1
2023-11-22 04:59:07,554 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6250, loss[loss=0.07601, simple_loss=0.09557, pruned_loss=0.01501, audio_tagging_loss=0.01321, over 16472.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09437, pruned_loss=0.01565, audio_tagging_loss=0.009598, over 3062790.16 frames. ], batch size: 61, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:59:11,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1805140.0, ans=0.125
2023-11-22 04:59:19,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1805206.6666666667, ans=0.125
2023-11-22 04:59:23,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1805206.6666666667, ans=0.2
2023-11-22 04:59:27,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1805206.6666666667, ans=0.1
2023-11-22 04:59:31,742 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.01 vs. limit=22.5
2023-11-22 04:59:42,748 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270800
2023-11-22 04:59:43,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.53 vs. limit=6.0
2023-11-22 04:59:51,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.75 vs. limit=10.0
2023-11-22 04:59:52,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1805340.0, ans=0.07
2023-11-22 04:59:54,998 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.05 vs. limit=22.5
2023-11-22 05:00:11,871 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6300, loss[loss=0.07346, simple_loss=0.1005, pruned_loss=0.01235, audio_tagging_loss=0.01086, over 16377.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.09459, pruned_loss=0.01549, audio_tagging_loss=0.009721, over 3059109.04 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:00:43,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1805606.6666666667, ans=0.0
2023-11-22 05:00:46,738 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270850
2023-11-22 05:00:47,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1805606.6666666667, ans=0.1
2023-11-22 05:00:49,019 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.345e+01 8.951e+01 9.677e+01 1.248e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 05:00:54,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1805673.3333333333, ans=0.0
2023-11-22 05:01:00,037 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.27 vs. limit=15.0
2023-11-22 05:01:16,578 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6350, loss[loss=0.06417, simple_loss=0.08627, pruned_loss=0.01176, audio_tagging_loss=0.009272, over 15405.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.0957, pruned_loss=0.01568, audio_tagging_loss=0.009546, over 3059346.56 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:01:18,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1805806.6666666667, ans=0.0
2023-11-22 05:01:34,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1805873.3333333333, ans=0.0
2023-11-22 05:01:43,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1805940.0, ans=0.0
2023-11-22 05:01:45,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1805940.0, ans=0.125
2023-11-22 05:01:50,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270900
2023-11-22 05:01:57,053 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.72 vs. limit=12.0
2023-11-22 05:02:04,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1806006.6666666667, ans=0.125
2023-11-22 05:02:10,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1806073.3333333333, ans=0.2
2023-11-22 05:02:10,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1806073.3333333333, ans=0.0
2023-11-22 05:02:12,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1806073.3333333333, ans=0.1
2023-11-22 05:02:20,583 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6400, loss[loss=0.08287, simple_loss=0.1017, pruned_loss=0.01745, audio_tagging_loss=0.01459, over 14858.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09598, pruned_loss=0.0158, audio_tagging_loss=0.009673, over 3051413.84 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:02:54,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 270950
2023-11-22 05:02:57,225 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.214e+01 8.231e+01 8.800e+01 9.655e+01 1.370e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 05:03:16,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1806406.6666666667, ans=0.1
2023-11-22 05:03:23,824 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6450, loss[loss=0.06886, simple_loss=0.09333, pruned_loss=0.01251, audio_tagging_loss=0.009691, over 16317.00 frames. ], tot_loss[loss=0.07312, simple_loss=0.09551, pruned_loss=0.01562, audio_tagging_loss=0.009745, over 3045993.65 frames. ], batch size: 59, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:03:25,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1806473.3333333333, ans=0.125
2023-11-22 05:03:48,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1806606.6666666667, ans=0.025
2023-11-22 05:03:58,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271000
2023-11-22 05:04:00,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1806606.6666666667, ans=0.125
2023-11-22 05:04:28,839 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6500, loss[loss=0.07787, simple_loss=0.1068, pruned_loss=0.01658, audio_tagging_loss=0.007866, over 15356.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09464, pruned_loss=0.01563, audio_tagging_loss=0.009616, over 3043154.42 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:05:02,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271050
2023-11-22 05:05:05,185 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.277e+01 8.344e+01 8.837e+01 9.717e+01 1.139e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 05:05:08,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1807006.6666666667, ans=0.09899494936611666
2023-11-22 05:05:12,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1807006.6666666667, ans=0.125
2023-11-22 05:05:14,586 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.98 vs. limit=22.5
2023-11-22 05:05:17,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1807006.6666666667, ans=0.1
2023-11-22 05:05:27,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1807073.3333333333, ans=0.0
2023-11-22 05:05:31,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1807140.0, ans=0.2
2023-11-22 05:05:32,574 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6550, loss[loss=0.06072, simple_loss=0.08469, pruned_loss=0.009213, audio_tagging_loss=0.009158, over 14972.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09427, pruned_loss=0.01563, audio_tagging_loss=0.009531, over 3043707.92 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:05:34,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1807140.0, ans=0.1
2023-11-22 05:05:55,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.23 vs. limit=12.0
2023-11-22 05:06:01,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1807273.3333333333, ans=0.125
2023-11-22 05:06:07,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271100
2023-11-22 05:06:09,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1807273.3333333333, ans=0.1
2023-11-22 05:06:36,246 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6600, loss[loss=0.05108, simple_loss=0.06808, pruned_loss=0.0106, audio_tagging_loss=0.006438, over 16306.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09435, pruned_loss=0.01575, audio_tagging_loss=0.009409, over 3042115.94 frames. ], batch size: 63, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:06:47,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.34 vs. limit=15.0
2023-11-22 05:06:48,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1807540.0, ans=0.0
2023-11-22 05:06:55,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.31 vs. limit=22.5
2023-11-22 05:07:06,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1807606.6666666667, ans=0.125
2023-11-22 05:07:09,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1807606.6666666667, ans=0.125
2023-11-22 05:07:11,736 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271150
2023-11-22 05:07:15,163 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.298e+01 8.931e+01 9.754e+01 1.164e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 05:07:17,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1807673.3333333333, ans=0.0
2023-11-22 05:07:20,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1807673.3333333333, ans=0.0
2023-11-22 05:07:35,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1807740.0, ans=0.125
2023-11-22 05:07:39,364 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.00 vs. limit=15.0
2023-11-22 05:07:40,528 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6650, loss[loss=0.06781, simple_loss=0.09346, pruned_loss=0.01303, audio_tagging_loss=0.008044, over 14744.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09388, pruned_loss=0.01556, audio_tagging_loss=0.009311, over 3044734.04 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:08:15,358 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271200
2023-11-22 05:08:45,696 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6700, loss[loss=0.08341, simple_loss=0.1132, pruned_loss=0.0196, audio_tagging_loss=0.007197, over 14970.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09432, pruned_loss=0.01566, audio_tagging_loss=0.009294, over 3040455.25 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:09:20,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271250
2023-11-22 05:09:24,047 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.573e+01 8.261e+01 8.925e+01 9.693e+01 1.242e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 05:09:50,110 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6750, loss[loss=0.07907, simple_loss=0.1076, pruned_loss=0.01988, audio_tagging_loss=0.005403, over 13490.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09329, pruned_loss=0.01539, audio_tagging_loss=0.00935, over 3039553.12 frames. ], batch size: 50, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:09:56,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1808473.3333333333, ans=0.0
2023-11-22 05:09:59,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1808473.3333333333, ans=0.125
2023-11-22 05:10:04,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1808540.0, ans=0.1
2023-11-22 05:10:08,030 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:10:17,624 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.05 vs. limit=15.0
2023-11-22 05:10:17,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.05 vs. limit=10.0
2023-11-22 05:10:25,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271300
2023-11-22 05:10:26,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1808606.6666666667, ans=0.0
2023-11-22 05:10:48,878 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:10:54,820 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6800, loss[loss=0.07617, simple_loss=0.1017, pruned_loss=0.0192, audio_tagging_loss=0.006118, over 14332.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09317, pruned_loss=0.01528, audio_tagging_loss=0.009301, over 3035640.02 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:11:09,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1808873.3333333333, ans=0.0
2023-11-22 05:11:12,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1808873.3333333333, ans=0.125
2023-11-22 05:11:21,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1808940.0, ans=0.0
2023-11-22 05:11:23,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1808940.0, ans=0.125
2023-11-22 05:11:25,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1808940.0, ans=0.125
2023-11-22 05:11:27,355 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.21 vs. limit=15.0
2023-11-22 05:11:31,329 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271350
2023-11-22 05:11:31,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1808940.0, ans=0.1
2023-11-22 05:11:34,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=1809006.6666666667, ans=0.2
2023-11-22 05:11:34,841 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.372e+01 8.044e+01 8.828e+01 9.747e+01 1.352e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 05:11:35,790 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.14 vs. limit=10.0
2023-11-22 05:11:55,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1809073.3333333333, ans=0.0
2023-11-22 05:12:00,307 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6850, loss[loss=0.05041, simple_loss=0.06447, pruned_loss=0.007208, audio_tagging_loss=0.01097, over 16456.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09342, pruned_loss=0.01536, audio_tagging_loss=0.00933, over 3038001.37 frames. ], batch size: 63, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:12:03,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1809140.0, ans=0.0
2023-11-22 05:12:24,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1809206.6666666667, ans=0.2
2023-11-22 05:12:34,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1809273.3333333333, ans=0.1
2023-11-22 05:12:35,798 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271400
2023-11-22 05:12:46,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1809340.0, ans=0.035
2023-11-22 05:13:01,517 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:13:06,059 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6900, loss[loss=0.0823, simple_loss=0.09842, pruned_loss=0.01991, audio_tagging_loss=0.01318, over 14953.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09406, pruned_loss=0.01535, audio_tagging_loss=0.009298, over 3042631.97 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:13:10,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1809473.3333333333, ans=0.1
2023-11-22 05:13:21,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1809540.0, ans=0.2
2023-11-22 05:13:28,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1809540.0, ans=0.125
2023-11-22 05:13:30,468 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.28 vs. limit=12.0
2023-11-22 05:13:40,992 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271450
2023-11-22 05:13:41,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1809606.6666666667, ans=0.2
2023-11-22 05:13:46,997 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.045e+01 8.623e+01 9.194e+01 1.221e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 05:13:55,817 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:13:57,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1809740.0, ans=0.0
2023-11-22 05:13:58,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1809740.0, ans=0.2
2023-11-22 05:14:02,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1809740.0, ans=0.1
2023-11-22 05:14:11,288 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 6950, loss[loss=0.0749, simple_loss=0.1057, pruned_loss=0.01266, audio_tagging_loss=0.009416, over 14686.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09438, pruned_loss=0.01535, audio_tagging_loss=0.009338, over 3049490.18 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:14:46,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271500
2023-11-22 05:14:46,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1809940.0, ans=0.0
2023-11-22 05:14:53,971 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.64 vs. limit=15.0
2023-11-22 05:15:03,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1810073.3333333333, ans=0.0
2023-11-22 05:15:09,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1810073.3333333333, ans=0.125
2023-11-22 05:15:15,662 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7000, loss[loss=0.06632, simple_loss=0.0849, pruned_loss=0.01646, audio_tagging_loss=0.00741, over 14801.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09355, pruned_loss=0.01529, audio_tagging_loss=0.00943, over 3050183.60 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:15:35,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1810206.6666666667, ans=0.0
2023-11-22 05:15:41,970 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:15:45,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1810273.3333333333, ans=0.125
2023-11-22 05:15:50,522 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271550
2023-11-22 05:15:54,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1810340.0, ans=0.125
2023-11-22 05:15:54,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1810340.0, ans=0.025
2023-11-22 05:15:55,914 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.160e+01 8.872e+01 9.665e+01 1.181e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 05:15:58,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1810340.0, ans=0.125
2023-11-22 05:16:07,559 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:16:20,955 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7050, loss[loss=0.0961, simple_loss=0.1145, pruned_loss=0.0257, audio_tagging_loss=0.01312, over 14760.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.0934, pruned_loss=0.01543, audio_tagging_loss=0.009498, over 3047029.31 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:16:24,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1810473.3333333333, ans=0.125
2023-11-22 05:16:32,082 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.59 vs. limit=15.0
2023-11-22 05:16:39,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1810540.0, ans=0.125
2023-11-22 05:16:45,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1810606.6666666667, ans=0.125
2023-11-22 05:16:46,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1810606.6666666667, ans=0.2
2023-11-22 05:16:55,772 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271600
2023-11-22 05:17:11,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1810673.3333333333, ans=0.125
2023-11-22 05:17:26,015 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7100, loss[loss=0.06495, simple_loss=0.07151, pruned_loss=0.01542, audio_tagging_loss=0.01378, over 15212.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09298, pruned_loss=0.01517, audio_tagging_loss=0.009614, over 3052299.75 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:17:32,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1810806.6666666667, ans=0.1
2023-11-22 05:17:42,879 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.90 vs. limit=22.5
2023-11-22 05:17:59,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271650
2023-11-22 05:18:02,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1810940.0, ans=0.07
2023-11-22 05:18:05,324 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.205e+01 9.069e+01 1.012e+02 2.750e+02, threshold=1.814e+02, percent-clipped=1.0
2023-11-22 05:18:23,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1811073.3333333333, ans=0.2
2023-11-22 05:18:29,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1811140.0, ans=0.125
2023-11-22 05:18:30,056 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7150, loss[loss=0.08614, simple_loss=0.1074, pruned_loss=0.02213, audio_tagging_loss=0.01033, over 15318.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.0941, pruned_loss=0.0154, audio_tagging_loss=0.009628, over 3051350.56 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:18:57,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1811273.3333333333, ans=0.0
2023-11-22 05:19:04,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1811273.3333333333, ans=0.2
2023-11-22 05:19:05,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271700
2023-11-22 05:19:09,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.59 vs. limit=8.0
2023-11-22 05:19:34,519 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7200, loss[loss=0.0673, simple_loss=0.0996, pruned_loss=0.01051, audio_tagging_loss=0.006989, over 15246.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09507, pruned_loss=0.01543, audio_tagging_loss=0.009671, over 3055433.84 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:19:35,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-22 05:20:09,902 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271750
2023-11-22 05:20:14,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.124e+01 9.002e+01 9.849e+01 1.230e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 05:20:25,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1811740.0, ans=0.0
2023-11-22 05:20:35,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1811740.0, ans=0.5
2023-11-22 05:20:37,927 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:20:40,016 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7250, loss[loss=0.06373, simple_loss=0.07954, pruned_loss=0.01312, audio_tagging_loss=0.01083, over 15473.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09466, pruned_loss=0.01522, audio_tagging_loss=0.009686, over 3049404.46 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:20:40,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1811806.6666666667, ans=0.125
2023-11-22 05:21:01,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-22 05:21:04,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1811940.0, ans=0.0
2023-11-22 05:21:06,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1811940.0, ans=0.1
2023-11-22 05:21:06,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1811940.0, ans=0.125
2023-11-22 05:21:14,412 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271800
2023-11-22 05:21:44,755 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7300, loss[loss=0.08455, simple_loss=0.1127, pruned_loss=0.01936, audio_tagging_loss=0.008833, over 15284.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09555, pruned_loss=0.01539, audio_tagging_loss=0.009587, over 3044758.89 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:21:47,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1812140.0, ans=0.0
2023-11-22 05:22:18,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1812273.3333333333, ans=0.125
2023-11-22 05:22:20,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271850
2023-11-22 05:22:23,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1812340.0, ans=0.2
2023-11-22 05:22:25,967 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.604e+01 8.310e+01 9.010e+01 9.721e+01 2.858e+02, threshold=1.802e+02, percent-clipped=1.0
2023-11-22 05:22:26,577 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.65 vs. limit=15.0
2023-11-22 05:22:27,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.13 vs. limit=15.0
2023-11-22 05:22:43,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.40 vs. limit=22.5
2023-11-22 05:22:49,424 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7350, loss[loss=0.066, simple_loss=0.09353, pruned_loss=0.01178, audio_tagging_loss=0.00746, over 15048.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09637, pruned_loss=0.01569, audio_tagging_loss=0.009386, over 3049037.78 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:23:25,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271900
2023-11-22 05:23:35,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1812673.3333333333, ans=0.125
2023-11-22 05:23:44,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1812740.0, ans=0.125
2023-11-22 05:23:52,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1812740.0, ans=0.125
2023-11-22 05:23:54,484 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7400, loss[loss=0.05692, simple_loss=0.07522, pruned_loss=0.01023, audio_tagging_loss=0.009074, over 16080.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09561, pruned_loss=0.0156, audio_tagging_loss=0.009306, over 3047300.29 frames. ], batch size: 61, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:23:55,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1812806.6666666667, ans=0.125
2023-11-22 05:24:19,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1812940.0, ans=0.0
2023-11-22 05:24:30,096 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 271950
2023-11-22 05:24:36,087 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 7.965e+01 8.642e+01 9.211e+01 1.272e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-22 05:24:57,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1813073.3333333333, ans=0.125
2023-11-22 05:25:00,072 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7450, loss[loss=0.07546, simple_loss=0.1002, pruned_loss=0.01543, audio_tagging_loss=0.009934, over 15518.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09579, pruned_loss=0.0156, audio_tagging_loss=0.009258, over 3045221.83 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:25:12,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1813206.6666666667, ans=0.125
2023-11-22 05:25:13,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1813206.6666666667, ans=0.125
2023-11-22 05:25:18,682 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.06 vs. limit=15.0
2023-11-22 05:25:20,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-22 05:25:24,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1813273.3333333333, ans=0.125
2023-11-22 05:25:34,677 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272000
2023-11-22 05:25:47,198 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0
2023-11-22 05:26:00,065 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.44 vs. limit=12.0
2023-11-22 05:26:08,226 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7500, loss[loss=0.07803, simple_loss=0.1026, pruned_loss=0.01777, audio_tagging_loss=0.008955, over 15027.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09633, pruned_loss=0.01578, audio_tagging_loss=0.00922, over 3042531.22 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:26:09,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1813473.3333333333, ans=0.0
2023-11-22 05:26:16,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1813473.3333333333, ans=0.125
2023-11-22 05:26:23,253 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.45 vs. limit=22.5
2023-11-22 05:26:42,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1813606.6666666667, ans=0.125
2023-11-22 05:26:43,594 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272050
2023-11-22 05:26:48,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1813673.3333333333, ans=0.0
2023-11-22 05:26:49,563 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.448e+01 8.968e+01 9.582e+01 2.186e+02, threshold=1.794e+02, percent-clipped=1.0
2023-11-22 05:26:50,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1813673.3333333333, ans=0.0
2023-11-22 05:27:13,283 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7550, loss[loss=0.07795, simple_loss=0.1009, pruned_loss=0.01546, audio_tagging_loss=0.01204, over 15501.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09615, pruned_loss=0.01584, audio_tagging_loss=0.009144, over 3042294.69 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:27:29,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1813873.3333333333, ans=0.04949747468305833
2023-11-22 05:27:45,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1813940.0, ans=0.125
2023-11-22 05:27:48,111 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272100
2023-11-22 05:27:54,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1814006.6666666667, ans=0.1
2023-11-22 05:28:18,029 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7600, loss[loss=0.09169, simple_loss=0.1287, pruned_loss=0.01979, audio_tagging_loss=0.007528, over 15073.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09478, pruned_loss=0.01555, audio_tagging_loss=0.009271, over 3047163.95 frames. ], batch size: 53, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:28:48,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1814273.3333333333, ans=0.0
2023-11-22 05:28:51,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272150
2023-11-22 05:28:51,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=1814273.3333333333, ans=0.02
2023-11-22 05:28:54,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1814340.0, ans=0.1
2023-11-22 05:28:59,136 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.478e+01 8.103e+01 8.710e+01 9.390e+01 1.351e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 05:29:07,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1814406.6666666667, ans=0.2
2023-11-22 05:29:20,741 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7650, loss[loss=0.06257, simple_loss=0.07847, pruned_loss=0.01335, audio_tagging_loss=0.009989, over 14400.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09409, pruned_loss=0.01551, audio_tagging_loss=0.009261, over 3050204.06 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:29:24,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1814473.3333333333, ans=0.125
2023-11-22 05:29:29,422 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.26 vs. limit=15.0
2023-11-22 05:29:31,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1814473.3333333333, ans=0.0
2023-11-22 05:29:43,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=1814540.0, ans=8.0
2023-11-22 05:29:56,437 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272200
2023-11-22 05:30:04,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1814673.3333333333, ans=0.1
2023-11-22 05:30:05,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1814673.3333333333, ans=10.0
2023-11-22 05:30:06,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0
2023-11-22 05:30:15,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1814740.0, ans=0.04949747468305833
2023-11-22 05:30:25,239 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.89 vs. limit=15.0
2023-11-22 05:30:25,806 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7700, loss[loss=0.07022, simple_loss=0.0927, pruned_loss=0.0141, audio_tagging_loss=0.00977, over 14943.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09427, pruned_loss=0.01532, audio_tagging_loss=0.009302, over 3055355.71 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:30:29,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1814806.6666666667, ans=0.0
2023-11-22 05:30:31,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1814806.6666666667, ans=0.0
2023-11-22 05:31:01,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272250
2023-11-22 05:31:08,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1815006.6666666667, ans=0.0
2023-11-22 05:31:09,430 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 7.911e+01 8.690e+01 9.336e+01 1.339e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 05:31:17,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1815073.3333333333, ans=0.05
2023-11-22 05:31:20,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.64 vs. limit=15.0
2023-11-22 05:31:31,240 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7750, loss[loss=0.04912, simple_loss=0.05751, pruned_loss=0.012, audio_tagging_loss=0.00836, over 14926.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.0942, pruned_loss=0.01518, audio_tagging_loss=0.009288, over 3053639.17 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:32:00,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1815273.3333333333, ans=0.125
2023-11-22 05:32:02,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1815273.3333333333, ans=0.2
2023-11-22 05:32:06,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272300
2023-11-22 05:32:36,667 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7800, loss[loss=0.06761, simple_loss=0.09321, pruned_loss=0.01181, audio_tagging_loss=0.009195, over 14598.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09423, pruned_loss=0.01515, audio_tagging_loss=0.009412, over 3051114.06 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:32:44,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1815473.3333333333, ans=0.125
2023-11-22 05:33:01,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1815606.6666666667, ans=0.0
2023-11-22 05:33:03,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.59 vs. limit=22.5
2023-11-22 05:33:11,368 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272350
2023-11-22 05:33:18,417 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:33:19,312 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.265e+01 8.937e+01 9.454e+01 1.226e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 05:33:39,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1815740.0, ans=0.05
2023-11-22 05:33:41,781 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7850, loss[loss=0.07503, simple_loss=0.09983, pruned_loss=0.01382, audio_tagging_loss=0.0113, over 15345.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09351, pruned_loss=0.01522, audio_tagging_loss=0.009611, over 3055416.14 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:33:51,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.49 vs. limit=15.0
2023-11-22 05:33:52,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1815806.6666666667, ans=0.0
2023-11-22 05:34:04,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1815873.3333333333, ans=0.125
2023-11-22 05:34:06,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1815940.0, ans=0.0
2023-11-22 05:34:17,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272400
2023-11-22 05:34:28,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1816006.6666666667, ans=0.1
2023-11-22 05:34:35,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1816073.3333333333, ans=0.125
2023-11-22 05:34:45,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1816073.3333333333, ans=0.125
2023-11-22 05:34:47,293 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7900, loss[loss=0.07958, simple_loss=0.09464, pruned_loss=0.02141, audio_tagging_loss=0.01084, over 14855.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09226, pruned_loss=0.01502, audio_tagging_loss=0.009782, over 3053176.96 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:34:51,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1816140.0, ans=0.0
2023-11-22 05:34:55,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1816140.0, ans=0.1
2023-11-22 05:34:56,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1816140.0, ans=0.0
2023-11-22 05:34:57,338 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.80 vs. limit=15.0
2023-11-22 05:35:08,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1816206.6666666667, ans=0.05
2023-11-22 05:35:09,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1816206.6666666667, ans=0.1
2023-11-22 05:35:18,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1816273.3333333333, ans=0.04949747468305833
2023-11-22 05:35:22,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272450
2023-11-22 05:35:30,025 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.475e+01 8.108e+01 8.831e+01 9.576e+01 1.163e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 05:35:30,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1816340.0, ans=0.125
2023-11-22 05:35:51,649 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 7950, loss[loss=0.06358, simple_loss=0.08601, pruned_loss=0.01095, audio_tagging_loss=0.009624, over 14892.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09221, pruned_loss=0.0151, audio_tagging_loss=0.009831, over 3052983.89 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:36:01,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1816473.3333333333, ans=0.0
2023-11-22 05:36:04,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1816540.0, ans=0.125
2023-11-22 05:36:07,926 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:36:17,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1816606.6666666667, ans=0.0
2023-11-22 05:36:27,066 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272500
2023-11-22 05:36:27,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1816606.6666666667, ans=0.0
2023-11-22 05:36:48,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1816740.0, ans=0.125
2023-11-22 05:36:57,585 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8000, loss[loss=0.08265, simple_loss=0.109, pruned_loss=0.01696, audio_tagging_loss=0.01121, over 15698.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09307, pruned_loss=0.01543, audio_tagging_loss=0.009867, over 3044509.26 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:37:19,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1816873.3333333333, ans=0.0
2023-11-22 05:37:28,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1816940.0, ans=0.125
2023-11-22 05:37:32,109 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272550
2023-11-22 05:37:36,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1817006.6666666667, ans=0.125
2023-11-22 05:37:40,494 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.372e+01 8.294e+01 8.813e+01 9.754e+01 1.240e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 05:38:02,636 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8050, loss[loss=0.06821, simple_loss=0.09024, pruned_loss=0.0122, audio_tagging_loss=0.01089, over 14629.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09294, pruned_loss=0.01539, audio_tagging_loss=0.009872, over 3042538.89 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:38:25,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.50 vs. limit=22.5
2023-11-22 05:38:32,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1817273.3333333333, ans=0.125
2023-11-22 05:38:37,256 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272600
2023-11-22 05:39:06,913 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8100, loss[loss=0.05699, simple_loss=0.07921, pruned_loss=0.01091, audio_tagging_loss=0.006471, over 15594.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09331, pruned_loss=0.0153, audio_tagging_loss=0.009743, over 3044765.94 frames. ], batch size: 61, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:39:12,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1817473.3333333333, ans=0.04949747468305833
2023-11-22 05:39:41,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272650
2023-11-22 05:39:42,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=15.0
2023-11-22 05:39:43,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.05 vs. limit=15.0
2023-11-22 05:39:50,703 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.525e+01 8.265e+01 8.788e+01 9.527e+01 1.186e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 05:39:51,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.06 vs. limit=15.0
2023-11-22 05:39:55,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.04 vs. limit=15.0
2023-11-22 05:40:11,842 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8150, loss[loss=0.06413, simple_loss=0.07545, pruned_loss=0.01324, audio_tagging_loss=0.01317, over 14555.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09441, pruned_loss=0.01553, audio_tagging_loss=0.009586, over 3044636.53 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:40:36,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1817940.0, ans=0.0
2023-11-22 05:40:39,042 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.51 vs. limit=10.0
2023-11-22 05:40:43,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1817940.0, ans=0.125
2023-11-22 05:40:46,666 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272700
2023-11-22 05:40:46,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1817940.0, ans=0.2
2023-11-22 05:41:04,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1818073.3333333333, ans=0.0
2023-11-22 05:41:05,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1818073.3333333333, ans=0.125
2023-11-22 05:41:11,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1818073.3333333333, ans=0.125
2023-11-22 05:41:16,701 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8200, loss[loss=0.05409, simple_loss=0.06929, pruned_loss=0.01177, audio_tagging_loss=0.007665, over 15411.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09513, pruned_loss=0.0157, audio_tagging_loss=0.009332, over 3051085.88 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:41:16,766 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:41:25,991 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-22 05:41:30,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.20 vs. limit=15.0
2023-11-22 05:41:34,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1818206.6666666667, ans=0.95
2023-11-22 05:41:37,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.72 vs. limit=15.0
2023-11-22 05:41:50,917 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272750
2023-11-22 05:41:52,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.02 vs. limit=22.5
2023-11-22 05:42:00,282 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.465e+01 8.322e+01 8.892e+01 9.603e+01 1.403e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 05:42:03,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1818340.0, ans=0.125
2023-11-22 05:42:21,083 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8250, loss[loss=0.06544, simple_loss=0.08548, pruned_loss=0.01328, audio_tagging_loss=0.009422, over 15345.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09526, pruned_loss=0.0158, audio_tagging_loss=0.009267, over 3041719.17 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:42:26,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1818473.3333333333, ans=10.0
2023-11-22 05:42:35,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.08 vs. limit=22.5
2023-11-22 05:42:43,400 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.32 vs. limit=15.0
2023-11-22 05:42:56,149 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272800
2023-11-22 05:43:05,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1818673.3333333333, ans=0.2
2023-11-22 05:43:09,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1818673.3333333333, ans=0.2
2023-11-22 05:43:25,813 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8300, loss[loss=0.07203, simple_loss=0.08833, pruned_loss=0.01609, audio_tagging_loss=0.01177, over 14540.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09536, pruned_loss=0.01567, audio_tagging_loss=0.009275, over 3040646.46 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:44:01,260 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272850
2023-11-22 05:44:09,828 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.221e+01 8.844e+01 9.406e+01 1.147e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 05:44:21,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1819073.3333333333, ans=0.09899494936611666
2023-11-22 05:44:31,030 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8350, loss[loss=0.0757, simple_loss=0.1038, pruned_loss=0.01483, audio_tagging_loss=0.008967, over 14615.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09513, pruned_loss=0.01581, audio_tagging_loss=0.00929, over 3036951.06 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:44:46,357 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:44:57,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1819273.3333333333, ans=0.1
2023-11-22 05:45:02,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1819273.3333333333, ans=0.0
2023-11-22 05:45:04,967 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272900
2023-11-22 05:45:06,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1819273.3333333333, ans=0.1
2023-11-22 05:45:06,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1819273.3333333333, ans=0.125
2023-11-22 05:45:28,709 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.32 vs. limit=15.0
2023-11-22 05:45:34,869 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8400, loss[loss=0.07569, simple_loss=0.102, pruned_loss=0.01862, audio_tagging_loss=0.006044, over 15154.00 frames. ], tot_loss[loss=0.07164, simple_loss=0.09372, pruned_loss=0.01545, audio_tagging_loss=0.009331, over 3034313.72 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:45:35,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1819473.3333333333, ans=0.0
2023-11-22 05:45:59,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1819606.6666666667, ans=0.5
2023-11-22 05:46:09,196 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 272950
2023-11-22 05:46:09,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1819606.6666666667, ans=0.09899494936611666
2023-11-22 05:46:16,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1819673.3333333333, ans=0.2
2023-11-22 05:46:17,478 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.034e+01 8.663e+01 9.140e+01 1.267e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 05:46:17,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1819673.3333333333, ans=0.035
2023-11-22 05:46:23,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1819673.3333333333, ans=0.0
2023-11-22 05:46:37,826 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8450, loss[loss=0.05983, simple_loss=0.08081, pruned_loss=0.01093, audio_tagging_loss=0.0085, over 16270.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09323, pruned_loss=0.01544, audio_tagging_loss=0.009377, over 3033181.03 frames. ], batch size: 60, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:46:43,894 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.50 vs. limit=15.0
2023-11-22 05:46:56,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1819873.3333333333, ans=0.0
2023-11-22 05:47:01,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1819873.3333333333, ans=0.0
2023-11-22 05:47:05,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1819940.0, ans=0.125
2023-11-22 05:47:12,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273000
2023-11-22 05:47:12,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1819940.0, ans=0.0
2023-11-22 05:47:20,789 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:47:41,852 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8500, loss[loss=0.08458, simple_loss=0.1095, pruned_loss=0.01977, audio_tagging_loss=0.01006, over 15733.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.09365, pruned_loss=0.01535, audio_tagging_loss=0.009327, over 3033793.56 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:48:16,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273050
2023-11-22 05:48:25,397 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.253e+01 8.845e+01 9.555e+01 1.261e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 05:48:25,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1820340.0, ans=0.125
2023-11-22 05:48:33,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1820406.6666666667, ans=0.2
2023-11-22 05:48:38,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.55 vs. limit=10.0
2023-11-22 05:48:40,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1820406.6666666667, ans=0.125
2023-11-22 05:48:41,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1820406.6666666667, ans=0.125
2023-11-22 05:48:46,824 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8550, loss[loss=0.07738, simple_loss=0.1023, pruned_loss=0.01842, audio_tagging_loss=0.007808, over 15778.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.09399, pruned_loss=0.01535, audio_tagging_loss=0.009329, over 3046136.32 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:48:51,091 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.97 vs. limit=22.5
2023-11-22 05:49:00,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1820540.0, ans=0.0
2023-11-22 05:49:11,635 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.89 vs. limit=15.0
2023-11-22 05:49:21,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273100
2023-11-22 05:49:21,891 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.14 vs. limit=15.0
2023-11-22 05:49:47,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1820740.0, ans=0.125
2023-11-22 05:49:47,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1820740.0, ans=0.05
2023-11-22 05:49:50,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.46 vs. limit=15.0
2023-11-22 05:49:50,455 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8600, loss[loss=0.06973, simple_loss=0.0935, pruned_loss=0.01473, audio_tagging_loss=0.008247, over 15065.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09423, pruned_loss=0.01568, audio_tagging_loss=0.009385, over 3044018.44 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:50:02,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1820873.3333333333, ans=0.125
2023-11-22 05:50:25,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273150
2023-11-22 05:50:35,538 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.265e+01 8.649e+01 9.295e+01 1.205e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 05:50:47,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1821073.3333333333, ans=0.125
2023-11-22 05:50:54,626 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8650, loss[loss=0.07144, simple_loss=0.09826, pruned_loss=0.01147, audio_tagging_loss=0.01084, over 17280.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09448, pruned_loss=0.01563, audio_tagging_loss=0.009465, over 3050470.05 frames. ], batch size: 64, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:50:57,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1821140.0, ans=0.0
2023-11-22 05:51:05,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1821140.0, ans=0.0
2023-11-22 05:51:28,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1821273.3333333333, ans=0.125
2023-11-22 05:51:29,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273200
2023-11-22 05:51:29,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1821273.3333333333, ans=0.2
2023-11-22 05:51:35,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1821340.0, ans=0.125
2023-11-22 05:51:39,933 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:51:53,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1821406.6666666667, ans=0.125
2023-11-22 05:51:59,109 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8700, loss[loss=0.05899, simple_loss=0.07704, pruned_loss=0.01111, audio_tagging_loss=0.00937, over 14731.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09566, pruned_loss=0.01596, audio_tagging_loss=0.009528, over 3054311.29 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:52:00,843 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.74 vs. limit=22.5
2023-11-22 05:52:30,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1821606.6666666667, ans=0.125
2023-11-22 05:52:33,746 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273250
2023-11-22 05:52:43,985 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 7.986e+01 8.723e+01 9.547e+01 1.201e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 05:52:47,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0
2023-11-22 05:53:03,550 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8750, loss[loss=0.0582, simple_loss=0.07427, pruned_loss=0.008797, audio_tagging_loss=0.01227, over 14984.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.0964, pruned_loss=0.01595, audio_tagging_loss=0.009487, over 3062542.33 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:53:19,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.60 vs. limit=15.0
2023-11-22 05:53:20,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1821873.3333333333, ans=0.125
2023-11-22 05:53:21,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1821873.3333333333, ans=0.125
2023-11-22 05:53:21,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1821873.3333333333, ans=0.0
2023-11-22 05:53:38,796 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273300
2023-11-22 05:54:07,853 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8800, loss[loss=0.07891, simple_loss=0.1021, pruned_loss=0.01871, audio_tagging_loss=0.009157, over 15144.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.09737, pruned_loss=0.01605, audio_tagging_loss=0.009446, over 3063302.27 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 05:54:19,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1822206.6666666667, ans=0.1
2023-11-22 05:54:42,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273350
2023-11-22 05:54:52,326 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.080e+01 8.934e+01 9.573e+01 1.295e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 05:55:08,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1822406.6666666667, ans=0.125
2023-11-22 05:55:11,584 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8850, loss[loss=0.07147, simple_loss=0.09454, pruned_loss=0.01604, audio_tagging_loss=0.008161, over 16014.00 frames. ], tot_loss[loss=0.07503, simple_loss=0.09843, pruned_loss=0.0164, audio_tagging_loss=0.009421, over 3062873.84 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 05:55:17,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.38 vs. limit=22.5
2023-11-22 05:55:23,823 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:55:25,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1822540.0, ans=0.125
2023-11-22 05:55:46,295 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273400
2023-11-22 05:55:54,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1822673.3333333333, ans=0.1
2023-11-22 05:56:01,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1822673.3333333333, ans=0.1
2023-11-22 05:56:12,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.42 vs. limit=15.0
2023-11-22 05:56:16,868 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8900, loss[loss=0.08628, simple_loss=0.1182, pruned_loss=0.0204, audio_tagging_loss=0.006766, over 15696.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09835, pruned_loss=0.01643, audio_tagging_loss=0.009253, over 3050009.44 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 05:56:36,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1822873.3333333333, ans=0.1
2023-11-22 05:56:43,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1822940.0, ans=0.125
2023-11-22 05:56:43,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1822940.0, ans=0.1
2023-11-22 05:56:51,054 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273450
2023-11-22 05:56:51,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1822940.0, ans=0.0
2023-11-22 05:56:58,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1823006.6666666667, ans=0.5
2023-11-22 05:57:03,198 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.200e+01 8.739e+01 9.306e+01 1.295e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 05:57:20,951 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 8950, loss[loss=0.08718, simple_loss=0.1132, pruned_loss=0.02108, audio_tagging_loss=0.009488, over 14687.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09785, pruned_loss=0.01609, audio_tagging_loss=0.009154, over 3050212.23 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:57:32,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1823206.6666666667, ans=0.0
2023-11-22 05:57:35,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1823206.6666666667, ans=0.0
2023-11-22 05:57:48,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1823273.3333333333, ans=0.0
2023-11-22 05:57:56,302 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273500
2023-11-22 05:57:58,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=1823273.3333333333, ans=15.0
2023-11-22 05:58:01,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.34 vs. limit=15.0
2023-11-22 05:58:20,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1823406.6666666667, ans=0.0
2023-11-22 05:58:25,510 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9000, loss[loss=0.08063, simple_loss=0.1098, pruned_loss=0.01645, audio_tagging_loss=0.009305, over 15329.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09756, pruned_loss=0.01605, audio_tagging_loss=0.009071, over 3052793.00 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:58:25,510 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 05:58:46,886 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9016, 3.6243, 4.8769, 4.2993], device='cuda:2')
2023-11-22 05:59:06,013 INFO [train_asr.py:1253] (2/4) Epoch 23, validation: loss=0.06035, simple_loss=0.05169, pruned_loss=0.005137, audio_tagging_loss=0.02937, over 4681554.00 frames. 
2023-11-22 05:59:06,014 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 05:59:09,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1823473.3333333333, ans=0.0
2023-11-22 05:59:20,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1823540.0, ans=0.125
2023-11-22 05:59:35,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1823606.6666666667, ans=0.09899494936611666
2023-11-22 05:59:40,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273550
2023-11-22 05:59:46,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1823673.3333333333, ans=0.125
2023-11-22 05:59:52,549 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.165e+01 8.843e+01 9.497e+01 1.751e+02, threshold=1.769e+02, percent-clipped=1.0
2023-11-22 06:00:10,306 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9050, loss[loss=0.06542, simple_loss=0.09041, pruned_loss=0.01219, audio_tagging_loss=0.008025, over 15208.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09545, pruned_loss=0.01579, audio_tagging_loss=0.009255, over 3055351.18 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:00:19,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1823806.6666666667, ans=0.125
2023-11-22 06:00:25,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1823873.3333333333, ans=0.0
2023-11-22 06:00:45,202 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273600
2023-11-22 06:01:14,748 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9100, loss[loss=0.08033, simple_loss=0.1088, pruned_loss=0.01786, audio_tagging_loss=0.008078, over 15441.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09527, pruned_loss=0.01582, audio_tagging_loss=0.00924, over 3051989.59 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:01:31,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1824206.6666666667, ans=0.1
2023-11-22 06:01:42,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1824273.3333333333, ans=0.04949747468305833
2023-11-22 06:01:49,078 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=22.5
2023-11-22 06:01:49,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273650
2023-11-22 06:01:55,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1824340.0, ans=0.2
2023-11-22 06:02:01,426 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.073e+01 8.633e+01 9.456e+01 1.130e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-22 06:02:10,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1824406.6666666667, ans=0.0
2023-11-22 06:02:12,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1824406.6666666667, ans=0.2
2023-11-22 06:02:19,621 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9150, loss[loss=0.0601, simple_loss=0.07744, pruned_loss=0.0105, audio_tagging_loss=0.01089, over 15441.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09387, pruned_loss=0.01557, audio_tagging_loss=0.009262, over 3048435.28 frames. ], batch size: 60, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:02:32,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1824540.0, ans=0.2
2023-11-22 06:02:54,855 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273700
2023-11-22 06:03:00,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1824673.3333333333, ans=0.2
2023-11-22 06:03:04,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1824673.3333333333, ans=0.125
2023-11-22 06:03:04,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1824673.3333333333, ans=0.125
2023-11-22 06:03:25,052 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9200, loss[loss=0.07847, simple_loss=0.09923, pruned_loss=0.01789, audio_tagging_loss=0.01096, over 14781.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09437, pruned_loss=0.01566, audio_tagging_loss=0.009226, over 3052443.51 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:03:33,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1824806.6666666667, ans=0.125
2023-11-22 06:03:59,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273750
2023-11-22 06:04:09,928 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.40 vs. limit=22.5
2023-11-22 06:04:12,809 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.119e+01 8.781e+01 9.554e+01 1.352e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 06:04:14,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.91 vs. limit=15.0
2023-11-22 06:04:29,576 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9250, loss[loss=0.06792, simple_loss=0.08551, pruned_loss=0.01499, audio_tagging_loss=0.01017, over 14757.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09397, pruned_loss=0.01568, audio_tagging_loss=0.009281, over 3044753.00 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:04:33,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1825140.0, ans=0.125
2023-11-22 06:04:47,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1825206.6666666667, ans=0.125
2023-11-22 06:05:05,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273800
2023-11-22 06:05:35,257 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9300, loss[loss=0.07659, simple_loss=0.1013, pruned_loss=0.0169, audio_tagging_loss=0.009051, over 14429.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.09369, pruned_loss=0.01542, audio_tagging_loss=0.009363, over 3047070.33 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:05:56,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1825540.0, ans=0.0
2023-11-22 06:06:07,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1825606.6666666667, ans=0.95
2023-11-22 06:06:11,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273850
2023-11-22 06:06:24,303 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.188e+01 8.737e+01 9.420e+01 1.693e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 06:06:32,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.47 vs. limit=5.0
2023-11-22 06:06:34,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1825740.0, ans=0.125
2023-11-22 06:06:40,089 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.77 vs. limit=15.0
2023-11-22 06:06:40,814 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9350, loss[loss=0.0486, simple_loss=0.0648, pruned_loss=0.007305, audio_tagging_loss=0.008899, over 15013.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09404, pruned_loss=0.01535, audio_tagging_loss=0.009357, over 3049538.25 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:06:45,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1825806.6666666667, ans=0.125
2023-11-22 06:07:15,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273900
2023-11-22 06:07:17,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1825940.0, ans=0.5
2023-11-22 06:07:21,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1826006.6666666667, ans=22.5
2023-11-22 06:07:30,720 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.14 vs. limit=15.0
2023-11-22 06:07:45,339 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=10.02 vs. limit=10.0
2023-11-22 06:07:45,840 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9400, loss[loss=0.05621, simple_loss=0.07018, pruned_loss=0.01025, audio_tagging_loss=0.01087, over 13966.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09481, pruned_loss=0.01551, audio_tagging_loss=0.009396, over 3043591.51 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:07:57,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.81 vs. limit=15.0
2023-11-22 06:08:12,028 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.42 vs. limit=6.0
2023-11-22 06:08:19,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1826273.3333333333, ans=0.1
2023-11-22 06:08:20,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 273950
2023-11-22 06:08:27,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1826340.0, ans=0.125
2023-11-22 06:08:34,420 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.127e+01 8.861e+01 9.548e+01 1.246e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 06:08:48,506 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:08:50,910 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9450, loss[loss=0.07108, simple_loss=0.08806, pruned_loss=0.01462, audio_tagging_loss=0.01243, over 15231.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09459, pruned_loss=0.01549, audio_tagging_loss=0.009458, over 3042182.74 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:08:52,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1826473.3333333333, ans=10.0
2023-11-22 06:09:04,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1826540.0, ans=0.1
2023-11-22 06:09:18,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1826606.6666666667, ans=0.125
2023-11-22 06:09:22,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1826606.6666666667, ans=0.125
2023-11-22 06:09:25,877 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274000
2023-11-22 06:09:55,598 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9500, loss[loss=0.05812, simple_loss=0.0781, pruned_loss=0.01079, audio_tagging_loss=0.008278, over 15541.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09395, pruned_loss=0.01537, audio_tagging_loss=0.009549, over 3048574.72 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:10:13,682 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.94 vs. limit=10.0
2023-11-22 06:10:26,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1826940.0, ans=0.125
2023-11-22 06:10:30,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.08 vs. limit=15.0
2023-11-22 06:10:31,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274050
2023-11-22 06:10:31,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1826940.0, ans=0.2
2023-11-22 06:10:32,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1826940.0, ans=0.0
2023-11-22 06:10:43,863 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.476e+01 8.082e+01 8.870e+01 9.586e+01 1.219e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 06:10:53,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1827073.3333333333, ans=0.2
2023-11-22 06:11:00,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1827140.0, ans=0.125
2023-11-22 06:11:01,487 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9550, loss[loss=0.06942, simple_loss=0.08442, pruned_loss=0.01592, audio_tagging_loss=0.01129, over 13994.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09378, pruned_loss=0.01531, audio_tagging_loss=0.009737, over 3046357.93 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:11:01,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1827140.0, ans=0.95
2023-11-22 06:11:09,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1827140.0, ans=0.0
2023-11-22 06:11:36,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274100
2023-11-22 06:11:45,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1827340.0, ans=0.2
2023-11-22 06:12:04,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.37 vs. limit=15.0
2023-11-22 06:12:06,024 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9600, loss[loss=0.08266, simple_loss=0.1189, pruned_loss=0.01618, audio_tagging_loss=0.00704, over 14667.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09401, pruned_loss=0.01531, audio_tagging_loss=0.009759, over 3051281.30 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:12:20,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1827540.0, ans=0.125
2023-11-22 06:12:22,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1827540.0, ans=0.125
2023-11-22 06:12:40,922 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274150
2023-11-22 06:12:42,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1827606.6666666667, ans=0.125
2023-11-22 06:12:53,499 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.292e+01 9.125e+01 9.909e+01 1.275e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-22 06:12:58,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1827740.0, ans=0.95
2023-11-22 06:13:01,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1827740.0, ans=0.2
2023-11-22 06:13:09,792 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9650, loss[loss=0.06552, simple_loss=0.07583, pruned_loss=0.01604, audio_tagging_loss=0.01156, over 15293.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09359, pruned_loss=0.01525, audio_tagging_loss=0.009615, over 3045902.04 frames. ], batch size: 61, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:13:16,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1827806.6666666667, ans=0.0
2023-11-22 06:13:16,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1827806.6666666667, ans=6.0
2023-11-22 06:13:17,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1827806.6666666667, ans=0.2
2023-11-22 06:13:25,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1827873.3333333333, ans=0.125
2023-11-22 06:13:26,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.20 vs. limit=15.0
2023-11-22 06:13:32,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1827873.3333333333, ans=0.09899494936611666
2023-11-22 06:13:45,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274200
2023-11-22 06:13:59,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1828006.6666666667, ans=0.125
2023-11-22 06:14:14,380 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9700, loss[loss=0.08157, simple_loss=0.1059, pruned_loss=0.01704, audio_tagging_loss=0.01158, over 15621.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09398, pruned_loss=0.01533, audio_tagging_loss=0.009487, over 3046896.65 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:14:19,563 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:14:23,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1828140.0, ans=0.125
2023-11-22 06:14:24,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1828140.0, ans=0.1
2023-11-22 06:14:49,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274250
2023-11-22 06:14:51,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1828340.0, ans=0.0
2023-11-22 06:14:59,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1828340.0, ans=0.2
2023-11-22 06:15:01,328 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.325e+01 8.777e+01 9.483e+01 1.567e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 06:15:09,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1828406.6666666667, ans=0.05
2023-11-22 06:15:15,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1828406.6666666667, ans=0.1
2023-11-22 06:15:19,146 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9750, loss[loss=0.05945, simple_loss=0.07101, pruned_loss=0.01221, audio_tagging_loss=0.01174, over 16509.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09346, pruned_loss=0.01525, audio_tagging_loss=0.009395, over 3048316.08 frames. ], batch size: 63, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:15:25,956 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.32 vs. limit=10.0
2023-11-22 06:15:53,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274300
2023-11-22 06:16:14,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1828740.0, ans=0.125
2023-11-22 06:16:23,200 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9800, loss[loss=0.06389, simple_loss=0.09107, pruned_loss=0.0111, audio_tagging_loss=0.007251, over 16004.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09325, pruned_loss=0.0153, audio_tagging_loss=0.009358, over 3056762.07 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:16:23,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1828806.6666666667, ans=0.1
2023-11-22 06:16:23,914 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-22 06:16:28,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1828806.6666666667, ans=0.125
2023-11-22 06:16:57,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1828940.0, ans=0.125
2023-11-22 06:16:58,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274350
2023-11-22 06:17:10,640 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.412e+01 8.125e+01 8.803e+01 9.670e+01 1.768e+02, threshold=1.761e+02, percent-clipped=1.0
2023-11-22 06:17:13,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.82 vs. limit=15.0
2023-11-22 06:17:19,778 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:17:27,060 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9850, loss[loss=0.07923, simple_loss=0.09844, pruned_loss=0.02013, audio_tagging_loss=0.009874, over 16844.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09464, pruned_loss=0.01554, audio_tagging_loss=0.009258, over 3055292.96 frames. ], batch size: 62, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:17:40,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1829206.6666666667, ans=0.125
2023-11-22 06:17:55,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1829273.3333333333, ans=0.1
2023-11-22 06:17:56,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1829273.3333333333, ans=0.2
2023-11-22 06:17:56,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1829273.3333333333, ans=0.1
2023-11-22 06:17:58,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1829273.3333333333, ans=0.0
2023-11-22 06:18:01,806 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274400
2023-11-22 06:18:10,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1829340.0, ans=15.0
2023-11-22 06:18:11,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1829340.0, ans=0.05
2023-11-22 06:18:31,228 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9900, loss[loss=0.05537, simple_loss=0.07302, pruned_loss=0.007268, audio_tagging_loss=0.01159, over 13759.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09487, pruned_loss=0.01553, audio_tagging_loss=0.009205, over 3047513.31 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:18:39,638 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.78 vs. limit=6.0
2023-11-22 06:18:50,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1829540.0, ans=0.0
2023-11-22 06:18:59,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1829606.6666666667, ans=0.0
2023-11-22 06:18:59,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1829606.6666666667, ans=0.125
2023-11-22 06:19:01,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1829606.6666666667, ans=0.125
2023-11-22 06:19:05,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274450
2023-11-22 06:19:17,950 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.069e+01 8.830e+01 9.400e+01 1.279e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 06:19:35,134 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 9950, loss[loss=0.08064, simple_loss=0.1055, pruned_loss=0.0175, audio_tagging_loss=0.0104, over 16076.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.09589, pruned_loss=0.01572, audio_tagging_loss=0.009215, over 3053618.98 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:19:37,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.63 vs. limit=12.0
2023-11-22 06:19:42,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1829806.6666666667, ans=0.1
2023-11-22 06:19:54,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1829873.3333333333, ans=0.125
2023-11-22 06:20:09,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274500
2023-11-22 06:20:22,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1830006.6666666667, ans=0.1
2023-11-22 06:20:25,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1830073.3333333333, ans=0.125
2023-11-22 06:20:30,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1830073.3333333333, ans=0.125
2023-11-22 06:20:39,463 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10000, loss[loss=0.05606, simple_loss=0.07629, pruned_loss=0.008585, audio_tagging_loss=0.009327, over 16359.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.0951, pruned_loss=0.01555, audio_tagging_loss=0.009221, over 3048514.38 frames. ], batch size: 60, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:21:07,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0
2023-11-22 06:21:14,527 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274550
2023-11-22 06:21:17,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1830340.0, ans=0.1
2023-11-22 06:21:27,653 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:21:28,399 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.149e+01 8.699e+01 9.689e+01 1.229e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 06:21:31,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1830406.6666666667, ans=0.1
2023-11-22 06:21:43,795 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10050, loss[loss=0.06709, simple_loss=0.08548, pruned_loss=0.01625, audio_tagging_loss=0.008101, over 14267.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09459, pruned_loss=0.01531, audio_tagging_loss=0.009202, over 3044691.20 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:22:02,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1830540.0, ans=0.5
2023-11-22 06:22:18,475 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274600
2023-11-22 06:22:48,256 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10100, loss[loss=0.07036, simple_loss=0.08628, pruned_loss=0.01256, audio_tagging_loss=0.01466, over 14283.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09406, pruned_loss=0.01521, audio_tagging_loss=0.009326, over 3042014.68 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:22:53,661 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.99 vs. limit=15.0
2023-11-22 06:23:22,220 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274650
2023-11-22 06:23:23,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1830940.0, ans=0.125
2023-11-22 06:23:36,109 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 8.114e+01 8.687e+01 9.390e+01 1.135e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 06:23:39,244 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:23:44,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1831073.3333333333, ans=0.125
2023-11-22 06:23:52,060 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10150, loss[loss=0.07132, simple_loss=0.09074, pruned_loss=0.01775, audio_tagging_loss=0.0082, over 15888.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09401, pruned_loss=0.01531, audio_tagging_loss=0.009365, over 3042170.50 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:23:57,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1831140.0, ans=0.0
2023-11-22 06:24:02,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1831140.0, ans=0.0
2023-11-22 06:24:07,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1831206.6666666667, ans=0.125
2023-11-22 06:24:14,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1831206.6666666667, ans=0.0
2023-11-22 06:24:21,541 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:24:21,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1831273.3333333333, ans=0.0
2023-11-22 06:24:25,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1831273.3333333333, ans=0.2
2023-11-22 06:24:26,617 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274700
2023-11-22 06:24:37,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1831340.0, ans=0.125
2023-11-22 06:24:39,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1831340.0, ans=0.0
2023-11-22 06:24:56,403 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10200, loss[loss=0.07608, simple_loss=0.09106, pruned_loss=0.01835, audio_tagging_loss=0.0122, over 14102.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.0937, pruned_loss=0.01538, audio_tagging_loss=0.009367, over 3036759.36 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:24:56,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-22 06:25:15,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1831540.0, ans=0.125
2023-11-22 06:25:19,852 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:25:26,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1831606.6666666667, ans=0.1
2023-11-22 06:25:31,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274750
2023-11-22 06:25:32,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1831606.6666666667, ans=0.125
2023-11-22 06:25:45,213 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.234e+01 8.918e+01 9.479e+01 1.480e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 06:25:54,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1831740.0, ans=0.0
2023-11-22 06:26:00,407 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10250, loss[loss=0.05604, simple_loss=0.07336, pruned_loss=0.009933, audio_tagging_loss=0.009427, over 15311.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09373, pruned_loss=0.0153, audio_tagging_loss=0.009462, over 3043852.57 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:26:03,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1831806.6666666667, ans=0.0
2023-11-22 06:26:19,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1831873.3333333333, ans=0.125
2023-11-22 06:26:27,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1831940.0, ans=0.125
2023-11-22 06:26:33,288 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0
2023-11-22 06:26:35,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274800
2023-11-22 06:26:42,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1832006.6666666667, ans=0.07
2023-11-22 06:26:42,616 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.21 vs. limit=10.0
2023-11-22 06:26:48,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.45 vs. limit=15.0
2023-11-22 06:27:05,158 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10300, loss[loss=0.07426, simple_loss=0.09507, pruned_loss=0.01872, audio_tagging_loss=0.008003, over 14486.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09414, pruned_loss=0.01547, audio_tagging_loss=0.009601, over 3049026.29 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:27:15,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1832140.0, ans=0.125
2023-11-22 06:27:18,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1832206.6666666667, ans=0.125
2023-11-22 06:27:31,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.50 vs. limit=15.0
2023-11-22 06:27:39,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274850
2023-11-22 06:27:39,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1832273.3333333333, ans=0.2
2023-11-22 06:27:53,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.276e+01 9.014e+01 9.790e+01 1.332e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-22 06:27:57,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1832406.6666666667, ans=0.125
2023-11-22 06:27:57,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1832406.6666666667, ans=0.0
2023-11-22 06:28:03,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1832406.6666666667, ans=0.0
2023-11-22 06:28:09,396 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10350, loss[loss=0.07716, simple_loss=0.09517, pruned_loss=0.0171, audio_tagging_loss=0.01248, over 16116.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09509, pruned_loss=0.01567, audio_tagging_loss=0.009674, over 3044595.98 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:28:13,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1832473.3333333333, ans=0.2
2023-11-22 06:28:18,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1832473.3333333333, ans=0.2
2023-11-22 06:28:28,876 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.30 vs. limit=15.0
2023-11-22 06:28:37,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.57 vs. limit=15.0
2023-11-22 06:28:44,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274900
2023-11-22 06:28:46,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1832673.3333333333, ans=0.125
2023-11-22 06:28:57,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1832673.3333333333, ans=0.2
2023-11-22 06:28:59,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1832740.0, ans=0.5
2023-11-22 06:29:04,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.72 vs. limit=12.0
2023-11-22 06:29:11,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1832740.0, ans=15.0
2023-11-22 06:29:13,102 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10400, loss[loss=0.05531, simple_loss=0.06803, pruned_loss=0.01175, audio_tagging_loss=0.009554, over 15049.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09394, pruned_loss=0.01535, audio_tagging_loss=0.009746, over 3040382.59 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:29:26,126 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.47 vs. limit=6.0
2023-11-22 06:29:41,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1832940.0, ans=0.1
2023-11-22 06:29:47,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 274950
2023-11-22 06:30:01,645 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.321e+01 8.235e+01 8.763e+01 9.443e+01 1.759e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 06:30:17,055 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10450, loss[loss=0.06643, simple_loss=0.08027, pruned_loss=0.01426, audio_tagging_loss=0.01203, over 16058.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09363, pruned_loss=0.01534, audio_tagging_loss=0.009722, over 3049481.28 frames. ], batch size: 62, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:30:26,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1833140.0, ans=0.125
2023-11-22 06:30:36,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1833206.6666666667, ans=0.0
2023-11-22 06:30:51,966 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275000
2023-11-22 06:30:52,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1833273.3333333333, ans=0.1
2023-11-22 06:31:15,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1833406.6666666667, ans=0.1
2023-11-22 06:31:15,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1833406.6666666667, ans=0.5
2023-11-22 06:31:21,973 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10500, loss[loss=0.08343, simple_loss=0.1152, pruned_loss=0.01804, audio_tagging_loss=0.007773, over 15738.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09251, pruned_loss=0.01515, audio_tagging_loss=0.009668, over 3049069.11 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:31:22,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1833473.3333333333, ans=0.0
2023-11-22 06:31:26,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.96 vs. limit=15.0
2023-11-22 06:31:43,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1833540.0, ans=0.125
2023-11-22 06:31:56,532 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275050
2023-11-22 06:32:10,960 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.246e+01 8.781e+01 9.587e+01 1.309e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 06:32:16,693 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.05 vs. limit=15.0
2023-11-22 06:32:26,266 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10550, loss[loss=0.07914, simple_loss=0.1081, pruned_loss=0.01821, audio_tagging_loss=0.006878, over 15664.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09239, pruned_loss=0.0151, audio_tagging_loss=0.00959, over 3045626.70 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:32:32,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.35 vs. limit=22.5
2023-11-22 06:32:33,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1833806.6666666667, ans=0.95
2023-11-22 06:32:33,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1833806.6666666667, ans=0.125
2023-11-22 06:33:00,715 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275100
2023-11-22 06:33:04,840 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0
2023-11-22 06:33:08,528 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.01 vs. limit=15.0
2023-11-22 06:33:29,413 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10600, loss[loss=0.07223, simple_loss=0.0989, pruned_loss=0.01352, audio_tagging_loss=0.009261, over 14234.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09338, pruned_loss=0.01543, audio_tagging_loss=0.00945, over 3047667.94 frames. ], batch size: 53, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:33:43,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1834206.6666666667, ans=0.125
2023-11-22 06:34:00,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1834273.3333333333, ans=15.0
2023-11-22 06:34:05,845 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275150
2023-11-22 06:34:19,497 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.081e+01 8.805e+01 9.218e+01 1.246e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 06:34:25,872 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.18 vs. limit=15.0
2023-11-22 06:34:31,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1834406.6666666667, ans=0.125
2023-11-22 06:34:36,102 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10650, loss[loss=0.06429, simple_loss=0.07703, pruned_loss=0.01395, audio_tagging_loss=0.01183, over 14976.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09446, pruned_loss=0.01569, audio_tagging_loss=0.009326, over 3040321.11 frames. ], batch size: 59, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:34:48,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.75 vs. limit=6.0
2023-11-22 06:35:06,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1834606.6666666667, ans=0.0
2023-11-22 06:35:10,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275200
2023-11-22 06:35:23,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1834673.3333333333, ans=0.125
2023-11-22 06:35:24,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1834673.3333333333, ans=0.1
2023-11-22 06:35:41,429 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10700, loss[loss=0.08124, simple_loss=0.1021, pruned_loss=0.02119, audio_tagging_loss=0.008985, over 14351.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09415, pruned_loss=0.01571, audio_tagging_loss=0.009389, over 3029456.60 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:35:47,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=1834806.6666666667, ans=0.025
2023-11-22 06:35:50,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1834806.6666666667, ans=0.125
2023-11-22 06:36:08,111 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:36:15,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275250
2023-11-22 06:36:28,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.67 vs. limit=15.0
2023-11-22 06:36:30,350 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.298e+01 8.136e+01 8.812e+01 9.444e+01 1.567e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 06:36:42,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1835073.3333333333, ans=0.0
2023-11-22 06:36:44,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1835140.0, ans=0.1
2023-11-22 06:36:45,210 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10750, loss[loss=0.07217, simple_loss=0.09568, pruned_loss=0.01701, audio_tagging_loss=0.007317, over 15393.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09415, pruned_loss=0.01571, audio_tagging_loss=0.009319, over 3044770.79 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:36:51,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1835140.0, ans=0.2
2023-11-22 06:37:02,756 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:37:21,030 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275300
2023-11-22 06:37:49,824 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10800, loss[loss=0.08009, simple_loss=0.1131, pruned_loss=0.01606, audio_tagging_loss=0.007477, over 15544.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09411, pruned_loss=0.01565, audio_tagging_loss=0.009212, over 3040769.62 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:37:56,158 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.36 vs. limit=15.0
2023-11-22 06:38:25,507 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275350
2023-11-22 06:38:26,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1835606.6666666667, ans=0.1
2023-11-22 06:38:38,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1835673.3333333333, ans=0.1
2023-11-22 06:38:41,119 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.004e+01 8.229e+01 8.855e+01 9.329e+01 1.142e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 06:38:56,679 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10850, loss[loss=0.07839, simple_loss=0.1072, pruned_loss=0.01532, audio_tagging_loss=0.009478, over 14527.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09419, pruned_loss=0.01555, audio_tagging_loss=0.009217, over 3044397.76 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:39:28,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1835940.0, ans=0.0
2023-11-22 06:39:31,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275400
2023-11-22 06:39:38,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.44 vs. limit=15.0
2023-11-22 06:39:47,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1836006.6666666667, ans=0.1
2023-11-22 06:39:56,755 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:40:01,513 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10900, loss[loss=0.06993, simple_loss=0.09086, pruned_loss=0.01571, audio_tagging_loss=0.008782, over 15227.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09432, pruned_loss=0.01552, audio_tagging_loss=0.009238, over 3047349.59 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:40:04,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1836140.0, ans=0.125
2023-11-22 06:40:17,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1836206.6666666667, ans=0.1
2023-11-22 06:40:37,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275450
2023-11-22 06:40:38,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1836273.3333333333, ans=0.0
2023-11-22 06:40:51,643 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.279e+01 8.746e+01 9.747e+01 1.923e+02, threshold=1.749e+02, percent-clipped=1.0
2023-11-22 06:41:01,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1836406.6666666667, ans=0.0
2023-11-22 06:41:03,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1836406.6666666667, ans=0.2
2023-11-22 06:41:05,875 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 10950, loss[loss=0.08139, simple_loss=0.1118, pruned_loss=0.01682, audio_tagging_loss=0.008651, over 16181.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09464, pruned_loss=0.01546, audio_tagging_loss=0.00923, over 3049034.38 frames. ], batch size: 59, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:41:09,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0
2023-11-22 06:41:12,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1836473.3333333333, ans=0.125
2023-11-22 06:41:40,357 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275500
2023-11-22 06:42:07,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1836740.0, ans=0.125
2023-11-22 06:42:09,620 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11000, loss[loss=0.07055, simple_loss=0.08247, pruned_loss=0.01939, audio_tagging_loss=0.009925, over 15020.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09496, pruned_loss=0.01546, audio_tagging_loss=0.009267, over 3046952.92 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:42:20,678 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:42:27,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1836873.3333333333, ans=0.025
2023-11-22 06:42:29,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1836873.3333333333, ans=0.125
2023-11-22 06:42:44,584 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275550
2023-11-22 06:43:00,978 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.356e+01 8.870e+01 9.515e+01 1.517e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 06:43:10,016 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:43:14,625 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11050, loss[loss=0.04526, simple_loss=0.05461, pruned_loss=0.006318, audio_tagging_loss=0.01164, over 15323.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09503, pruned_loss=0.01554, audio_tagging_loss=0.009395, over 3050256.03 frames. ], batch size: 59, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:43:20,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1837140.0, ans=0.125
2023-11-22 06:43:39,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1837273.3333333333, ans=0.0
2023-11-22 06:43:42,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1837273.3333333333, ans=0.125
2023-11-22 06:43:49,031 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275600
2023-11-22 06:43:53,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.37 vs. limit=22.5
2023-11-22 06:43:54,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1837340.0, ans=0.0
2023-11-22 06:44:06,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.82 vs. limit=15.0
2023-11-22 06:44:08,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1837406.6666666667, ans=0.2
2023-11-22 06:44:18,962 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11100, loss[loss=0.07024, simple_loss=0.09798, pruned_loss=0.01425, audio_tagging_loss=0.00701, over 15206.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09521, pruned_loss=0.01541, audio_tagging_loss=0.009465, over 3051445.64 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:44:36,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1837540.0, ans=0.0
2023-11-22 06:44:53,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275650
2023-11-22 06:45:09,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1837740.0, ans=0.0
2023-11-22 06:45:11,536 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.680e+01 8.196e+01 8.803e+01 9.419e+01 1.192e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 06:45:11,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1837740.0, ans=0.0
2023-11-22 06:45:20,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1837740.0, ans=0.0
2023-11-22 06:45:23,225 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11150, loss[loss=0.05785, simple_loss=0.06845, pruned_loss=0.00908, audio_tagging_loss=0.01454, over 15900.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09426, pruned_loss=0.01528, audio_tagging_loss=0.00965, over 3051418.00 frames. ], batch size: 62, lr: 2.94e-03, grad_scale: 8.0
2023-11-22 06:45:25,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.87 vs. limit=22.5
2023-11-22 06:45:58,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275700
2023-11-22 06:46:01,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1838006.6666666667, ans=22.5
2023-11-22 06:46:07,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1838006.6666666667, ans=0.1
2023-11-22 06:46:28,409 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11200, loss[loss=0.06193, simple_loss=0.0817, pruned_loss=0.01212, audio_tagging_loss=0.00896, over 14276.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09445, pruned_loss=0.01533, audio_tagging_loss=0.009587, over 3045280.99 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:46:30,318 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.65 vs. limit=15.0
2023-11-22 06:46:34,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1838140.0, ans=0.2
2023-11-22 06:46:39,216 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.22 vs. limit=15.0
2023-11-22 06:47:02,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275750
2023-11-22 06:47:14,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1838340.0, ans=0.125
2023-11-22 06:47:17,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1838340.0, ans=0.125
2023-11-22 06:47:21,153 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.197e+01 8.985e+01 9.663e+01 1.096e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 06:47:32,686 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11250, loss[loss=0.06876, simple_loss=0.09355, pruned_loss=0.01416, audio_tagging_loss=0.007823, over 15146.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.095, pruned_loss=0.0155, audio_tagging_loss=0.009501, over 3043713.53 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:47:40,937 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.63 vs. limit=15.0
2023-11-22 06:47:56,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1838540.0, ans=0.2
2023-11-22 06:48:07,797 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275800
2023-11-22 06:48:16,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.97 vs. limit=15.0
2023-11-22 06:48:17,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1838673.3333333333, ans=0.125
2023-11-22 06:48:23,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1838673.3333333333, ans=0.2
2023-11-22 06:48:24,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1838740.0, ans=0.2
2023-11-22 06:48:38,271 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11300, loss[loss=0.05825, simple_loss=0.07613, pruned_loss=0.01159, audio_tagging_loss=0.008593, over 15809.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09446, pruned_loss=0.01551, audio_tagging_loss=0.009382, over 3052750.60 frames. ], batch size: 60, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:49:02,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1838873.3333333333, ans=0.0
2023-11-22 06:49:13,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275850
2023-11-22 06:49:20,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.61 vs. limit=15.0
2023-11-22 06:49:31,033 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 7.924e+01 8.561e+01 9.574e+01 1.338e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-22 06:49:42,773 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11350, loss[loss=0.06945, simple_loss=0.08403, pruned_loss=0.01697, audio_tagging_loss=0.01047, over 15019.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09412, pruned_loss=0.01544, audio_tagging_loss=0.009259, over 3050173.89 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:49:44,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.32 vs. limit=15.0
2023-11-22 06:49:56,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1839206.6666666667, ans=0.0
2023-11-22 06:50:01,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1839206.6666666667, ans=0.0
2023-11-22 06:50:17,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275900
2023-11-22 06:50:19,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1839273.3333333333, ans=0.125
2023-11-22 06:50:25,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1839340.0, ans=0.125
2023-11-22 06:50:28,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1839340.0, ans=0.05
2023-11-22 06:50:37,446 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.82 vs. limit=10.0
2023-11-22 06:50:47,660 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11400, loss[loss=0.0735, simple_loss=0.09979, pruned_loss=0.01707, audio_tagging_loss=0.006536, over 14971.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09462, pruned_loss=0.01548, audio_tagging_loss=0.009173, over 3044772.05 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:50:56,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1839473.3333333333, ans=0.125
2023-11-22 06:51:07,321 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.92 vs. limit=15.0
2023-11-22 06:51:08,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1839540.0, ans=0.07
2023-11-22 06:51:22,122 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 275950
2023-11-22 06:51:26,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten.whitening_limit, batch_count=1839673.3333333333, ans=22.5
2023-11-22 06:51:27,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1839673.3333333333, ans=0.0
2023-11-22 06:51:28,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1839673.3333333333, ans=0.125
2023-11-22 06:51:33,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1839673.3333333333, ans=0.125
2023-11-22 06:51:40,304 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.209e+01 8.167e+01 8.873e+01 9.595e+01 1.109e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 06:51:51,929 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11450, loss[loss=0.0753, simple_loss=0.1051, pruned_loss=0.01451, audio_tagging_loss=0.008255, over 14542.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.09375, pruned_loss=0.0154, audio_tagging_loss=0.009205, over 3037526.79 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:51:57,418 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.95 vs. limit=15.0
2023-11-22 06:51:59,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1839806.6666666667, ans=0.125
2023-11-22 06:52:05,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1839873.3333333333, ans=0.125
2023-11-22 06:52:06,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1839873.3333333333, ans=0.1
2023-11-22 06:52:27,306 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276000
2023-11-22 06:52:59,777 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11500, loss[loss=0.09501, simple_loss=0.1243, pruned_loss=0.02556, audio_tagging_loss=0.007276, over 14317.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09367, pruned_loss=0.01529, audio_tagging_loss=0.009214, over 3038739.74 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:53:07,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1840140.0, ans=0.1
2023-11-22 06:53:08,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1840140.0, ans=0.125
2023-11-22 06:53:09,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.33 vs. limit=15.0
2023-11-22 06:53:34,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276050
2023-11-22 06:53:46,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1840340.0, ans=0.0
2023-11-22 06:53:52,664 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.162e+01 8.710e+01 9.515e+01 1.226e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 06:53:59,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1840406.6666666667, ans=0.125
2023-11-22 06:54:03,884 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11550, loss[loss=0.06128, simple_loss=0.07161, pruned_loss=0.01419, audio_tagging_loss=0.01128, over 14807.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09449, pruned_loss=0.01548, audio_tagging_loss=0.00925, over 3042629.03 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:54:04,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1840473.3333333333, ans=0.125
2023-11-22 06:54:33,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1840606.6666666667, ans=0.125
2023-11-22 06:54:39,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276100
2023-11-22 06:54:44,960 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:54:55,525 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.01 vs. limit=15.0
2023-11-22 06:55:09,702 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11600, loss[loss=0.06952, simple_loss=0.08732, pruned_loss=0.01394, audio_tagging_loss=0.01191, over 15037.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09501, pruned_loss=0.01575, audio_tagging_loss=0.009281, over 3045043.65 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:55:11,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1840806.6666666667, ans=0.2
2023-11-22 06:55:19,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1840806.6666666667, ans=0.125
2023-11-22 06:55:31,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1840873.3333333333, ans=0.0
2023-11-22 06:55:33,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1840873.3333333333, ans=0.07
2023-11-22 06:55:37,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1840940.0, ans=0.04949747468305833
2023-11-22 06:55:44,367 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276150
2023-11-22 06:56:03,208 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.255e+01 8.370e+01 8.963e+01 9.598e+01 1.244e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-22 06:56:15,233 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11650, loss[loss=0.09011, simple_loss=0.1196, pruned_loss=0.0208, audio_tagging_loss=0.009542, over 15982.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09436, pruned_loss=0.01564, audio_tagging_loss=0.009441, over 3048587.78 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:56:24,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1841140.0, ans=0.0
2023-11-22 06:56:29,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1841206.6666666667, ans=0.1
2023-11-22 06:56:32,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1841206.6666666667, ans=0.0
2023-11-22 06:56:39,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1841273.3333333333, ans=0.1
2023-11-22 06:56:50,391 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276200
2023-11-22 06:57:07,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1841406.6666666667, ans=0.0
2023-11-22 06:57:19,040 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11700, loss[loss=0.07452, simple_loss=0.09833, pruned_loss=0.01519, audio_tagging_loss=0.01017, over 16254.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.09445, pruned_loss=0.01576, audio_tagging_loss=0.009485, over 3050234.79 frames. ], batch size: 60, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:57:25,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1841473.3333333333, ans=0.125
2023-11-22 06:57:46,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1841606.6666666667, ans=0.0
2023-11-22 06:57:47,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.73 vs. limit=22.5
2023-11-22 06:57:52,111 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.19 vs. limit=15.0
2023-11-22 06:57:54,714 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276250
2023-11-22 06:57:54,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1841606.6666666667, ans=0.05
2023-11-22 06:58:06,116 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.99 vs. limit=15.0
2023-11-22 06:58:11,823 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.445e+01 9.108e+01 1.005e+02 1.343e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-22 06:58:16,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1841740.0, ans=0.125
2023-11-22 06:58:18,850 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:58:24,072 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11750, loss[loss=0.0579, simple_loss=0.07271, pruned_loss=0.01068, audio_tagging_loss=0.01086, over 14540.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09382, pruned_loss=0.01555, audio_tagging_loss=0.009522, over 3051707.16 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:58:42,693 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=2.554e-03
2023-11-22 06:58:58,288 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276300
2023-11-22 06:59:02,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1842006.6666666667, ans=0.0
2023-11-22 06:59:21,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1842073.3333333333, ans=0.0
2023-11-22 06:59:28,428 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11800, loss[loss=0.07209, simple_loss=0.09376, pruned_loss=0.01304, audio_tagging_loss=0.01217, over 15715.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09394, pruned_loss=0.01551, audio_tagging_loss=0.009627, over 3048280.49 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:59:32,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.02 vs. limit=15.0
2023-11-22 06:59:59,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1842273.3333333333, ans=0.0
2023-11-22 07:00:02,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1842273.3333333333, ans=0.0
2023-11-22 07:00:03,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276350
2023-11-22 07:00:11,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1842340.0, ans=0.0
2023-11-22 07:00:23,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1842406.6666666667, ans=0.125
2023-11-22 07:00:24,080 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.151e+01 8.673e+01 9.167e+01 1.144e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 07:00:31,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1842406.6666666667, ans=0.125
2023-11-22 07:00:34,075 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11850, loss[loss=0.05829, simple_loss=0.07433, pruned_loss=0.01132, audio_tagging_loss=0.009803, over 15816.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09358, pruned_loss=0.0155, audio_tagging_loss=0.009625, over 3038614.95 frames. ], batch size: 59, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 07:00:39,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1842473.3333333333, ans=0.2
2023-11-22 07:01:08,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276400
2023-11-22 07:01:38,739 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11900, loss[loss=0.07362, simple_loss=0.09619, pruned_loss=0.01491, audio_tagging_loss=0.01061, over 15719.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09407, pruned_loss=0.01544, audio_tagging_loss=0.009642, over 3048011.78 frames. ], batch size: 60, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 07:01:41,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1842806.6666666667, ans=0.07
2023-11-22 07:02:14,057 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276450
2023-11-22 07:02:33,090 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 7.969e+01 8.417e+01 9.242e+01 1.141e+02, threshold=1.683e+02, percent-clipped=0.0
2023-11-22 07:02:40,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1843073.3333333333, ans=0.125
2023-11-22 07:02:43,980 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 11950, loss[loss=0.05927, simple_loss=0.07704, pruned_loss=0.01136, audio_tagging_loss=0.009388, over 16282.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09333, pruned_loss=0.01535, audio_tagging_loss=0.009805, over 3053214.54 frames. ], batch size: 62, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 07:02:48,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.02 vs. limit=12.0
2023-11-22 07:02:50,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1843140.0, ans=0.125
2023-11-22 07:02:52,833 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0
2023-11-22 07:03:00,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1843206.6666666667, ans=0.0
2023-11-22 07:03:04,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1843206.6666666667, ans=0.125
2023-11-22 07:03:06,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1843206.6666666667, ans=0.95
2023-11-22 07:03:18,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276500
2023-11-22 07:03:22,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1843340.0, ans=0.1
2023-11-22 07:03:46,905 INFO [train_asr.py:1221] (2/4) Epoch 23, batch 12000, loss[loss=0.082, simple_loss=0.1163, pruned_loss=0.01704, audio_tagging_loss=0.006823, over 15069.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09327, pruned_loss=0.0152, audio_tagging_loss=0.009768, over 3053119.06 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 07:03:46,906 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 07:04:10,798 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.4725, 2.8249, 2.5151, 3.1520, 2.9018, 2.8800, 2.8371, 2.6480],
       device='cuda:2')
2023-11-22 07:04:14,067 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.0267, 2.9922, 3.3233, 2.9535, 3.7409, 3.7872, 3.3483, 3.1838],
       device='cuda:2')
2023-11-22 07:04:27,936 INFO [train_asr.py:1253] (2/4) Epoch 23, validation: loss=0.05966, simple_loss=0.05174, pruned_loss=0.005186, audio_tagging_loss=0.02861, over 4681554.00 frames. 
2023-11-22 07:04:27,937 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 07:04:32,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1843473.3333333333, ans=0.0
2023-11-22 07:04:41,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1843540.0, ans=0.04949747468305833
2023-11-22 07:05:34,196 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 0, loss[loss=0.07481, simple_loss=0.08483, pruned_loss=0.01305, audio_tagging_loss=0.01935, over 15415.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.08483, pruned_loss=0.01305, audio_tagging_loss=0.01935, over 15415.00 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:05:34,197 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 07:06:09,685 INFO [train_asr.py:1253] (2/4) Epoch 24, validation: loss=0.05907, simple_loss=0.05179, pruned_loss=0.005258, audio_tagging_loss=0.02792, over 4681554.00 frames. 
2023-11-22 07:06:09,685 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 07:06:13,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276550
2023-11-22 07:06:13,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1843633.3333333333, ans=0.1
2023-11-22 07:06:32,166 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.350e+01 8.889e+01 9.652e+01 1.254e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 07:06:41,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1843766.6666666667, ans=0.0
2023-11-22 07:06:43,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1843766.6666666667, ans=0.0
2023-11-22 07:07:02,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1843900.0, ans=0.0
2023-11-22 07:07:13,671 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 50, loss[loss=0.08876, simple_loss=0.1234, pruned_loss=0.01431, audio_tagging_loss=0.01274, over 15586.00 frames. ], tot_loss[loss=0.0814, simple_loss=0.09547, pruned_loss=0.01537, audio_tagging_loss=0.0183, over 684660.54 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:07:16,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1843966.6666666667, ans=0.07
2023-11-22 07:07:17,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276600
2023-11-22 07:07:41,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1844100.0, ans=0.125
2023-11-22 07:07:45,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1844100.0, ans=0.125
2023-11-22 07:07:57,889 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0
2023-11-22 07:08:04,689 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.90 vs. limit=15.0
2023-11-22 07:08:07,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1844233.3333333333, ans=0.125
2023-11-22 07:08:09,689 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.50 vs. limit=5.0
2023-11-22 07:08:16,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1844300.0, ans=0.2
2023-11-22 07:08:17,361 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 100, loss[loss=0.08666, simple_loss=0.1088, pruned_loss=0.01666, audio_tagging_loss=0.01562, over 15180.00 frames. ], tot_loss[loss=0.07963, simple_loss=0.09376, pruned_loss=0.01521, audio_tagging_loss=0.01753, over 1205504.36 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:08:21,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276650
2023-11-22 07:08:35,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1844366.6666666667, ans=0.0
2023-11-22 07:08:40,998 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.235e+01 8.891e+01 9.359e+01 9.974e+01 1.363e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-22 07:08:42,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1844433.3333333333, ans=10.0
2023-11-22 07:08:44,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1844433.3333333333, ans=0.125
2023-11-22 07:09:02,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1844500.0, ans=0.125
2023-11-22 07:09:21,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1844633.3333333333, ans=0.0
2023-11-22 07:09:22,247 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 150, loss[loss=0.09039, simple_loss=0.118, pruned_loss=0.02067, audio_tagging_loss=0.01072, over 15415.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09397, pruned_loss=0.01558, audio_tagging_loss=0.01572, over 1609589.57 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:09:26,042 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276700
2023-11-22 07:09:38,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1844700.0, ans=0.0
2023-11-22 07:09:47,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1844766.6666666667, ans=0.125
2023-11-22 07:09:53,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1844766.6666666667, ans=0.05
2023-11-22 07:10:01,241 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.44 vs. limit=15.0
2023-11-22 07:10:03,228 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.04 vs. limit=15.0
2023-11-22 07:10:16,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1844900.0, ans=0.125
2023-11-22 07:10:16,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1844900.0, ans=0.0
2023-11-22 07:10:20,007 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.65 vs. limit=8.0
2023-11-22 07:10:27,707 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 200, loss[loss=0.07212, simple_loss=0.09106, pruned_loss=0.01543, audio_tagging_loss=0.01116, over 15152.00 frames. ], tot_loss[loss=0.07674, simple_loss=0.09476, pruned_loss=0.0157, audio_tagging_loss=0.01366, over 1928505.33 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:10:30,973 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=15.0
2023-11-22 07:10:31,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276750
2023-11-22 07:10:32,906 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:10:51,663 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.378e+01 9.056e+01 9.976e+01 1.254e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 07:11:19,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1845233.3333333333, ans=0.125
2023-11-22 07:11:31,359 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 250, loss[loss=0.09131, simple_loss=0.1224, pruned_loss=0.02129, audio_tagging_loss=0.008842, over 15792.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09524, pruned_loss=0.01577, audio_tagging_loss=0.01229, over 2169294.87 frames. ], batch size: 60, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:11:31,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1845300.0, ans=0.125
2023-11-22 07:11:35,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276800
2023-11-22 07:11:42,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1845300.0, ans=0.1
2023-11-22 07:12:08,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1845433.3333333333, ans=0.125
2023-11-22 07:12:28,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1845566.6666666667, ans=0.1
2023-11-22 07:12:34,890 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.61 vs. limit=12.0
2023-11-22 07:12:36,693 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 300, loss[loss=0.07051, simple_loss=0.08626, pruned_loss=0.01678, audio_tagging_loss=0.0106, over 14741.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09457, pruned_loss=0.01557, audio_tagging_loss=0.0116, over 2362124.86 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:12:40,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276850
2023-11-22 07:12:43,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1845633.3333333333, ans=0.125
2023-11-22 07:12:46,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1845633.3333333333, ans=0.0
2023-11-22 07:12:48,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1845700.0, ans=0.125
2023-11-22 07:12:51,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1845700.0, ans=0.125
2023-11-22 07:12:58,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.08 vs. limit=15.0
2023-11-22 07:13:00,921 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.553e+01 8.200e+01 8.925e+01 9.618e+01 1.197e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 07:13:02,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1845766.6666666667, ans=0.0
2023-11-22 07:13:05,430 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.82 vs. limit=10.0
2023-11-22 07:13:06,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1845766.6666666667, ans=0.0
2023-11-22 07:13:08,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1845766.6666666667, ans=0.125
2023-11-22 07:13:09,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1845766.6666666667, ans=0.125
2023-11-22 07:13:39,902 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 350, loss[loss=0.07487, simple_loss=0.09775, pruned_loss=0.01838, audio_tagging_loss=0.007614, over 15314.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09481, pruned_loss=0.01555, audio_tagging_loss=0.01107, over 2517876.43 frames. ], batch size: 59, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:13:41,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1845966.6666666667, ans=0.0
2023-11-22 07:13:44,773 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276900
2023-11-22 07:13:51,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1845966.6666666667, ans=0.1
2023-11-22 07:13:57,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1846033.3333333333, ans=0.0
2023-11-22 07:14:04,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1846100.0, ans=0.125
2023-11-22 07:14:13,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1846100.0, ans=0.0
2023-11-22 07:14:28,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1846166.6666666667, ans=0.0
2023-11-22 07:14:31,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1846233.3333333333, ans=0.125
2023-11-22 07:14:44,725 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 400, loss[loss=0.06742, simple_loss=0.07883, pruned_loss=0.01603, audio_tagging_loss=0.01198, over 14043.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09496, pruned_loss=0.0154, audio_tagging_loss=0.01055, over 2634041.72 frames. ], batch size: 54, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:14:45,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1846300.0, ans=0.125
2023-11-22 07:14:48,639 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 276950
2023-11-22 07:14:55,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.37 vs. limit=15.0
2023-11-22 07:15:02,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1846366.6666666667, ans=0.0
2023-11-22 07:15:09,169 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.931e+01 8.582e+01 9.377e+01 1.207e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 07:15:21,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1846433.3333333333, ans=0.1
2023-11-22 07:15:49,037 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 450, loss[loss=0.08463, simple_loss=0.1145, pruned_loss=0.01961, audio_tagging_loss=0.007764, over 15573.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09409, pruned_loss=0.01524, audio_tagging_loss=0.01042, over 2725973.98 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:15:53,315 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277000
2023-11-22 07:15:53,523 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:16:00,100 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:16:05,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.21 vs. limit=15.0
2023-11-22 07:16:28,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1846833.3333333333, ans=0.0
2023-11-22 07:16:41,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1846900.0, ans=0.1
2023-11-22 07:16:53,439 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 500, loss[loss=0.07652, simple_loss=0.09611, pruned_loss=0.0183, audio_tagging_loss=0.01016, over 14969.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09448, pruned_loss=0.01539, audio_tagging_loss=0.01019, over 2796152.21 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:16:57,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277050
2023-11-22 07:17:00,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.35 vs. limit=15.0
2023-11-22 07:17:18,052 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.239e+01 8.246e+01 8.888e+01 9.960e+01 1.317e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 07:17:26,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1847100.0, ans=0.125
2023-11-22 07:17:29,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1847100.0, ans=0.125
2023-11-22 07:17:32,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1847166.6666666667, ans=0.125
2023-11-22 07:17:39,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1847166.6666666667, ans=0.1
2023-11-22 07:17:44,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1847233.3333333333, ans=0.2
2023-11-22 07:17:53,974 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.46 vs. limit=15.0
2023-11-22 07:17:58,806 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 550, loss[loss=0.05796, simple_loss=0.08065, pruned_loss=0.0102, audio_tagging_loss=0.007429, over 15293.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09404, pruned_loss=0.01533, audio_tagging_loss=0.01005, over 2857296.40 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:18:02,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277100
2023-11-22 07:18:26,458 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.84 vs. limit=15.0
2023-11-22 07:18:40,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1847500.0, ans=0.0
2023-11-22 07:18:43,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1847500.0, ans=0.125
2023-11-22 07:18:46,235 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0
2023-11-22 07:19:00,527 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.06 vs. limit=22.5
2023-11-22 07:19:02,828 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 600, loss[loss=0.0501, simple_loss=0.05429, pruned_loss=0.008518, audio_tagging_loss=0.01444, over 14170.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09391, pruned_loss=0.01528, audio_tagging_loss=0.009924, over 2897517.25 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:19:07,275 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277150
2023-11-22 07:19:12,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1847633.3333333333, ans=0.125
2023-11-22 07:19:27,309 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.641e+01 8.220e+01 8.869e+01 9.857e+01 1.255e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 07:19:33,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.27 vs. limit=22.5
2023-11-22 07:19:44,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.83 vs. limit=22.5
2023-11-22 07:19:54,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1847900.0, ans=0.125
2023-11-22 07:20:01,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.10 vs. limit=6.0
2023-11-22 07:20:07,409 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 650, loss[loss=0.05213, simple_loss=0.06774, pruned_loss=0.006291, audio_tagging_loss=0.01197, over 14615.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09281, pruned_loss=0.01507, audio_tagging_loss=0.009945, over 2939705.53 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:20:11,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277200
2023-11-22 07:20:58,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1848233.3333333333, ans=0.125
2023-11-22 07:21:11,146 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 700, loss[loss=0.06559, simple_loss=0.08692, pruned_loss=0.0125, audio_tagging_loss=0.00963, over 14945.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09383, pruned_loss=0.0152, audio_tagging_loss=0.009743, over 2964186.44 frames. ], batch size: 59, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:21:15,056 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277250
2023-11-22 07:21:15,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1848300.0, ans=0.0
2023-11-22 07:21:27,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1848366.6666666667, ans=0.0
2023-11-22 07:21:30,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.05 vs. limit=15.0
2023-11-22 07:21:37,767 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.366e+01 8.189e+01 8.629e+01 9.381e+01 1.193e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 07:21:38,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1848433.3333333333, ans=0.125
2023-11-22 07:21:54,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1848500.0, ans=0.2
2023-11-22 07:22:16,025 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 750, loss[loss=0.08325, simple_loss=0.111, pruned_loss=0.01775, audio_tagging_loss=0.01002, over 14468.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09491, pruned_loss=0.01554, audio_tagging_loss=0.009724, over 2979182.03 frames. ], batch size: 53, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:22:16,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1848633.3333333333, ans=0.1
2023-11-22 07:22:20,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277300
2023-11-22 07:22:25,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1848633.3333333333, ans=0.1
2023-11-22 07:22:29,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1848700.0, ans=0.125
2023-11-22 07:22:34,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1848700.0, ans=0.0
2023-11-22 07:22:40,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1848700.0, ans=0.125
2023-11-22 07:22:42,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1848766.6666666667, ans=0.0
2023-11-22 07:22:57,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1848833.3333333333, ans=0.125
2023-11-22 07:23:21,111 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 800, loss[loss=0.05663, simple_loss=0.07276, pruned_loss=0.009299, audio_tagging_loss=0.01095, over 15640.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09382, pruned_loss=0.01521, audio_tagging_loss=0.009739, over 2989082.94 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:23:24,850 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277350
2023-11-22 07:23:28,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1848966.6666666667, ans=0.125
2023-11-22 07:23:29,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1848966.6666666667, ans=0.0
2023-11-22 07:23:31,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1848966.6666666667, ans=0.1
2023-11-22 07:23:46,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=12.0
2023-11-22 07:23:46,995 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.163e+01 8.225e+01 8.846e+01 9.560e+01 1.170e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 07:23:50,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=1849100.0, ans=0.2
2023-11-22 07:23:54,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1849100.0, ans=0.125
2023-11-22 07:23:57,657 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.92 vs. limit=15.0
2023-11-22 07:24:01,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1849166.6666666667, ans=0.0
2023-11-22 07:24:08,801 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:24:13,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1849233.3333333333, ans=0.125
2023-11-22 07:24:20,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1849233.3333333333, ans=0.125
2023-11-22 07:24:24,996 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 850, loss[loss=0.08048, simple_loss=0.103, pruned_loss=0.01837, audio_tagging_loss=0.0106, over 15770.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09438, pruned_loss=0.01537, audio_tagging_loss=0.009737, over 3005292.97 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:24:28,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277400
2023-11-22 07:24:36,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.87 vs. limit=15.0
2023-11-22 07:24:37,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1849366.6666666667, ans=0.125
2023-11-22 07:25:01,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1849433.3333333333, ans=0.125
2023-11-22 07:25:15,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1849566.6666666667, ans=0.125
2023-11-22 07:25:23,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1849566.6666666667, ans=0.125
2023-11-22 07:25:24,836 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=15.0
2023-11-22 07:25:28,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.05 vs. limit=22.5
2023-11-22 07:25:28,878 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 900, loss[loss=0.08618, simple_loss=0.1224, pruned_loss=0.01689, audio_tagging_loss=0.008117, over 15134.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09408, pruned_loss=0.01541, audio_tagging_loss=0.009736, over 3017086.21 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:25:33,299 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277450
2023-11-22 07:25:56,242 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.215e+01 8.871e+01 9.558e+01 1.650e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 07:26:00,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1849766.6666666667, ans=0.2
2023-11-22 07:26:01,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1849766.6666666667, ans=0.1
2023-11-22 07:26:20,196 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.03 vs. limit=15.0
2023-11-22 07:26:26,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1849900.0, ans=0.0
2023-11-22 07:26:28,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1849900.0, ans=0.1
2023-11-22 07:26:28,511 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:26:33,273 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 950, loss[loss=0.07395, simple_loss=0.103, pruned_loss=0.01479, audio_tagging_loss=0.007641, over 16343.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09473, pruned_loss=0.01561, audio_tagging_loss=0.009628, over 3031159.86 frames. ], batch size: 61, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:26:37,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277500
2023-11-22 07:26:47,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1850033.3333333333, ans=0.1
2023-11-22 07:26:48,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1850033.3333333333, ans=0.125
2023-11-22 07:27:36,853 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1000, loss[loss=0.07139, simple_loss=0.09916, pruned_loss=0.01487, audio_tagging_loss=0.006931, over 15222.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09487, pruned_loss=0.01556, audio_tagging_loss=0.009491, over 3032285.15 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:27:38,807 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.27 vs. limit=15.0
2023-11-22 07:27:40,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277550
2023-11-22 07:27:58,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1850366.6666666667, ans=0.95
2023-11-22 07:28:04,353 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.008e+01 8.714e+01 9.298e+01 1.223e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 07:28:04,463 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:28:08,921 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.95 vs. limit=10.0
2023-11-22 07:28:16,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1850500.0, ans=0.125
2023-11-22 07:28:24,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1850500.0, ans=0.2
2023-11-22 07:28:33,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1850566.6666666667, ans=0.125
2023-11-22 07:28:39,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=1850633.3333333333, ans=0.02
2023-11-22 07:28:40,599 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1050, loss[loss=0.07189, simple_loss=0.105, pruned_loss=0.01167, audio_tagging_loss=0.007703, over 14129.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09382, pruned_loss=0.01536, audio_tagging_loss=0.009489, over 3030547.21 frames. ], batch size: 54, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:28:44,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277600
2023-11-22 07:29:17,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1850766.6666666667, ans=0.2
2023-11-22 07:29:21,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1850833.3333333333, ans=0.0
2023-11-22 07:29:46,036 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1100, loss[loss=0.0453, simple_loss=0.05321, pruned_loss=0.009325, audio_tagging_loss=0.00937, over 14610.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09328, pruned_loss=0.01521, audio_tagging_loss=0.009325, over 3037405.89 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:29:49,653 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:29:49,738 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277650
2023-11-22 07:29:51,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1850966.6666666667, ans=0.0
2023-11-22 07:29:58,016 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.08 vs. limit=15.0
2023-11-22 07:30:04,251 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.42 vs. limit=15.0
2023-11-22 07:30:11,992 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.181e+01 8.784e+01 9.508e+01 1.137e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 07:30:16,620 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.37 vs. limit=15.0
2023-11-22 07:30:17,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1851100.0, ans=0.125
2023-11-22 07:30:39,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1851233.3333333333, ans=0.0
2023-11-22 07:30:50,559 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1150, loss[loss=0.09626, simple_loss=0.1273, pruned_loss=0.02344, audio_tagging_loss=0.009175, over 15313.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09348, pruned_loss=0.01523, audio_tagging_loss=0.009319, over 3034829.27 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:30:54,375 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277700
2023-11-22 07:30:55,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1851300.0, ans=0.125
2023-11-22 07:30:55,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1851300.0, ans=0.125
2023-11-22 07:30:59,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.68 vs. limit=15.0
2023-11-22 07:31:33,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1851500.0, ans=0.125
2023-11-22 07:31:33,516 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:31:45,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1851566.6666666667, ans=0.0
2023-11-22 07:31:53,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1851633.3333333333, ans=0.0
2023-11-22 07:31:55,003 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1200, loss[loss=0.0578, simple_loss=0.06841, pruned_loss=0.01253, audio_tagging_loss=0.01106, over 14558.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09371, pruned_loss=0.01529, audio_tagging_loss=0.00926, over 3035367.52 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:31:58,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277750
2023-11-22 07:32:02,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1851633.3333333333, ans=0.125
2023-11-22 07:32:23,094 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.243e+01 8.919e+01 9.746e+01 1.203e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 07:33:00,926 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1250, loss[loss=0.07411, simple_loss=0.1043, pruned_loss=0.01181, audio_tagging_loss=0.01015, over 15883.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09357, pruned_loss=0.01522, audio_tagging_loss=0.009232, over 3036036.67 frames. ], batch size: 59, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:33:04,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277800
2023-11-22 07:33:20,601 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.83 vs. limit=6.0
2023-11-22 07:34:06,750 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1300, loss[loss=0.104, simple_loss=0.132, pruned_loss=0.0278, audio_tagging_loss=0.01017, over 14683.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09316, pruned_loss=0.01526, audio_tagging_loss=0.009276, over 3037782.37 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:34:10,639 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277850
2023-11-22 07:34:12,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1852300.0, ans=0.125
2023-11-22 07:34:27,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1852366.6666666667, ans=0.09899494936611666
2023-11-22 07:34:33,403 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.077e+01 8.549e+01 9.237e+01 1.175e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 07:34:53,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1852500.0, ans=0.1
2023-11-22 07:34:53,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1852500.0, ans=0.1
2023-11-22 07:35:04,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1852566.6666666667, ans=0.0
2023-11-22 07:35:11,084 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1350, loss[loss=0.0916, simple_loss=0.1151, pruned_loss=0.02493, audio_tagging_loss=0.009108, over 15339.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09365, pruned_loss=0.0153, audio_tagging_loss=0.009187, over 3037193.66 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:35:14,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277900
2023-11-22 07:35:15,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1852633.3333333333, ans=0.125
2023-11-22 07:35:22,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1852700.0, ans=0.125
2023-11-22 07:35:40,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1852766.6666666667, ans=0.2
2023-11-22 07:35:42,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1852766.6666666667, ans=0.035
2023-11-22 07:35:47,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1852766.6666666667, ans=0.0
2023-11-22 07:35:57,571 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:35:59,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1852833.3333333333, ans=0.0
2023-11-22 07:36:16,060 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1400, loss[loss=0.0609, simple_loss=0.07857, pruned_loss=0.01079, audio_tagging_loss=0.01082, over 15696.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09379, pruned_loss=0.01522, audio_tagging_loss=0.009255, over 3040365.29 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:36:19,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 277950
2023-11-22 07:36:21,329 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:36:27,526 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:36:33,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1853033.3333333333, ans=0.0
2023-11-22 07:36:33,760 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.11 vs. limit=15.0
2023-11-22 07:36:42,781 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.691e+01 8.253e+01 8.948e+01 9.652e+01 1.188e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 07:36:45,971 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.53 vs. limit=15.0
2023-11-22 07:37:00,147 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.73 vs. limit=15.0
2023-11-22 07:37:04,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1853166.6666666667, ans=0.0
2023-11-22 07:37:20,255 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1450, loss[loss=0.08512, simple_loss=0.1031, pruned_loss=0.02357, audio_tagging_loss=0.009999, over 14097.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09385, pruned_loss=0.01525, audio_tagging_loss=0.009333, over 3028630.45 frames. ], batch size: 53, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:37:20,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1853300.0, ans=0.1
2023-11-22 07:37:24,560 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278000
2023-11-22 07:37:29,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1853300.0, ans=0.1
2023-11-22 07:37:39,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1853366.6666666667, ans=0.125
2023-11-22 07:38:09,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1853500.0, ans=0.125
2023-11-22 07:38:24,846 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1500, loss[loss=0.09403, simple_loss=0.1147, pruned_loss=0.02603, audio_tagging_loss=0.01068, over 15050.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09473, pruned_loss=0.01533, audio_tagging_loss=0.00937, over 3033242.97 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:38:28,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278050
2023-11-22 07:38:33,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1853633.3333333333, ans=0.125
2023-11-22 07:38:33,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1853633.3333333333, ans=0.0
2023-11-22 07:38:43,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1853700.0, ans=0.0
2023-11-22 07:38:52,912 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.167e+01 8.667e+01 9.503e+01 1.230e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 07:38:53,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1853766.6666666667, ans=0.0
2023-11-22 07:39:02,067 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.79 vs. limit=6.0
2023-11-22 07:39:04,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1853833.3333333333, ans=0.125
2023-11-22 07:39:12,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1853833.3333333333, ans=0.125
2023-11-22 07:39:29,696 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1550, loss[loss=0.06801, simple_loss=0.0843, pruned_loss=0.01603, audio_tagging_loss=0.00983, over 14796.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09449, pruned_loss=0.01524, audio_tagging_loss=0.009526, over 3028192.99 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:39:34,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278100
2023-11-22 07:39:37,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1853966.6666666667, ans=0.125
2023-11-22 07:39:40,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.10 vs. limit=22.5
2023-11-22 07:40:29,716 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:40:34,402 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1600, loss[loss=0.07468, simple_loss=0.1026, pruned_loss=0.01429, audio_tagging_loss=0.009078, over 15848.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09424, pruned_loss=0.01527, audio_tagging_loss=0.009589, over 3035481.11 frames. ], batch size: 60, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:40:38,774 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278150
2023-11-22 07:41:04,080 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.136e+01 8.791e+01 9.603e+01 1.214e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 07:41:15,463 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.83 vs. limit=15.0
2023-11-22 07:41:39,055 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1650, loss[loss=0.09148, simple_loss=0.1185, pruned_loss=0.02404, audio_tagging_loss=0.008193, over 15234.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09465, pruned_loss=0.01536, audio_tagging_loss=0.009674, over 3042892.26 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:41:42,736 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278200
2023-11-22 07:42:02,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1854700.0, ans=0.1
2023-11-22 07:42:03,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1854766.6666666667, ans=0.125
2023-11-22 07:42:41,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.96 vs. limit=22.5
2023-11-22 07:42:43,474 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1700, loss[loss=0.08335, simple_loss=0.117, pruned_loss=0.0196, audio_tagging_loss=0.005262, over 15545.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.09337, pruned_loss=0.01521, audio_tagging_loss=0.009781, over 3044272.18 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:42:44,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1854966.6666666667, ans=0.1
2023-11-22 07:42:47,854 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278250
2023-11-22 07:43:13,669 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.170e+01 8.792e+01 9.443e+01 1.155e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 07:43:17,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1855100.0, ans=0.125
2023-11-22 07:43:23,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1855166.6666666667, ans=0.0
2023-11-22 07:43:36,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1855233.3333333333, ans=0.1
2023-11-22 07:43:45,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1855233.3333333333, ans=0.125
2023-11-22 07:43:48,088 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1750, loss[loss=0.0595, simple_loss=0.08067, pruned_loss=0.01043, audio_tagging_loss=0.008739, over 15590.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09485, pruned_loss=0.0154, audio_tagging_loss=0.009616, over 3046564.98 frames. ], batch size: 58, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:43:51,878 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278300
2023-11-22 07:43:54,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1855300.0, ans=0.125
2023-11-22 07:43:55,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1855300.0, ans=0.2
2023-11-22 07:44:04,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.60 vs. limit=15.0
2023-11-22 07:44:10,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1855366.6666666667, ans=0.125
2023-11-22 07:44:30,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1855500.0, ans=0.125
2023-11-22 07:44:50,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1855566.6666666667, ans=0.0
2023-11-22 07:44:52,591 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1800, loss[loss=0.06049, simple_loss=0.08239, pruned_loss=0.00972, audio_tagging_loss=0.009581, over 15481.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09542, pruned_loss=0.01536, audio_tagging_loss=0.009464, over 3051494.29 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:44:53,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.19 vs. limit=15.0
2023-11-22 07:44:56,935 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278350
2023-11-22 07:45:03,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1855633.3333333333, ans=0.0
2023-11-22 07:45:12,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1855700.0, ans=0.125
2023-11-22 07:45:22,549 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 7.991e+01 8.555e+01 9.443e+01 1.169e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-22 07:45:22,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1855766.6666666667, ans=0.0
2023-11-22 07:45:32,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1855833.3333333333, ans=0.2
2023-11-22 07:45:57,062 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1850, loss[loss=0.0777, simple_loss=0.1068, pruned_loss=0.01619, audio_tagging_loss=0.00814, over 14917.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09487, pruned_loss=0.01537, audio_tagging_loss=0.009499, over 3048925.50 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:46:01,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278400
2023-11-22 07:46:03,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.96 vs. limit=15.0
2023-11-22 07:46:14,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1856033.3333333333, ans=0.2
2023-11-22 07:46:20,047 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.75 vs. limit=22.5
2023-11-22 07:46:21,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1856033.3333333333, ans=0.0
2023-11-22 07:46:40,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1856166.6666666667, ans=0.0
2023-11-22 07:46:41,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1856166.6666666667, ans=0.125
2023-11-22 07:46:48,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1856233.3333333333, ans=0.125
2023-11-22 07:47:02,776 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1900, loss[loss=0.07874, simple_loss=0.11, pruned_loss=0.0149, audio_tagging_loss=0.008843, over 15819.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09486, pruned_loss=0.01528, audio_tagging_loss=0.009336, over 3056225.90 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:47:06,538 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278450
2023-11-22 07:47:29,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.54 vs. limit=15.0
2023-11-22 07:47:32,029 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 7.925e+01 8.678e+01 9.471e+01 1.078e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-22 07:47:45,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1856500.0, ans=0.125
2023-11-22 07:47:47,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1856500.0, ans=0.125
2023-11-22 07:47:56,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0
2023-11-22 07:47:57,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1856566.6666666667, ans=0.125
2023-11-22 07:48:00,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1856566.6666666667, ans=0.1
2023-11-22 07:48:07,437 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 1950, loss[loss=0.06484, simple_loss=0.08058, pruned_loss=0.01519, audio_tagging_loss=0.009355, over 15239.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09371, pruned_loss=0.01508, audio_tagging_loss=0.009321, over 3048969.39 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:48:11,218 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278500
2023-11-22 07:48:13,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1856633.3333333333, ans=0.125
2023-11-22 07:48:34,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1856766.6666666667, ans=0.2
2023-11-22 07:48:37,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=15.0
2023-11-22 07:48:52,003 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.60 vs. limit=15.0
2023-11-22 07:49:12,021 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2000, loss[loss=0.0798, simple_loss=0.1094, pruned_loss=0.01719, audio_tagging_loss=0.007928, over 14795.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09405, pruned_loss=0.01525, audio_tagging_loss=0.009341, over 3048078.50 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:49:16,445 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278550
2023-11-22 07:49:19,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1856966.6666666667, ans=0.1
2023-11-22 07:49:22,620 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0
2023-11-22 07:49:28,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1857033.3333333333, ans=0.2
2023-11-22 07:49:41,988 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.370e+01 9.075e+01 9.910e+01 1.919e+02, threshold=1.815e+02, percent-clipped=1.0
2023-11-22 07:49:52,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1857166.6666666667, ans=0.125
2023-11-22 07:49:56,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1857166.6666666667, ans=0.125
2023-11-22 07:49:59,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1857166.6666666667, ans=0.125
2023-11-22 07:50:04,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1857233.3333333333, ans=22.5
2023-11-22 07:50:07,910 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.34 vs. limit=12.0
2023-11-22 07:50:14,489 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.92 vs. limit=15.0
2023-11-22 07:50:14,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.40 vs. limit=22.5
2023-11-22 07:50:17,584 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2050, loss[loss=0.0962, simple_loss=0.1356, pruned_loss=0.0221, audio_tagging_loss=0.00632, over 15608.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09561, pruned_loss=0.01552, audio_tagging_loss=0.009222, over 3052135.59 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:50:21,363 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278600
2023-11-22 07:50:30,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1857366.6666666667, ans=0.0
2023-11-22 07:50:46,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1857433.3333333333, ans=0.0
2023-11-22 07:51:16,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1857566.6666666667, ans=0.125
2023-11-22 07:51:17,061 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.52 vs. limit=15.0
2023-11-22 07:51:23,215 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2100, loss[loss=0.075, simple_loss=0.1001, pruned_loss=0.01404, audio_tagging_loss=0.01092, over 15093.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.0949, pruned_loss=0.01526, audio_tagging_loss=0.009159, over 3045876.76 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:51:26,962 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278650
2023-11-22 07:51:27,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1857633.3333333333, ans=0.125
2023-11-22 07:51:44,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1857700.0, ans=0.125
2023-11-22 07:51:52,695 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.224e+01 8.868e+01 9.706e+01 1.360e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 07:51:56,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1857766.6666666667, ans=0.09899494936611666
2023-11-22 07:51:57,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1857766.6666666667, ans=0.2
2023-11-22 07:52:02,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.32 vs. limit=15.0
2023-11-22 07:52:05,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.99 vs. limit=22.5
2023-11-22 07:52:23,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1857900.0, ans=0.125
2023-11-22 07:52:26,019 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2150, loss[loss=0.07545, simple_loss=0.09964, pruned_loss=0.01805, audio_tagging_loss=0.007582, over 15251.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09481, pruned_loss=0.01535, audio_tagging_loss=0.00917, over 3040700.10 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:52:29,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278700
2023-11-22 07:52:49,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1858033.3333333333, ans=0.125
2023-11-22 07:52:52,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1858100.0, ans=0.1
2023-11-22 07:52:52,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1858100.0, ans=0.125
2023-11-22 07:52:55,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1858100.0, ans=0.1
2023-11-22 07:53:01,224 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.06 vs. limit=6.0
2023-11-22 07:53:06,683 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:53:31,932 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2200, loss[loss=0.05733, simple_loss=0.06535, pruned_loss=0.01369, audio_tagging_loss=0.01097, over 14510.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09465, pruned_loss=0.01548, audio_tagging_loss=0.009231, over 3037477.04 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:53:35,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278750
2023-11-22 07:54:00,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.169e+01 8.248e+01 8.861e+01 9.671e+01 1.420e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 07:54:15,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.11 vs. limit=10.0
2023-11-22 07:54:19,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1858500.0, ans=0.1
2023-11-22 07:54:36,104 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2250, loss[loss=0.0928, simple_loss=0.1241, pruned_loss=0.02354, audio_tagging_loss=0.007213, over 14367.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09513, pruned_loss=0.0155, audio_tagging_loss=0.009377, over 3040540.41 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:54:37,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1858633.3333333333, ans=0.2
2023-11-22 07:54:39,804 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278800
2023-11-22 07:54:43,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1858633.3333333333, ans=0.125
2023-11-22 07:55:07,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1858766.6666666667, ans=0.125
2023-11-22 07:55:19,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1858833.3333333333, ans=0.2
2023-11-22 07:55:23,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1858833.3333333333, ans=0.1
2023-11-22 07:55:30,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1858900.0, ans=0.125
2023-11-22 07:55:39,292 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.03 vs. limit=12.0
2023-11-22 07:55:39,798 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2300, loss[loss=0.07225, simple_loss=0.08835, pruned_loss=0.01672, audio_tagging_loss=0.01136, over 15748.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09492, pruned_loss=0.01545, audio_tagging_loss=0.009448, over 3045695.03 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:55:43,501 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278850
2023-11-22 07:56:00,071 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:56:01,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1859033.3333333333, ans=0.1
2023-11-22 07:56:09,718 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.100e+01 8.737e+01 9.435e+01 1.101e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 07:56:14,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1859100.0, ans=0.0
2023-11-22 07:56:16,088 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.33 vs. limit=22.5
2023-11-22 07:56:36,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1859233.3333333333, ans=0.125
2023-11-22 07:56:37,688 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:56:40,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1859233.3333333333, ans=0.0
2023-11-22 07:56:45,054 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2350, loss[loss=0.07362, simple_loss=0.096, pruned_loss=0.01678, audio_tagging_loss=0.008851, over 14483.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09452, pruned_loss=0.01545, audio_tagging_loss=0.009524, over 3045265.34 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:56:49,567 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278900
2023-11-22 07:57:09,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1859433.3333333333, ans=0.0
2023-11-22 07:57:15,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1859433.3333333333, ans=0.2
2023-11-22 07:57:33,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1859500.0, ans=0.125
2023-11-22 07:57:34,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1859500.0, ans=0.125
2023-11-22 07:57:44,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1859566.6666666667, ans=0.1
2023-11-22 07:57:49,788 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2400, loss[loss=0.06289, simple_loss=0.08516, pruned_loss=0.01069, audio_tagging_loss=0.009615, over 14475.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.0944, pruned_loss=0.01544, audio_tagging_loss=0.009515, over 3040325.07 frames. ], batch size: 53, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:57:53,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 278950
2023-11-22 07:58:00,260 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0
2023-11-22 07:58:18,926 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:58:19,861 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.551e+01 8.282e+01 8.837e+01 9.594e+01 1.331e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 07:58:23,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1859766.6666666667, ans=0.0
2023-11-22 07:58:32,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1859833.3333333333, ans=0.1
2023-11-22 07:58:41,023 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.49 vs. limit=15.0
2023-11-22 07:58:51,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.48 vs. limit=15.0
2023-11-22 07:58:53,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.55 vs. limit=15.0
2023-11-22 07:58:53,797 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2450, loss[loss=0.08439, simple_loss=0.1124, pruned_loss=0.02011, audio_tagging_loss=0.008086, over 15910.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09499, pruned_loss=0.01555, audio_tagging_loss=0.0095, over 3049551.84 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:58:57,582 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279000
2023-11-22 07:59:07,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1860033.3333333333, ans=0.125
2023-11-22 07:59:15,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1860033.3333333333, ans=0.0
2023-11-22 07:59:20,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1860100.0, ans=0.035
2023-11-22 07:59:24,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1860100.0, ans=0.125
2023-11-22 07:59:27,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1860100.0, ans=0.0
2023-11-22 07:59:28,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1860100.0, ans=0.125
2023-11-22 07:59:47,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.04 vs. limit=15.0
2023-11-22 07:59:57,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1860300.0, ans=0.125
2023-11-22 07:59:58,131 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2500, loss[loss=0.08968, simple_loss=0.1137, pruned_loss=0.02291, audio_tagging_loss=0.009905, over 15885.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09536, pruned_loss=0.01575, audio_tagging_loss=0.009604, over 3051230.09 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:59:58,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1860300.0, ans=0.07
2023-11-22 08:00:02,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279050
2023-11-22 08:00:11,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1860366.6666666667, ans=0.0
2023-11-22 08:00:29,442 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.264e+01 8.898e+01 9.687e+01 1.183e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 08:00:45,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1860500.0, ans=0.0
2023-11-22 08:00:53,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1860566.6666666667, ans=0.125
2023-11-22 08:00:55,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1860566.6666666667, ans=0.1
2023-11-22 08:01:00,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1860566.6666666667, ans=0.125
2023-11-22 08:01:03,353 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2550, loss[loss=0.08089, simple_loss=0.1172, pruned_loss=0.01537, audio_tagging_loss=0.006899, over 16683.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09414, pruned_loss=0.01555, audio_tagging_loss=0.009465, over 3051284.33 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:01:07,202 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279100
2023-11-22 08:01:23,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1860700.0, ans=0.125
2023-11-22 08:01:59,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1860900.0, ans=0.2
2023-11-22 08:02:09,028 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2600, loss[loss=0.06158, simple_loss=0.07424, pruned_loss=0.01376, audio_tagging_loss=0.0107, over 14672.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.0947, pruned_loss=0.01546, audio_tagging_loss=0.009297, over 3059174.62 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:02:09,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1860966.6666666667, ans=0.0
2023-11-22 08:02:10,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1860966.6666666667, ans=0.1
2023-11-22 08:02:12,948 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279150
2023-11-22 08:02:23,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1861033.3333333333, ans=0.2
2023-11-22 08:02:39,905 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.406e+01 8.247e+01 8.743e+01 9.488e+01 1.623e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 08:02:42,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1861100.0, ans=0.0
2023-11-22 08:03:11,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1861300.0, ans=0.0
2023-11-22 08:03:12,770 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2650, loss[loss=0.06872, simple_loss=0.09724, pruned_loss=0.01434, audio_tagging_loss=0.005756, over 14943.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09485, pruned_loss=0.01547, audio_tagging_loss=0.009228, over 3048669.48 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:03:16,474 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.63 vs. limit=6.0
2023-11-22 08:03:17,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279200
2023-11-22 08:03:31,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1861366.6666666667, ans=0.0
2023-11-22 08:03:37,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1861366.6666666667, ans=0.125
2023-11-22 08:03:39,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1861433.3333333333, ans=0.0
2023-11-22 08:03:43,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1861433.3333333333, ans=0.0
2023-11-22 08:04:01,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1861500.0, ans=0.125
2023-11-22 08:04:01,635 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.43 vs. limit=15.0
2023-11-22 08:04:07,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.64 vs. limit=10.0
2023-11-22 08:04:18,509 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2700, loss[loss=0.07655, simple_loss=0.08854, pruned_loss=0.01884, audio_tagging_loss=0.01345, over 14542.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09401, pruned_loss=0.01536, audio_tagging_loss=0.009226, over 3046329.00 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 08:04:22,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279250
2023-11-22 08:04:50,484 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 7.998e+01 8.544e+01 9.286e+01 1.150e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-22 08:05:01,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0
2023-11-22 08:05:10,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.42 vs. limit=15.0
2023-11-22 08:05:23,842 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2750, loss[loss=0.07541, simple_loss=0.09813, pruned_loss=0.01689, audio_tagging_loss=0.009452, over 14827.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09389, pruned_loss=0.01523, audio_tagging_loss=0.009253, over 3051289.52 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 08:05:27,557 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279300
2023-11-22 08:05:51,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1862100.0, ans=0.125
2023-11-22 08:06:17,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1862233.3333333333, ans=0.0
2023-11-22 08:06:20,792 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:06:28,056 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2800, loss[loss=0.06429, simple_loss=0.07452, pruned_loss=0.01268, audio_tagging_loss=0.01435, over 14605.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09333, pruned_loss=0.01529, audio_tagging_loss=0.009286, over 3040665.23 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:06:28,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1862300.0, ans=0.125
2023-11-22 08:06:32,368 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279350
2023-11-22 08:06:35,632 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.54 vs. limit=15.0
2023-11-22 08:06:55,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1862433.3333333333, ans=0.2
2023-11-22 08:07:01,169 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 7.937e+01 8.597e+01 9.316e+01 1.176e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-22 08:07:06,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1862500.0, ans=0.125
2023-11-22 08:07:13,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1862500.0, ans=0.125
2023-11-22 08:07:33,296 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2850, loss[loss=0.08488, simple_loss=0.1141, pruned_loss=0.02072, audio_tagging_loss=0.007134, over 15448.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09356, pruned_loss=0.01533, audio_tagging_loss=0.009201, over 3041972.21 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:07:37,074 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279400
2023-11-22 08:07:37,542 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.58 vs. limit=10.0
2023-11-22 08:07:46,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1862700.0, ans=0.0
2023-11-22 08:07:50,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1862700.0, ans=0.0
2023-11-22 08:08:23,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1862833.3333333333, ans=0.125
2023-11-22 08:08:37,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1862966.6666666667, ans=0.125
2023-11-22 08:08:38,451 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2900, loss[loss=0.08265, simple_loss=0.1097, pruned_loss=0.02241, audio_tagging_loss=0.005377, over 16431.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09349, pruned_loss=0.01521, audio_tagging_loss=0.009191, over 3042354.16 frames. ], batch size: 61, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:08:40,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1862966.6666666667, ans=0.125
2023-11-22 08:08:42,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279450
2023-11-22 08:08:52,448 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.99 vs. limit=6.0
2023-11-22 08:08:53,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1863033.3333333333, ans=0.1
2023-11-22 08:09:05,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1863100.0, ans=0.125
2023-11-22 08:09:11,276 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.213e+01 8.733e+01 9.343e+01 1.106e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 08:09:17,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.17 vs. limit=15.0
2023-11-22 08:09:20,797 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0
2023-11-22 08:09:25,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1863166.6666666667, ans=0.025
2023-11-22 08:09:43,411 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 2950, loss[loss=0.05926, simple_loss=0.07652, pruned_loss=0.01066, audio_tagging_loss=0.01033, over 15714.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09362, pruned_loss=0.01513, audio_tagging_loss=0.009239, over 3044579.02 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:09:45,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=1863300.0, ans=0.1
2023-11-22 08:09:47,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279500
2023-11-22 08:10:07,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1863366.6666666667, ans=0.1
2023-11-22 08:10:35,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5
2023-11-22 08:10:40,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1863566.6666666667, ans=0.125
2023-11-22 08:10:47,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1863566.6666666667, ans=0.2
2023-11-22 08:10:49,484 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3000, loss[loss=0.0824, simple_loss=0.1163, pruned_loss=0.01894, audio_tagging_loss=0.005302, over 15650.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09453, pruned_loss=0.01522, audio_tagging_loss=0.009264, over 3045275.22 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:10:49,485 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 08:11:14,325 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4303, 3.7651, 2.6401, 3.6276], device='cuda:2')
2023-11-22 08:11:20,249 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.0930, 2.9695, 3.3171, 2.9903, 3.7676, 3.7409, 3.2982, 3.2190],
       device='cuda:2')
2023-11-22 08:11:21,586 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.1641, 2.3564, 5.0303, 2.8759], device='cuda:2')
2023-11-22 08:11:29,461 INFO [train_asr.py:1253] (2/4) Epoch 24, validation: loss=0.0588, simple_loss=0.05168, pruned_loss=0.005124, audio_tagging_loss=0.02784, over 4681554.00 frames. 
2023-11-22 08:11:29,462 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 08:11:33,222 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279550
2023-11-22 08:11:41,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1863700.0, ans=0.1
2023-11-22 08:12:02,512 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.135e+01 8.700e+01 9.423e+01 1.185e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 08:12:34,538 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3050, loss[loss=0.05726, simple_loss=0.06609, pruned_loss=0.01246, audio_tagging_loss=0.01175, over 14428.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09475, pruned_loss=0.01542, audio_tagging_loss=0.009309, over 3044363.02 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:12:36,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1863966.6666666667, ans=0.125
2023-11-22 08:12:38,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279600
2023-11-22 08:12:51,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.31 vs. limit=15.0
2023-11-22 08:13:14,126 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:13:22,635 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.87 vs. limit=12.0
2023-11-22 08:13:41,249 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3100, loss[loss=0.08751, simple_loss=0.1226, pruned_loss=0.01753, audio_tagging_loss=0.008686, over 15991.00 frames. ], tot_loss[loss=0.07319, simple_loss=0.09635, pruned_loss=0.01558, audio_tagging_loss=0.009427, over 3048085.19 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:13:45,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279650
2023-11-22 08:14:13,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.146e+01 8.664e+01 9.339e+01 1.098e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 08:14:29,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1864500.0, ans=0.125
2023-11-22 08:14:32,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1864566.6666666667, ans=0.2
2023-11-22 08:14:38,064 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.32 vs. limit=15.0
2023-11-22 08:14:46,731 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3150, loss[loss=0.05501, simple_loss=0.06318, pruned_loss=0.01027, audio_tagging_loss=0.01315, over 14627.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09578, pruned_loss=0.01554, audio_tagging_loss=0.00951, over 3046105.64 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:14:47,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1864633.3333333333, ans=0.0
2023-11-22 08:14:50,548 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279700
2023-11-22 08:14:54,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.71 vs. limit=22.5
2023-11-22 08:15:14,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1864766.6666666667, ans=0.1
2023-11-22 08:15:14,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1864766.6666666667, ans=0.0
2023-11-22 08:15:19,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.71 vs. limit=15.0
2023-11-22 08:15:24,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1864833.3333333333, ans=0.125
2023-11-22 08:15:32,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1864833.3333333333, ans=0.125
2023-11-22 08:15:47,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1864900.0, ans=0.125
2023-11-22 08:15:50,948 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3200, loss[loss=0.0581, simple_loss=0.06478, pruned_loss=0.01023, audio_tagging_loss=0.01548, over 15439.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09488, pruned_loss=0.01544, audio_tagging_loss=0.009724, over 3049568.23 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:15:55,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279750
2023-11-22 08:16:02,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.69 vs. limit=15.0
2023-11-22 08:16:17,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1865100.0, ans=0.0
2023-11-22 08:16:21,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1865100.0, ans=0.1
2023-11-22 08:16:24,181 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.350e+01 8.310e+01 8.993e+01 9.711e+01 1.257e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 08:16:27,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1865100.0, ans=0.125
2023-11-22 08:16:32,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.05 vs. limit=10.0
2023-11-22 08:16:55,352 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.92 vs. limit=15.0
2023-11-22 08:16:57,013 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3250, loss[loss=0.056, simple_loss=0.06842, pruned_loss=0.01147, audio_tagging_loss=0.01032, over 15415.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09479, pruned_loss=0.0155, audio_tagging_loss=0.009677, over 3050590.98 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:17:00,802 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279800
2023-11-22 08:17:01,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.88 vs. limit=15.0
2023-11-22 08:17:17,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1865366.6666666667, ans=0.0
2023-11-22 08:17:46,703 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.19 vs. limit=22.5
2023-11-22 08:18:02,039 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3300, loss[loss=0.08395, simple_loss=0.1144, pruned_loss=0.01686, audio_tagging_loss=0.009875, over 14914.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09452, pruned_loss=0.01544, audio_tagging_loss=0.009716, over 3055121.38 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:18:06,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279850
2023-11-22 08:18:10,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1865633.3333333333, ans=0.125
2023-11-22 08:18:35,003 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.287e+01 8.648e+01 9.394e+01 1.159e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 08:18:48,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1865833.3333333333, ans=0.0
2023-11-22 08:18:50,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.72 vs. limit=15.0
2023-11-22 08:18:51,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1865833.3333333333, ans=0.125
2023-11-22 08:18:54,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1865900.0, ans=0.125
2023-11-22 08:18:58,549 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.09 vs. limit=15.0
2023-11-22 08:18:59,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1865900.0, ans=0.125
2023-11-22 08:19:00,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1865900.0, ans=0.0
2023-11-22 08:19:06,602 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3350, loss[loss=0.08713, simple_loss=0.1162, pruned_loss=0.01956, audio_tagging_loss=0.00945, over 15992.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09517, pruned_loss=0.01564, audio_tagging_loss=0.009564, over 3064474.55 frames. ], batch size: 58, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:19:10,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279900
2023-11-22 08:19:21,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1866033.3333333333, ans=0.2
2023-11-22 08:19:32,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.91 vs. limit=10.0
2023-11-22 08:19:36,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1866100.0, ans=0.125
2023-11-22 08:19:41,963 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.21 vs. limit=15.0
2023-11-22 08:19:50,704 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.96 vs. limit=15.0
2023-11-22 08:19:58,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1866233.3333333333, ans=0.0
2023-11-22 08:20:06,345 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:20:11,684 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3400, loss[loss=0.09193, simple_loss=0.1176, pruned_loss=0.02552, audio_tagging_loss=0.007615, over 15421.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09514, pruned_loss=0.01557, audio_tagging_loss=0.009484, over 3062754.86 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:20:15,451 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 279950
2023-11-22 08:20:37,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1866433.3333333333, ans=0.1
2023-11-22 08:20:38,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1866433.3333333333, ans=0.0
2023-11-22 08:20:42,973 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.164e+01 8.350e+01 8.855e+01 9.450e+01 1.235e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 08:20:44,555 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:21:12,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1866566.6666666667, ans=0.125
2023-11-22 08:21:14,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1866633.3333333333, ans=0.0
2023-11-22 08:21:15,133 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3450, loss[loss=0.06381, simple_loss=0.08781, pruned_loss=0.01116, audio_tagging_loss=0.008747, over 15067.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09563, pruned_loss=0.0156, audio_tagging_loss=0.009338, over 3056709.87 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:21:15,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1866633.3333333333, ans=0.1
2023-11-22 08:21:16,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1866633.3333333333, ans=0.2
2023-11-22 08:21:19,571 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280000
2023-11-22 08:22:08,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1866833.3333333333, ans=0.2
2023-11-22 08:22:23,251 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3500, loss[loss=0.06943, simple_loss=0.09231, pruned_loss=0.014, audio_tagging_loss=0.009266, over 14464.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09491, pruned_loss=0.01541, audio_tagging_loss=0.009257, over 3049702.42 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:22:23,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1866966.6666666667, ans=0.1
2023-11-22 08:22:25,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1866966.6666666667, ans=0.125
2023-11-22 08:22:26,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280050
2023-11-22 08:22:27,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1866966.6666666667, ans=0.09899494936611666
2023-11-22 08:22:56,267 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.244e+01 8.926e+01 9.794e+01 1.238e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 08:22:57,679 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:22:58,410 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.67 vs. limit=15.0
2023-11-22 08:23:29,253 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3550, loss[loss=0.09284, simple_loss=0.1306, pruned_loss=0.02157, audio_tagging_loss=0.005975, over 15448.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09496, pruned_loss=0.01555, audio_tagging_loss=0.009411, over 3045007.36 frames. ], batch size: 53, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:23:33,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280100
2023-11-22 08:23:35,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1867300.0, ans=0.125
2023-11-22 08:23:43,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1867366.6666666667, ans=0.1
2023-11-22 08:23:50,419 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.55 vs. limit=22.5
2023-11-22 08:24:17,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1867500.0, ans=0.0
2023-11-22 08:24:26,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.86 vs. limit=15.0
2023-11-22 08:24:31,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1867566.6666666667, ans=0.125
2023-11-22 08:24:34,297 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3600, loss[loss=0.05934, simple_loss=0.08351, pruned_loss=0.01003, audio_tagging_loss=0.007549, over 15115.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09458, pruned_loss=0.01544, audio_tagging_loss=0.009285, over 3046472.06 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:24:38,062 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280150
2023-11-22 08:24:53,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1867700.0, ans=0.1
2023-11-22 08:25:07,391 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 7.927e+01 8.588e+01 9.278e+01 1.263e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 08:25:39,381 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3650, loss[loss=0.06259, simple_loss=0.08059, pruned_loss=0.01242, audio_tagging_loss=0.009873, over 15501.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09463, pruned_loss=0.0154, audio_tagging_loss=0.009161, over 3045206.92 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:25:43,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280200
2023-11-22 08:25:43,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1867966.6666666667, ans=0.125
2023-11-22 08:25:59,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1868033.3333333333, ans=0.125
2023-11-22 08:26:06,954 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.67 vs. limit=15.0
2023-11-22 08:26:07,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1868100.0, ans=0.125
2023-11-22 08:26:12,914 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:26:19,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1868166.6666666667, ans=0.125
2023-11-22 08:26:44,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1868300.0, ans=0.125
2023-11-22 08:26:45,341 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3700, loss[loss=0.0566, simple_loss=0.07541, pruned_loss=0.01014, audio_tagging_loss=0.008755, over 15243.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09472, pruned_loss=0.01552, audio_tagging_loss=0.009231, over 3049551.81 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:26:45,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.66 vs. limit=15.0
2023-11-22 08:26:47,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1868300.0, ans=0.0
2023-11-22 08:26:49,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280250
2023-11-22 08:27:12,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1868433.3333333333, ans=0.125
2023-11-22 08:27:16,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1868433.3333333333, ans=0.125
2023-11-22 08:27:18,804 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.045e+01 8.787e+01 9.464e+01 1.178e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 08:27:29,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1868500.0, ans=0.125
2023-11-22 08:27:31,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1868500.0, ans=0.125
2023-11-22 08:27:38,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1868566.6666666667, ans=6.0
2023-11-22 08:27:49,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.37 vs. limit=15.0
2023-11-22 08:27:50,871 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3750, loss[loss=0.05815, simple_loss=0.07043, pruned_loss=0.01049, audio_tagging_loss=0.01245, over 15302.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09398, pruned_loss=0.01544, audio_tagging_loss=0.009386, over 3054354.96 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:27:54,649 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280300
2023-11-22 08:27:56,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1868633.3333333333, ans=0.125
2023-11-22 08:28:08,104 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.98 vs. limit=15.0
2023-11-22 08:28:35,789 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:28:37,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1868833.3333333333, ans=0.125
2023-11-22 08:28:47,998 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=22.5
2023-11-22 08:28:53,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1868900.0, ans=0.0
2023-11-22 08:28:55,634 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3800, loss[loss=0.06379, simple_loss=0.08654, pruned_loss=0.01226, audio_tagging_loss=0.008264, over 15957.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09417, pruned_loss=0.01541, audio_tagging_loss=0.009448, over 3056337.10 frames. ], batch size: 61, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:28:59,333 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280350
2023-11-22 08:29:28,707 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.943e+01 8.492e+01 9.128e+01 1.004e+02 1.242e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-22 08:29:30,764 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:29:33,330 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:29:36,819 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:29:55,010 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.80 vs. limit=15.0
2023-11-22 08:29:59,893 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3850, loss[loss=0.07303, simple_loss=0.1036, pruned_loss=0.01331, audio_tagging_loss=0.007933, over 14864.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.0939, pruned_loss=0.01534, audio_tagging_loss=0.009517, over 3047670.53 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:30:04,375 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280400
2023-11-22 08:30:40,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1869500.0, ans=0.125
2023-11-22 08:31:01,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1869566.6666666667, ans=0.125
2023-11-22 08:31:04,438 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3900, loss[loss=0.07895, simple_loss=0.1177, pruned_loss=0.01428, audio_tagging_loss=0.00579, over 14864.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09393, pruned_loss=0.01527, audio_tagging_loss=0.009554, over 3039872.70 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:31:08,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280450
2023-11-22 08:31:33,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1869766.6666666667, ans=0.125
2023-11-22 08:31:33,592 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.77 vs. limit=15.0
2023-11-22 08:31:35,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1869766.6666666667, ans=0.2
2023-11-22 08:31:38,008 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.174e+01 8.873e+01 9.506e+01 1.121e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 08:31:46,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1869833.3333333333, ans=0.125
2023-11-22 08:32:00,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1869900.0, ans=6.0
2023-11-22 08:32:06,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1869900.0, ans=0.125
2023-11-22 08:32:10,102 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 3950, loss[loss=0.09401, simple_loss=0.1285, pruned_loss=0.02239, audio_tagging_loss=0.007358, over 15817.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09388, pruned_loss=0.01512, audio_tagging_loss=0.009627, over 3052229.40 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:32:13,971 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280500
2023-11-22 08:32:32,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1870033.3333333333, ans=0.125
2023-11-22 08:32:35,160 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.61 vs. limit=15.0
2023-11-22 08:32:43,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1870100.0, ans=0.125
2023-11-22 08:32:47,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1870166.6666666667, ans=0.05
2023-11-22 08:32:53,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1870166.6666666667, ans=0.125
2023-11-22 08:32:58,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1870166.6666666667, ans=0.125
2023-11-22 08:33:03,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1870233.3333333333, ans=0.125
2023-11-22 08:33:13,443 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4000, loss[loss=0.06505, simple_loss=0.09143, pruned_loss=0.01107, audio_tagging_loss=0.008268, over 15088.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09397, pruned_loss=0.01531, audio_tagging_loss=0.009714, over 3054829.19 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:33:13,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1870300.0, ans=0.125
2023-11-22 08:33:17,824 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280550
2023-11-22 08:33:43,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.52 vs. limit=10.0
2023-11-22 08:33:44,716 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.37 vs. limit=15.0
2023-11-22 08:33:47,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1870433.3333333333, ans=0.09899494936611666
2023-11-22 08:33:48,394 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.016e+01 8.419e+01 9.002e+01 9.753e+01 1.228e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 08:34:13,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1870566.6666666667, ans=0.125
2023-11-22 08:34:18,115 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4050, loss[loss=0.08575, simple_loss=0.1218, pruned_loss=0.01581, audio_tagging_loss=0.009018, over 15694.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09432, pruned_loss=0.01544, audio_tagging_loss=0.009727, over 3052407.57 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:34:21,993 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:34:22,022 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280600
2023-11-22 08:34:28,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1870633.3333333333, ans=0.2
2023-11-22 08:34:41,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1870700.0, ans=0.0
2023-11-22 08:34:42,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1870700.0, ans=0.2
2023-11-22 08:34:43,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1870766.6666666667, ans=0.0
2023-11-22 08:34:43,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1870766.6666666667, ans=0.125
2023-11-22 08:34:49,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1870766.6666666667, ans=0.0
2023-11-22 08:34:56,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1870833.3333333333, ans=0.125
2023-11-22 08:35:03,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1870833.3333333333, ans=0.125
2023-11-22 08:35:08,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1870833.3333333333, ans=0.2
2023-11-22 08:35:22,994 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4100, loss[loss=0.08273, simple_loss=0.1092, pruned_loss=0.01746, audio_tagging_loss=0.01069, over 16349.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09438, pruned_loss=0.01548, audio_tagging_loss=0.009694, over 3047614.20 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:35:27,401 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280650
2023-11-22 08:35:41,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1871033.3333333333, ans=0.125
2023-11-22 08:35:42,843 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:35:47,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.07 vs. limit=22.5
2023-11-22 08:35:57,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1871100.0, ans=0.125
2023-11-22 08:35:58,177 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.230e+01 8.854e+01 9.579e+01 2.804e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-22 08:35:59,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.18 vs. limit=15.0
2023-11-22 08:36:00,145 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.84 vs. limit=15.0
2023-11-22 08:36:15,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1871233.3333333333, ans=0.125
2023-11-22 08:36:15,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1871233.3333333333, ans=0.1
2023-11-22 08:36:27,872 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4150, loss[loss=0.0792, simple_loss=0.108, pruned_loss=0.01744, audio_tagging_loss=0.00775, over 15563.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09483, pruned_loss=0.01549, audio_tagging_loss=0.00956, over 3048531.43 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:36:31,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280700
2023-11-22 08:36:42,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1871366.6666666667, ans=0.125
2023-11-22 08:36:47,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1871366.6666666667, ans=0.125
2023-11-22 08:37:00,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1871433.3333333333, ans=0.05
2023-11-22 08:37:06,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1871500.0, ans=0.125
2023-11-22 08:37:10,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1871500.0, ans=0.1
2023-11-22 08:37:15,481 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:37:32,645 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4200, loss[loss=0.07553, simple_loss=0.1043, pruned_loss=0.01516, audio_tagging_loss=0.008229, over 15276.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.0954, pruned_loss=0.01567, audio_tagging_loss=0.009381, over 3052513.34 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:37:35,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1871633.3333333333, ans=0.2
2023-11-22 08:37:36,359 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280750
2023-11-22 08:37:43,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1871700.0, ans=0.0
2023-11-22 08:37:59,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.84 vs. limit=15.0
2023-11-22 08:38:05,111 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:38:06,973 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.52 vs. limit=15.0
2023-11-22 08:38:07,287 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.240e+01 8.912e+01 9.752e+01 1.165e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 08:38:09,575 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.22 vs. limit=15.0
2023-11-22 08:38:37,259 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4250, loss[loss=0.072, simple_loss=0.08287, pruned_loss=0.01701, audio_tagging_loss=0.01356, over 15459.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09624, pruned_loss=0.01581, audio_tagging_loss=0.00926, over 3055544.41 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:38:40,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1871966.6666666667, ans=0.125
2023-11-22 08:38:41,787 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280800
2023-11-22 08:39:18,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1872166.6666666667, ans=0.025
2023-11-22 08:39:29,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1872233.3333333333, ans=0.1
2023-11-22 08:39:43,325 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4300, loss[loss=0.07491, simple_loss=0.1003, pruned_loss=0.01501, audio_tagging_loss=0.009732, over 15322.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09603, pruned_loss=0.01579, audio_tagging_loss=0.009245, over 3057686.52 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:39:43,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1872300.0, ans=0.04949747468305833
2023-11-22 08:39:47,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280850
2023-11-22 08:40:10,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1872433.3333333333, ans=0.0
2023-11-22 08:40:18,677 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.496e+01 9.037e+01 9.736e+01 1.212e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-22 08:40:19,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1872433.3333333333, ans=0.1
2023-11-22 08:40:28,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1872500.0, ans=0.2
2023-11-22 08:40:30,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1872500.0, ans=0.0
2023-11-22 08:40:37,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1872566.6666666667, ans=0.0
2023-11-22 08:40:49,248 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4350, loss[loss=0.05656, simple_loss=0.07096, pruned_loss=0.01044, audio_tagging_loss=0.01065, over 14944.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09504, pruned_loss=0.01558, audio_tagging_loss=0.00923, over 3050394.11 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:40:53,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280900
2023-11-22 08:41:05,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1872700.0, ans=0.2
2023-11-22 08:41:32,442 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.35 vs. limit=15.0
2023-11-22 08:41:45,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.12 vs. limit=15.0
2023-11-22 08:41:53,076 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4400, loss[loss=0.0733, simple_loss=0.095, pruned_loss=0.01614, audio_tagging_loss=0.009665, over 16087.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09488, pruned_loss=0.0155, audio_tagging_loss=0.009267, over 3051273.73 frames. ], batch size: 61, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:41:56,936 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 280950
2023-11-22 08:42:13,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.02 vs. limit=10.0
2023-11-22 08:42:27,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1873100.0, ans=0.125
2023-11-22 08:42:29,226 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.700e+01 8.342e+01 8.985e+01 9.628e+01 1.162e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 08:42:40,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1873166.6666666667, ans=0.0
2023-11-22 08:42:58,195 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4450, loss[loss=0.0876, simple_loss=0.1187, pruned_loss=0.01907, audio_tagging_loss=0.009207, over 15344.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09484, pruned_loss=0.01542, audio_tagging_loss=0.009206, over 3060991.57 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:43:01,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281000
2023-11-22 08:43:25,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1873433.3333333333, ans=0.125
2023-11-22 08:43:46,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1873500.0, ans=0.0
2023-11-22 08:44:03,752 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4500, loss[loss=0.08689, simple_loss=0.1104, pruned_loss=0.023, audio_tagging_loss=0.008687, over 14799.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09478, pruned_loss=0.01547, audio_tagging_loss=0.009239, over 3056434.73 frames. ], batch size: 53, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:44:07,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281050
2023-11-22 08:44:30,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1873766.6666666667, ans=0.125
2023-11-22 08:44:37,528 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.057e+01 8.748e+01 9.801e+01 1.166e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 08:44:42,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1873833.3333333333, ans=0.125
2023-11-22 08:44:42,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1873833.3333333333, ans=0.0
2023-11-22 08:44:44,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1873833.3333333333, ans=0.0
2023-11-22 08:44:45,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1873833.3333333333, ans=0.0
2023-11-22 08:44:47,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1873833.3333333333, ans=0.1
2023-11-22 08:45:07,107 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4550, loss[loss=0.08483, simple_loss=0.1076, pruned_loss=0.02189, audio_tagging_loss=0.009138, over 14639.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09425, pruned_loss=0.01534, audio_tagging_loss=0.009259, over 3051502.98 frames. ], batch size: 53, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:45:10,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281100
2023-11-22 08:45:11,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1873966.6666666667, ans=0.125
2023-11-22 08:45:27,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1874033.3333333333, ans=0.0
2023-11-22 08:45:34,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1874100.0, ans=0.0
2023-11-22 08:45:42,674 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.37 vs. limit=6.0
2023-11-22 08:45:55,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1874166.6666666667, ans=0.125
2023-11-22 08:45:56,335 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:45:56,821 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.02 vs. limit=15.0
2023-11-22 08:46:11,532 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4600, loss[loss=0.07713, simple_loss=0.1038, pruned_loss=0.01734, audio_tagging_loss=0.007861, over 15319.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09412, pruned_loss=0.01535, audio_tagging_loss=0.009343, over 3043235.68 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:46:15,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281150
2023-11-22 08:46:34,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.32 vs. limit=22.5
2023-11-22 08:46:40,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1874433.3333333333, ans=0.0
2023-11-22 08:46:44,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.26 vs. limit=22.5
2023-11-22 08:46:46,496 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 7.994e+01 8.789e+01 9.560e+01 1.302e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 08:47:00,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.77 vs. limit=15.0
2023-11-22 08:47:00,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1874500.0, ans=0.125
2023-11-22 08:47:02,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1874566.6666666667, ans=0.125
2023-11-22 08:47:13,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1874566.6666666667, ans=0.2
2023-11-22 08:47:16,075 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4650, loss[loss=0.07327, simple_loss=0.09678, pruned_loss=0.01463, audio_tagging_loss=0.01026, over 15454.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09442, pruned_loss=0.01539, audio_tagging_loss=0.009346, over 3049924.63 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:47:20,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281200
2023-11-22 08:47:26,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1874633.3333333333, ans=0.125
2023-11-22 08:47:31,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1874700.0, ans=0.0
2023-11-22 08:47:35,688 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.67 vs. limit=12.0
2023-11-22 08:47:36,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1874700.0, ans=0.0
2023-11-22 08:47:38,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.03 vs. limit=15.0
2023-11-22 08:47:40,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.49 vs. limit=15.0
2023-11-22 08:48:13,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1874900.0, ans=0.1
2023-11-22 08:48:21,838 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4700, loss[loss=0.07032, simple_loss=0.08565, pruned_loss=0.01633, audio_tagging_loss=0.01117, over 14222.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09428, pruned_loss=0.01542, audio_tagging_loss=0.009368, over 3047023.58 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:48:23,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1874966.6666666667, ans=0.1
2023-11-22 08:48:25,655 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281250
2023-11-22 08:48:57,764 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 8.141e+01 8.777e+01 9.599e+01 1.409e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 08:49:00,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1875166.6666666667, ans=0.0
2023-11-22 08:49:02,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0
2023-11-22 08:49:13,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1875233.3333333333, ans=0.2
2023-11-22 08:49:18,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1875233.3333333333, ans=0.125
2023-11-22 08:49:25,816 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4750, loss[loss=0.0823, simple_loss=0.1012, pruned_loss=0.02178, audio_tagging_loss=0.009917, over 14636.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09489, pruned_loss=0.0154, audio_tagging_loss=0.009458, over 3046593.30 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:49:26,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1875300.0, ans=0.0
2023-11-22 08:49:29,571 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281300
2023-11-22 08:49:29,981 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=12.0
2023-11-22 08:49:33,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.45 vs. limit=12.0
2023-11-22 08:49:59,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1875433.3333333333, ans=0.2
2023-11-22 08:50:06,337 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.09 vs. limit=15.0
2023-11-22 08:50:21,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1875566.6666666667, ans=0.0
2023-11-22 08:50:29,583 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4800, loss[loss=0.08334, simple_loss=0.1023, pruned_loss=0.01925, audio_tagging_loss=0.01293, over 16094.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09431, pruned_loss=0.01533, audio_tagging_loss=0.009634, over 3047796.15 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:50:31,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1875633.3333333333, ans=0.125
2023-11-22 08:50:34,393 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281350
2023-11-22 08:50:37,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1875633.3333333333, ans=0.2
2023-11-22 08:50:56,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1875766.6666666667, ans=0.1
2023-11-22 08:50:59,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1875766.6666666667, ans=0.0
2023-11-22 08:51:04,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1875766.6666666667, ans=0.0
2023-11-22 08:51:07,406 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.460e+01 8.056e+01 8.734e+01 9.359e+01 1.284e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 08:51:15,535 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.44 vs. limit=6.0
2023-11-22 08:51:27,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1875900.0, ans=0.0
2023-11-22 08:51:34,895 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4850, loss[loss=0.09236, simple_loss=0.1175, pruned_loss=0.02502, audio_tagging_loss=0.008592, over 14708.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09522, pruned_loss=0.01563, audio_tagging_loss=0.009678, over 3054523.49 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:51:38,658 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281400
2023-11-22 08:51:40,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1875966.6666666667, ans=0.0
2023-11-22 08:51:44,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.51 vs. limit=15.0
2023-11-22 08:51:46,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1876033.3333333333, ans=0.125
2023-11-22 08:51:48,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1876033.3333333333, ans=0.0
2023-11-22 08:51:50,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1876033.3333333333, ans=0.1
2023-11-22 08:51:59,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1876100.0, ans=0.0
2023-11-22 08:52:00,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.38 vs. limit=15.0
2023-11-22 08:52:17,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_na.min_abs, batch_count=1876166.6666666667, ans=0.02
2023-11-22 08:52:31,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1876233.3333333333, ans=0.2
2023-11-22 08:52:39,572 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4900, loss[loss=0.06651, simple_loss=0.08019, pruned_loss=0.01178, audio_tagging_loss=0.01464, over 16040.00 frames. ], tot_loss[loss=0.07285, simple_loss=0.09529, pruned_loss=0.01562, audio_tagging_loss=0.009589, over 3054030.91 frames. ], batch size: 60, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:52:41,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1876300.0, ans=0.125
2023-11-22 08:52:43,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281450
2023-11-22 08:52:43,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1876300.0, ans=0.125
2023-11-22 08:53:16,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.163e+01 8.764e+01 9.409e+01 1.191e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 08:53:28,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1876500.0, ans=0.0
2023-11-22 08:53:28,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1876500.0, ans=0.125
2023-11-22 08:53:43,090 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 4950, loss[loss=0.06497, simple_loss=0.08849, pruned_loss=0.01475, audio_tagging_loss=0.005976, over 14958.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.09513, pruned_loss=0.01549, audio_tagging_loss=0.009459, over 3059491.98 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:53:47,680 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281500
2023-11-22 08:53:49,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1876633.3333333333, ans=0.0
2023-11-22 08:54:01,173 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.34 vs. limit=10.0
2023-11-22 08:54:31,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.39 vs. limit=15.0
2023-11-22 08:54:33,156 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.57 vs. limit=10.0
2023-11-22 08:54:35,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1876900.0, ans=0.125
2023-11-22 08:54:48,548 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5000, loss[loss=0.06606, simple_loss=0.08434, pruned_loss=0.01446, audio_tagging_loss=0.00943, over 15417.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.0949, pruned_loss=0.01531, audio_tagging_loss=0.00926, over 3050828.67 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:54:52,329 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281550
2023-11-22 08:55:00,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1877033.3333333333, ans=0.025
2023-11-22 08:55:02,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1877033.3333333333, ans=0.125
2023-11-22 08:55:23,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1877100.0, ans=0.0
2023-11-22 08:55:25,178 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.261e+01 8.670e+01 9.360e+01 1.108e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-22 08:55:50,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1877233.3333333333, ans=10.0
2023-11-22 08:55:52,729 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5050, loss[loss=0.06374, simple_loss=0.07835, pruned_loss=0.01524, audio_tagging_loss=0.009323, over 14140.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09498, pruned_loss=0.01522, audio_tagging_loss=0.009148, over 3049213.41 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:55:57,041 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281600
2023-11-22 08:56:12,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1877366.6666666667, ans=0.125
2023-11-22 08:56:37,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1877500.0, ans=0.05
2023-11-22 08:56:44,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1877566.6666666667, ans=0.125
2023-11-22 08:56:57,103 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5100, loss[loss=0.06949, simple_loss=0.08768, pruned_loss=0.01701, audio_tagging_loss=0.00863, over 14298.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.0951, pruned_loss=0.01517, audio_tagging_loss=0.009176, over 3047994.09 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:57:00,791 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281650
2023-11-22 08:57:02,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1877633.3333333333, ans=0.0
2023-11-22 08:57:29,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1877766.6666666667, ans=0.1
2023-11-22 08:57:32,723 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.75 vs. limit=12.0
2023-11-22 08:57:33,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1877766.6666666667, ans=0.1
2023-11-22 08:57:34,831 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.920e+01 7.966e+01 8.782e+01 9.481e+01 2.193e+02, threshold=1.756e+02, percent-clipped=1.0
2023-11-22 08:58:00,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1877966.6666666667, ans=0.125
2023-11-22 08:58:01,516 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5150, loss[loss=0.06367, simple_loss=0.08207, pruned_loss=0.01493, audio_tagging_loss=0.007708, over 15469.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09469, pruned_loss=0.01502, audio_tagging_loss=0.009139, over 3048784.16 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:58:05,200 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281700
2023-11-22 08:58:13,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1878033.3333333333, ans=0.2
2023-11-22 08:58:28,751 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.73 vs. limit=15.0
2023-11-22 08:58:29,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1878100.0, ans=0.1
2023-11-22 08:58:36,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1878100.0, ans=0.125
2023-11-22 08:58:42,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1878166.6666666667, ans=0.015
2023-11-22 08:58:51,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1878233.3333333333, ans=0.09899494936611666
2023-11-22 08:58:51,978 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.41 vs. limit=6.0
2023-11-22 08:58:52,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1878233.3333333333, ans=0.0
2023-11-22 08:59:04,829 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5200, loss[loss=0.05644, simple_loss=0.07105, pruned_loss=0.01063, audio_tagging_loss=0.01029, over 17049.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.0936, pruned_loss=0.01492, audio_tagging_loss=0.009175, over 3052398.92 frames. ], batch size: 66, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:59:08,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281750
2023-11-22 08:59:31,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1878433.3333333333, ans=0.125
2023-11-22 08:59:42,172 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.47 vs. limit=15.0
2023-11-22 08:59:42,533 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.104e+01 8.582e+01 9.336e+01 1.183e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 08:59:53,057 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.72 vs. limit=22.5
2023-11-22 09:00:09,543 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5250, loss[loss=0.07657, simple_loss=0.09676, pruned_loss=0.0182, audio_tagging_loss=0.009984, over 15614.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.0949, pruned_loss=0.01518, audio_tagging_loss=0.009114, over 3056671.74 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 09:00:13,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281800
2023-11-22 09:00:33,723 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.35 vs. limit=12.0
2023-11-22 09:00:45,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1878766.6666666667, ans=0.2
2023-11-22 09:00:54,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.33 vs. limit=22.5
2023-11-22 09:00:56,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1878833.3333333333, ans=0.2
2023-11-22 09:01:09,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1878900.0, ans=0.0
2023-11-22 09:01:14,902 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5300, loss[loss=0.06315, simple_loss=0.08576, pruned_loss=0.01185, audio_tagging_loss=0.008425, over 16894.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09554, pruned_loss=0.01525, audio_tagging_loss=0.009038, over 3047103.59 frames. ], batch size: 64, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 09:01:18,605 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281850
2023-11-22 09:01:42,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1879100.0, ans=0.0
2023-11-22 09:01:48,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1879100.0, ans=0.0
2023-11-22 09:01:53,085 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.133e+01 8.772e+01 9.528e+01 1.202e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 09:02:08,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1879233.3333333333, ans=0.0
2023-11-22 09:02:13,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1879233.3333333333, ans=0.0
2023-11-22 09:02:19,276 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5350, loss[loss=0.06891, simple_loss=0.09239, pruned_loss=0.01404, audio_tagging_loss=0.008669, over 15470.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09529, pruned_loss=0.01516, audio_tagging_loss=0.009048, over 3048511.95 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:02:23,084 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281900
2023-11-22 09:02:59,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1879500.0, ans=0.2
2023-11-22 09:03:23,609 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5400, loss[loss=0.08434, simple_loss=0.1159, pruned_loss=0.01975, audio_tagging_loss=0.006644, over 14829.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09621, pruned_loss=0.01543, audio_tagging_loss=0.009144, over 3049433.09 frames. ], batch size: 52, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:03:27,966 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 281950
2023-11-22 09:04:02,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.289e+01 8.769e+01 9.344e+01 1.157e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 09:04:09,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1879833.3333333333, ans=0.0
2023-11-22 09:04:11,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1879833.3333333333, ans=0.125
2023-11-22 09:04:13,461 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.62 vs. limit=22.5
2023-11-22 09:04:20,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1879900.0, ans=0.04949747468305833
2023-11-22 09:04:29,465 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5450, loss[loss=0.049, simple_loss=0.05621, pruned_loss=0.009265, audio_tagging_loss=0.01163, over 14972.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09491, pruned_loss=0.01524, audio_tagging_loss=0.00932, over 3043959.44 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:04:31,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1879966.6666666667, ans=0.1
2023-11-22 09:04:33,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282000
2023-11-22 09:04:36,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1879966.6666666667, ans=0.125
2023-11-22 09:04:37,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1879966.6666666667, ans=0.0
2023-11-22 09:04:44,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1880033.3333333333, ans=0.2
2023-11-22 09:04:54,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1880100.0, ans=0.125
2023-11-22 09:04:58,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1880100.0, ans=0.2
2023-11-22 09:05:04,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1880100.0, ans=0.125
2023-11-22 09:05:07,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1880166.6666666667, ans=0.125
2023-11-22 09:05:10,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1880166.6666666667, ans=0.0
2023-11-22 09:05:26,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.00 vs. limit=12.0
2023-11-22 09:05:33,760 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5500, loss[loss=0.07999, simple_loss=0.103, pruned_loss=0.01761, audio_tagging_loss=0.01091, over 14602.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09502, pruned_loss=0.01533, audio_tagging_loss=0.009314, over 3051247.96 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:05:37,531 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282050
2023-11-22 09:05:44,192 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.05 vs. limit=15.0
2023-11-22 09:05:52,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1880366.6666666667, ans=0.125
2023-11-22 09:06:12,692 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.257e+01 8.828e+01 9.557e+01 1.200e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 09:06:38,159 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5550, loss[loss=0.07378, simple_loss=0.09392, pruned_loss=0.01872, audio_tagging_loss=0.008097, over 15247.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09535, pruned_loss=0.01544, audio_tagging_loss=0.009314, over 3050113.86 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:06:41,293 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.21 vs. limit=6.0
2023-11-22 09:06:41,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282100
2023-11-22 09:06:45,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1880633.3333333333, ans=0.2
2023-11-22 09:07:12,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1880766.6666666667, ans=0.0
2023-11-22 09:07:31,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1880900.0, ans=0.0
2023-11-22 09:07:42,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1880966.6666666667, ans=0.125
2023-11-22 09:07:43,084 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5600, loss[loss=0.06458, simple_loss=0.08465, pruned_loss=0.01199, audio_tagging_loss=0.01026, over 15198.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09546, pruned_loss=0.01541, audio_tagging_loss=0.009442, over 3048629.36 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 09:07:47,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282150
2023-11-22 09:07:57,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1881033.3333333333, ans=10.0
2023-11-22 09:08:03,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1881033.3333333333, ans=0.0
2023-11-22 09:08:21,231 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.179e+01 8.745e+01 9.416e+01 1.882e+02, threshold=1.749e+02, percent-clipped=1.0
2023-11-22 09:08:26,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1881166.6666666667, ans=0.0
2023-11-22 09:08:30,614 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:08:41,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1881233.3333333333, ans=0.125
2023-11-22 09:08:47,861 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5650, loss[loss=0.08144, simple_loss=0.09685, pruned_loss=0.02299, audio_tagging_loss=0.01001, over 14920.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09517, pruned_loss=0.0154, audio_tagging_loss=0.00949, over 3047863.90 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:08:51,643 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282200
2023-11-22 09:08:51,784 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:08:58,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.67 vs. limit=15.0
2023-11-22 09:09:09,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1881366.6666666667, ans=0.0
2023-11-22 09:09:26,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1881500.0, ans=0.125
2023-11-22 09:09:35,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1881500.0, ans=0.0
2023-11-22 09:09:51,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1881633.3333333333, ans=0.125
2023-11-22 09:09:52,837 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5700, loss[loss=0.06658, simple_loss=0.08268, pruned_loss=0.0135, audio_tagging_loss=0.01174, over 15280.00 frames. ], tot_loss[loss=0.07219, simple_loss=0.09459, pruned_loss=0.01535, audio_tagging_loss=0.009545, over 3046814.97 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:09:56,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282250
2023-11-22 09:10:12,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1881700.0, ans=0.07
2023-11-22 09:10:21,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1881766.6666666667, ans=0.1
2023-11-22 09:10:24,812 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:10:30,658 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.915e+01 8.108e+01 8.715e+01 9.329e+01 1.149e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 09:10:55,633 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5750, loss[loss=0.08405, simple_loss=0.1098, pruned_loss=0.01971, audio_tagging_loss=0.009428, over 15175.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.0945, pruned_loss=0.01548, audio_tagging_loss=0.009538, over 3056595.74 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:10:59,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282300
2023-11-22 09:11:05,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1881966.6666666667, ans=0.2
2023-11-22 09:11:11,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1882033.3333333333, ans=0.125
2023-11-22 09:11:21,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.61 vs. limit=22.5
2023-11-22 09:11:21,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1882100.0, ans=0.125
2023-11-22 09:11:23,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.31 vs. limit=15.0
2023-11-22 09:11:51,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-22 09:12:00,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1882300.0, ans=0.125
2023-11-22 09:12:01,184 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5800, loss[loss=0.05218, simple_loss=0.06288, pruned_loss=0.01172, audio_tagging_loss=0.009024, over 17549.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09458, pruned_loss=0.01545, audio_tagging_loss=0.009378, over 3059805.27 frames. ], batch size: 71, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:12:04,899 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282350
2023-11-22 09:12:16,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1882366.6666666667, ans=0.125
2023-11-22 09:12:22,916 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.26 vs. limit=15.0
2023-11-22 09:12:39,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1882500.0, ans=0.0
2023-11-22 09:12:40,747 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 8.216e+01 8.857e+01 9.622e+01 1.311e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 09:12:46,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1882500.0, ans=0.125
2023-11-22 09:12:55,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1882566.6666666667, ans=0.125
2023-11-22 09:12:59,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1882566.6666666667, ans=0.125
2023-11-22 09:13:05,431 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5850, loss[loss=0.07301, simple_loss=0.09928, pruned_loss=0.01315, audio_tagging_loss=0.01022, over 14957.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09404, pruned_loss=0.01527, audio_tagging_loss=0.009326, over 3063605.47 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:13:09,299 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282400
2023-11-22 09:13:33,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1882766.6666666667, ans=15.0
2023-11-22 09:13:44,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.41 vs. limit=15.0
2023-11-22 09:13:58,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.92 vs. limit=15.0
2023-11-22 09:14:10,124 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5900, loss[loss=0.06734, simple_loss=0.0843, pruned_loss=0.01777, audio_tagging_loss=0.007413, over 15548.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09403, pruned_loss=0.0153, audio_tagging_loss=0.00927, over 3061489.13 frames. ], batch size: 61, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:14:14,466 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282450
2023-11-22 09:14:15,008 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.86 vs. limit=12.0
2023-11-22 09:14:22,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1883033.3333333333, ans=0.1
2023-11-22 09:14:32,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1883033.3333333333, ans=0.125
2023-11-22 09:14:38,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1883100.0, ans=0.1
2023-11-22 09:14:50,070 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 8.181e+01 9.057e+01 9.624e+01 1.134e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 09:14:50,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=1883166.6666666667, ans=0.1
2023-11-22 09:15:14,468 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 5950, loss[loss=0.0728, simple_loss=0.09271, pruned_loss=0.01653, audio_tagging_loss=0.009913, over 14138.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.094, pruned_loss=0.01543, audio_tagging_loss=0.009309, over 3060589.68 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:15:18,796 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282500
2023-11-22 09:15:29,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0
2023-11-22 09:15:34,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1883366.6666666667, ans=0.125
2023-11-22 09:15:56,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1883500.0, ans=0.0
2023-11-22 09:16:19,184 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6000, loss[loss=0.05342, simple_loss=0.0653, pruned_loss=0.01009, audio_tagging_loss=0.01069, over 14237.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09523, pruned_loss=0.01578, audio_tagging_loss=0.009268, over 3058306.12 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:16:19,184 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 09:17:01,482 INFO [train_asr.py:1253] (2/4) Epoch 24, validation: loss=0.05933, simple_loss=0.05174, pruned_loss=0.005222, audio_tagging_loss=0.02824, over 4681554.00 frames. 
2023-11-22 09:17:01,483 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 09:17:05,858 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282550
2023-11-22 09:17:39,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1883833.3333333333, ans=0.125
2023-11-22 09:17:42,986 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.160e+01 8.694e+01 9.357e+01 1.133e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 09:17:49,659 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:17:52,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1883900.0, ans=0.025
2023-11-22 09:18:01,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1883900.0, ans=0.0
2023-11-22 09:18:03,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1883900.0, ans=0.95
2023-11-22 09:18:06,501 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6050, loss[loss=0.05877, simple_loss=0.06441, pruned_loss=0.01379, audio_tagging_loss=0.01278, over 14046.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.0949, pruned_loss=0.01552, audio_tagging_loss=0.00921, over 3062436.91 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:18:09,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1883966.6666666667, ans=0.1
2023-11-22 09:18:10,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282600
2023-11-22 09:18:46,934 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.74 vs. limit=22.5
2023-11-22 09:18:50,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1884166.6666666667, ans=0.125
2023-11-22 09:18:56,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1884166.6666666667, ans=0.2
2023-11-22 09:18:57,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1884233.3333333333, ans=0.125
2023-11-22 09:19:12,046 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6100, loss[loss=0.06548, simple_loss=0.08157, pruned_loss=0.01507, audio_tagging_loss=0.009622, over 13269.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09472, pruned_loss=0.01565, audio_tagging_loss=0.009251, over 3057909.60 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:19:15,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282650
2023-11-22 09:19:37,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1884433.3333333333, ans=0.1
2023-11-22 09:19:52,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1884500.0, ans=0.2
2023-11-22 09:19:53,493 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.538e+01 9.030e+01 1.014e+02 1.305e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 09:19:55,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1884500.0, ans=0.125
2023-11-22 09:19:58,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1884500.0, ans=0.0
2023-11-22 09:20:16,938 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6150, loss[loss=0.07341, simple_loss=0.09899, pruned_loss=0.01589, audio_tagging_loss=0.008024, over 15836.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09444, pruned_loss=0.0157, audio_tagging_loss=0.009318, over 3052601.85 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:20:20,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282700
2023-11-22 09:20:30,814 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.42 vs. limit=15.0
2023-11-22 09:20:34,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.80 vs. limit=15.0
2023-11-22 09:20:39,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1884700.0, ans=0.1
2023-11-22 09:21:03,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1884833.3333333333, ans=0.125
2023-11-22 09:21:04,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1884833.3333333333, ans=0.1
2023-11-22 09:21:09,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1884900.0, ans=0.125
2023-11-22 09:21:18,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.50 vs. limit=10.0
2023-11-22 09:21:22,138 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6200, loss[loss=0.06577, simple_loss=0.08538, pruned_loss=0.01346, audio_tagging_loss=0.009615, over 15687.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09355, pruned_loss=0.01549, audio_tagging_loss=0.00948, over 3047451.26 frames. ], batch size: 60, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:21:23,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1884966.6666666667, ans=0.1
2023-11-22 09:21:26,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282750
2023-11-22 09:21:40,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1885033.3333333333, ans=0.0
2023-11-22 09:22:03,950 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.132e+01 8.774e+01 9.442e+01 1.403e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 09:22:04,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1885166.6666666667, ans=0.125
2023-11-22 09:22:13,493 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.35 vs. limit=8.0
2023-11-22 09:22:26,714 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6250, loss[loss=0.05918, simple_loss=0.06732, pruned_loss=0.01119, audio_tagging_loss=0.01434, over 14875.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.0923, pruned_loss=0.01521, audio_tagging_loss=0.009583, over 3045592.42 frames. ], batch size: 60, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:22:31,139 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282800
2023-11-22 09:22:34,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.whiten.whitening_limit, batch_count=1885300.0, ans=15.0
2023-11-22 09:22:41,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1885366.6666666667, ans=0.125
2023-11-22 09:22:57,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1885433.3333333333, ans=0.125
2023-11-22 09:22:57,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1885433.3333333333, ans=0.125
2023-11-22 09:22:59,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0
2023-11-22 09:23:11,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1885500.0, ans=0.125
2023-11-22 09:23:15,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.70 vs. limit=6.0
2023-11-22 09:23:29,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1885633.3333333333, ans=0.125
2023-11-22 09:23:30,896 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6300, loss[loss=0.04182, simple_loss=0.04478, pruned_loss=0.00715, audio_tagging_loss=0.01228, over 16150.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09347, pruned_loss=0.01533, audio_tagging_loss=0.009485, over 3046233.81 frames. ], batch size: 64, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:23:32,785 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.20 vs. limit=22.5
2023-11-22 09:23:35,188 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282850
2023-11-22 09:23:44,275 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:24:11,933 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.966e+01 8.510e+01 9.202e+01 1.035e+02 1.385e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-22 09:24:18,663 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=22.5
2023-11-22 09:24:35,090 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6350, loss[loss=0.07318, simple_loss=0.09318, pruned_loss=0.01794, audio_tagging_loss=0.008648, over 14284.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09434, pruned_loss=0.01559, audio_tagging_loss=0.009543, over 3039507.80 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:24:38,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282900
2023-11-22 09:25:03,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1886100.0, ans=0.125
2023-11-22 09:25:17,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1886166.6666666667, ans=0.125
2023-11-22 09:25:38,859 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6400, loss[loss=0.08064, simple_loss=0.1108, pruned_loss=0.0163, audio_tagging_loss=0.008931, over 16421.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09452, pruned_loss=0.0155, audio_tagging_loss=0.009589, over 3036407.78 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:25:43,224 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 282950
2023-11-22 09:25:59,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1886366.6666666667, ans=0.2
2023-11-22 09:26:14,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1886433.3333333333, ans=0.125
2023-11-22 09:26:19,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1886500.0, ans=0.0
2023-11-22 09:26:20,179 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.176e+01 8.732e+01 9.554e+01 1.218e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 09:26:43,531 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6450, loss[loss=0.07424, simple_loss=0.08674, pruned_loss=0.0185, audio_tagging_loss=0.01237, over 15033.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.0947, pruned_loss=0.01543, audio_tagging_loss=0.009737, over 3045991.50 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:26:45,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1886633.3333333333, ans=0.125
2023-11-22 09:26:47,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283000
2023-11-22 09:26:55,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1886700.0, ans=0.125
2023-11-22 09:27:02,496 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.11 vs. limit=6.0
2023-11-22 09:27:04,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1886700.0, ans=0.1
2023-11-22 09:27:06,848 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:27:14,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1886766.6666666667, ans=0.125
2023-11-22 09:27:22,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1886833.3333333333, ans=0.07
2023-11-22 09:27:31,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1886833.3333333333, ans=0.1
2023-11-22 09:27:48,212 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.48 vs. limit=6.0
2023-11-22 09:27:48,671 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6500, loss[loss=0.08554, simple_loss=0.1196, pruned_loss=0.01811, audio_tagging_loss=0.007624, over 16979.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.0954, pruned_loss=0.01545, audio_tagging_loss=0.009618, over 3042172.02 frames. ], batch size: 61, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:27:52,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283050
2023-11-22 09:27:53,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.44 vs. limit=15.0
2023-11-22 09:27:54,365 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.04 vs. limit=15.0
2023-11-22 09:28:06,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.46 vs. limit=22.5
2023-11-22 09:28:11,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1887033.3333333333, ans=0.125
2023-11-22 09:28:24,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1887100.0, ans=0.1
2023-11-22 09:28:25,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1887166.6666666667, ans=10.0
2023-11-22 09:28:29,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1887166.6666666667, ans=0.125
2023-11-22 09:28:30,692 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.224e+01 8.620e+01 9.457e+01 1.249e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-22 09:28:34,531 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.43 vs. limit=22.5
2023-11-22 09:28:36,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1887166.6666666667, ans=0.0
2023-11-22 09:28:45,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=12.0
2023-11-22 09:28:52,082 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6550, loss[loss=0.0917, simple_loss=0.1291, pruned_loss=0.02102, audio_tagging_loss=0.006136, over 15227.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09458, pruned_loss=0.01518, audio_tagging_loss=0.009474, over 3051833.50 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:28:55,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283100
2023-11-22 09:29:05,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1887366.6666666667, ans=0.125
2023-11-22 09:29:07,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.44 vs. limit=15.0
2023-11-22 09:29:19,240 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.92 vs. limit=22.5
2023-11-22 09:29:43,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1887566.6666666667, ans=0.2
2023-11-22 09:29:56,330 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6600, loss[loss=0.0484, simple_loss=0.05092, pruned_loss=0.009011, audio_tagging_loss=0.01393, over 14392.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09388, pruned_loss=0.01521, audio_tagging_loss=0.009291, over 3046238.69 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:29:59,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283150
2023-11-22 09:30:38,600 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.912e+01 8.247e+01 8.842e+01 9.573e+01 1.466e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 09:30:42,850 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.00 vs. limit=15.0
2023-11-22 09:30:43,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1887833.3333333333, ans=0.07
2023-11-22 09:30:44,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1887833.3333333333, ans=0.1
2023-11-22 09:31:00,484 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6650, loss[loss=0.07907, simple_loss=0.1061, pruned_loss=0.01799, audio_tagging_loss=0.008021, over 15154.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09422, pruned_loss=0.01526, audio_tagging_loss=0.009245, over 3046083.42 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:31:00,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1887966.6666666667, ans=0.0
2023-11-22 09:31:04,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283200
2023-11-22 09:31:08,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1887966.6666666667, ans=0.125
2023-11-22 09:31:13,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1888033.3333333333, ans=10.0
2023-11-22 09:31:31,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1888100.0, ans=0.125
2023-11-22 09:31:47,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1888166.6666666667, ans=0.125
2023-11-22 09:31:47,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1888166.6666666667, ans=0.0
2023-11-22 09:32:04,412 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6700, loss[loss=0.0553, simple_loss=0.06693, pruned_loss=0.01255, audio_tagging_loss=0.009286, over 15304.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09392, pruned_loss=0.01521, audio_tagging_loss=0.009249, over 3046848.48 frames. ], batch size: 60, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:32:08,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283250
2023-11-22 09:32:10,848 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:32:17,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1888366.6666666667, ans=0.1
2023-11-22 09:32:21,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1888366.6666666667, ans=0.125
2023-11-22 09:32:47,394 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.099e+01 8.643e+01 9.373e+01 1.139e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 09:32:50,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1888500.0, ans=0.125
2023-11-22 09:33:06,447 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:33:08,698 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6750, loss[loss=0.06809, simple_loss=0.079, pruned_loss=0.01788, audio_tagging_loss=0.01071, over 14188.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09406, pruned_loss=0.01528, audio_tagging_loss=0.009271, over 3039955.83 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:33:12,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283300
2023-11-22 09:33:48,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1888833.3333333333, ans=0.125
2023-11-22 09:33:58,751 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.43 vs. limit=6.0
2023-11-22 09:34:12,848 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6800, loss[loss=0.08453, simple_loss=0.117, pruned_loss=0.01726, audio_tagging_loss=0.008752, over 14983.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09465, pruned_loss=0.01553, audio_tagging_loss=0.00921, over 3046491.40 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:34:17,143 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283350
2023-11-22 09:34:23,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1888966.6666666667, ans=0.0
2023-11-22 09:34:41,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1889100.0, ans=0.1
2023-11-22 09:34:54,671 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 7.933e+01 8.700e+01 9.598e+01 1.312e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 09:35:15,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1889233.3333333333, ans=0.0
2023-11-22 09:35:17,331 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6850, loss[loss=0.08467, simple_loss=0.1219, pruned_loss=0.01765, audio_tagging_loss=0.006041, over 15959.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09455, pruned_loss=0.01544, audio_tagging_loss=0.009137, over 3044252.66 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:35:21,139 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283400
2023-11-22 09:35:24,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1889300.0, ans=0.125
2023-11-22 09:36:10,620 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:36:11,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1889566.6666666667, ans=0.2
2023-11-22 09:36:21,958 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6900, loss[loss=0.07864, simple_loss=0.1011, pruned_loss=0.01759, audio_tagging_loss=0.01049, over 14510.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09468, pruned_loss=0.01549, audio_tagging_loss=0.009203, over 3043681.76 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:36:23,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-22 09:36:25,796 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283450
2023-11-22 09:36:28,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=1889633.3333333333, ans=0.02
2023-11-22 09:36:30,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1889633.3333333333, ans=0.1
2023-11-22 09:36:33,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1889700.0, ans=0.2
2023-11-22 09:36:39,195 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.81 vs. limit=22.5
2023-11-22 09:37:03,973 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.035e+01 8.885e+01 9.571e+01 1.179e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 09:37:13,112 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:37:14,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1889900.0, ans=0.1
2023-11-22 09:37:15,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1889900.0, ans=0.1
2023-11-22 09:37:25,901 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 6950, loss[loss=0.07535, simple_loss=0.09223, pruned_loss=0.01897, audio_tagging_loss=0.01026, over 15647.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09382, pruned_loss=0.01527, audio_tagging_loss=0.009193, over 3038491.70 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:37:30,228 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283500
2023-11-22 09:37:49,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1890033.3333333333, ans=0.0
2023-11-22 09:37:49,921 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=12.0
2023-11-22 09:38:12,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1890166.6666666667, ans=0.2
2023-11-22 09:38:24,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1890233.3333333333, ans=0.0
2023-11-22 09:38:29,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=1890233.3333333333, ans=15.0
2023-11-22 09:38:31,033 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7000, loss[loss=0.07309, simple_loss=0.09505, pruned_loss=0.01551, audio_tagging_loss=0.01005, over 16032.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09303, pruned_loss=0.0151, audio_tagging_loss=0.009324, over 3042094.22 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:38:34,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283550
2023-11-22 09:39:12,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1890500.0, ans=0.125
2023-11-22 09:39:13,198 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.396e+01 8.305e+01 8.768e+01 9.472e+01 1.582e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 09:39:16,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1890500.0, ans=0.0
2023-11-22 09:39:20,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1890500.0, ans=0.125
2023-11-22 09:39:30,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1890566.6666666667, ans=0.0
2023-11-22 09:39:35,637 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7050, loss[loss=0.07868, simple_loss=0.1023, pruned_loss=0.01862, audio_tagging_loss=0.008909, over 15965.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09302, pruned_loss=0.01491, audio_tagging_loss=0.009344, over 3042344.66 frames. ], batch size: 60, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:39:39,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283600
2023-11-22 09:39:40,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1890633.3333333333, ans=0.0
2023-11-22 09:40:12,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1890766.6666666667, ans=0.5
2023-11-22 09:40:36,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1890900.0, ans=0.125
2023-11-22 09:40:39,970 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7100, loss[loss=0.07272, simple_loss=0.1006, pruned_loss=0.0141, audio_tagging_loss=0.008302, over 16053.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09348, pruned_loss=0.01518, audio_tagging_loss=0.009397, over 3034613.69 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:40:43,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1890966.6666666667, ans=0.1
2023-11-22 09:40:44,367 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283650
2023-11-22 09:41:16,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1891100.0, ans=0.125
2023-11-22 09:41:23,143 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.301e+01 8.958e+01 9.633e+01 1.183e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 09:41:28,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1891166.6666666667, ans=0.125
2023-11-22 09:41:38,563 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.78 vs. limit=12.0
2023-11-22 09:41:43,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.88 vs. limit=15.0
2023-11-22 09:41:45,094 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7150, loss[loss=0.0613, simple_loss=0.07853, pruned_loss=0.01274, audio_tagging_loss=0.009294, over 15517.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09334, pruned_loss=0.01516, audio_tagging_loss=0.009536, over 3034599.04 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:41:48,887 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283700
2023-11-22 09:41:58,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1891366.6666666667, ans=0.2
2023-11-22 09:42:11,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1891433.3333333333, ans=0.1
2023-11-22 09:42:11,858 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.55 vs. limit=22.5
2023-11-22 09:42:32,513 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.30 vs. limit=12.0
2023-11-22 09:42:36,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1891566.6666666667, ans=0.125
2023-11-22 09:42:49,831 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7200, loss[loss=0.06602, simple_loss=0.08403, pruned_loss=0.01266, audio_tagging_loss=0.01134, over 15631.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09338, pruned_loss=0.01509, audio_tagging_loss=0.009642, over 3032725.15 frames. ], batch size: 62, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:42:54,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283750
2023-11-22 09:43:05,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1891700.0, ans=0.125
2023-11-22 09:43:32,278 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.224e+01 8.056e+01 8.758e+01 9.674e+01 1.275e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 09:43:48,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.76 vs. limit=15.0
2023-11-22 09:43:54,011 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7250, loss[loss=0.07757, simple_loss=0.1109, pruned_loss=0.01553, audio_tagging_loss=0.006568, over 15543.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09437, pruned_loss=0.01519, audio_tagging_loss=0.009584, over 3046633.69 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:43:55,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1891966.6666666667, ans=0.0
2023-11-22 09:43:57,756 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283800
2023-11-22 09:44:22,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1892100.0, ans=0.2
2023-11-22 09:44:29,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1892100.0, ans=0.125
2023-11-22 09:44:37,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1892166.6666666667, ans=0.2
2023-11-22 09:44:37,521 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=12.0
2023-11-22 09:44:39,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1892166.6666666667, ans=0.0
2023-11-22 09:44:40,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1892166.6666666667, ans=0.125
2023-11-22 09:44:48,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1892233.3333333333, ans=0.125
2023-11-22 09:44:55,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1892233.3333333333, ans=0.125
2023-11-22 09:44:59,077 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7300, loss[loss=0.06855, simple_loss=0.09153, pruned_loss=0.01465, audio_tagging_loss=0.008136, over 14607.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09333, pruned_loss=0.01494, audio_tagging_loss=0.009575, over 3046150.32 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:45:02,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283850
2023-11-22 09:45:10,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1892366.6666666667, ans=0.0
2023-11-22 09:45:11,963 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.07 vs. limit=15.0
2023-11-22 09:45:13,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1892366.6666666667, ans=0.0
2023-11-22 09:45:28,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.13 vs. limit=22.5
2023-11-22 09:45:41,610 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.095e+01 8.850e+01 9.580e+01 1.188e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 09:45:43,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=1892500.0, ans=0.2
2023-11-22 09:45:44,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1892500.0, ans=0.0
2023-11-22 09:45:45,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1892500.0, ans=0.125
2023-11-22 09:45:51,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0
2023-11-22 09:45:55,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1892566.6666666667, ans=0.125
2023-11-22 09:46:02,627 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7350, loss[loss=0.06251, simple_loss=0.08615, pruned_loss=0.01218, audio_tagging_loss=0.007258, over 15097.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09437, pruned_loss=0.01509, audio_tagging_loss=0.009387, over 3044211.86 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:46:06,253 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283900
2023-11-22 09:46:06,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1892633.3333333333, ans=0.2
2023-11-22 09:46:38,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.87 vs. limit=22.5
2023-11-22 09:46:42,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1892833.3333333333, ans=0.1
2023-11-22 09:46:50,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1892833.3333333333, ans=0.09899494936611666
2023-11-22 09:47:07,050 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7400, loss[loss=0.08264, simple_loss=0.1114, pruned_loss=0.01637, audio_tagging_loss=0.01059, over 14610.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09439, pruned_loss=0.0149, audio_tagging_loss=0.009345, over 3042967.27 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:47:10,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 283950
2023-11-22 09:47:20,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1893033.3333333333, ans=0.2
2023-11-22 09:47:41,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1893100.0, ans=0.125
2023-11-22 09:47:51,341 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.493e+01 8.994e+01 9.861e+01 1.121e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 09:48:02,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1893233.3333333333, ans=0.0
2023-11-22 09:48:02,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.98 vs. limit=12.0
2023-11-22 09:48:12,168 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7450, loss[loss=0.0778, simple_loss=0.1007, pruned_loss=0.01729, audio_tagging_loss=0.01016, over 15460.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.09488, pruned_loss=0.01495, audio_tagging_loss=0.009235, over 3048098.63 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:48:14,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1893300.0, ans=0.0
2023-11-22 09:48:15,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284000
2023-11-22 09:48:23,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1893300.0, ans=0.125
2023-11-22 09:48:25,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1893300.0, ans=0.2
2023-11-22 09:48:45,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-22 09:49:02,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1893500.0, ans=0.125
2023-11-22 09:49:10,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.80 vs. limit=6.0
2023-11-22 09:49:17,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1893566.6666666667, ans=0.125
2023-11-22 09:49:19,186 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7500, loss[loss=0.09022, simple_loss=0.1199, pruned_loss=0.02105, audio_tagging_loss=0.009202, over 16725.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09516, pruned_loss=0.01515, audio_tagging_loss=0.0091, over 3043053.91 frames. ], batch size: 62, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:49:19,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1893633.3333333333, ans=0.125
2023-11-22 09:49:22,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284050
2023-11-22 09:50:02,446 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.170e+01 8.839e+01 9.515e+01 1.142e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 09:50:10,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1893900.0, ans=0.125
2023-11-22 09:50:12,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1893900.0, ans=0.0
2023-11-22 09:50:23,607 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7550, loss[loss=0.08058, simple_loss=0.1179, pruned_loss=0.01421, audio_tagging_loss=0.007421, over 15763.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09607, pruned_loss=0.01534, audio_tagging_loss=0.009043, over 3049960.52 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:50:26,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1893966.6666666667, ans=0.2
2023-11-22 09:50:26,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1893966.6666666667, ans=0.125
2023-11-22 09:50:27,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284100
2023-11-22 09:50:30,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1893966.6666666667, ans=0.0
2023-11-22 09:50:37,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1894033.3333333333, ans=0.125
2023-11-22 09:50:39,841 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:50:41,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.29 vs. limit=22.5
2023-11-22 09:50:45,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1894033.3333333333, ans=0.125
2023-11-22 09:50:48,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1894100.0, ans=0.05
2023-11-22 09:50:53,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1894100.0, ans=0.0
2023-11-22 09:50:55,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1894100.0, ans=0.125
2023-11-22 09:51:02,368 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.78 vs. limit=15.0
2023-11-22 09:51:05,342 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.85 vs. limit=22.5
2023-11-22 09:51:23,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1894233.3333333333, ans=0.2
2023-11-22 09:51:28,601 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7600, loss[loss=0.05429, simple_loss=0.06663, pruned_loss=0.008439, audio_tagging_loss=0.01253, over 14619.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09599, pruned_loss=0.01539, audio_tagging_loss=0.009019, over 3046221.91 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:51:32,346 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284150
2023-11-22 09:52:00,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1894433.3333333333, ans=0.125
2023-11-22 09:52:04,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1894433.3333333333, ans=0.0
2023-11-22 09:52:12,158 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.245e+01 8.957e+01 9.580e+01 1.201e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 09:52:24,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1894566.6666666667, ans=0.0
2023-11-22 09:52:27,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1894566.6666666667, ans=0.0
2023-11-22 09:52:29,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0
2023-11-22 09:52:32,248 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7650, loss[loss=0.03676, simple_loss=0.03609, pruned_loss=0.00549, audio_tagging_loss=0.01322, over 14379.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09502, pruned_loss=0.01534, audio_tagging_loss=0.009091, over 3044617.80 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:52:36,141 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284200
2023-11-22 09:52:37,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1894633.3333333333, ans=0.125
2023-11-22 09:53:01,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.23 vs. limit=15.0
2023-11-22 09:53:15,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1894833.3333333333, ans=0.1
2023-11-22 09:53:20,977 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:53:36,915 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7700, loss[loss=0.07198, simple_loss=0.1019, pruned_loss=0.01385, audio_tagging_loss=0.007164, over 14697.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09532, pruned_loss=0.01535, audio_tagging_loss=0.009091, over 3050700.76 frames. ], batch size: 53, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:53:41,310 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284250
2023-11-22 09:53:45,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1894966.6666666667, ans=0.0
2023-11-22 09:54:02,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1895100.0, ans=0.0
2023-11-22 09:54:03,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1895100.0, ans=0.125
2023-11-22 09:54:04,982 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.76 vs. limit=22.5
2023-11-22 09:54:17,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1895166.6666666667, ans=0.125
2023-11-22 09:54:18,863 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.48 vs. limit=15.0
2023-11-22 09:54:20,907 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 7.998e+01 8.702e+01 9.312e+01 1.429e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 09:54:24,005 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.59 vs. limit=15.0
2023-11-22 09:54:24,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1895166.6666666667, ans=0.0
2023-11-22 09:54:35,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1895233.3333333333, ans=0.05
2023-11-22 09:54:41,580 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7750, loss[loss=0.07273, simple_loss=0.09768, pruned_loss=0.01436, audio_tagging_loss=0.009537, over 15701.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09517, pruned_loss=0.01532, audio_tagging_loss=0.009146, over 3047487.90 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:54:45,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284300
2023-11-22 09:54:59,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1895366.6666666667, ans=0.1
2023-11-22 09:55:01,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1895366.6666666667, ans=0.125
2023-11-22 09:55:13,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1895433.3333333333, ans=0.125
2023-11-22 09:55:22,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1895500.0, ans=0.0
2023-11-22 09:55:45,643 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7800, loss[loss=0.05652, simple_loss=0.07441, pruned_loss=0.008794, audio_tagging_loss=0.01052, over 14745.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09473, pruned_loss=0.01529, audio_tagging_loss=0.009114, over 3043302.65 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:55:49,337 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284350
2023-11-22 09:55:54,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1895633.3333333333, ans=0.04949747468305833
2023-11-22 09:56:15,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1895766.6666666667, ans=0.125
2023-11-22 09:56:29,302 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.112e+01 8.791e+01 9.537e+01 1.111e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 09:56:29,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1895833.3333333333, ans=0.1
2023-11-22 09:56:49,585 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7850, loss[loss=0.08665, simple_loss=0.1228, pruned_loss=0.01843, audio_tagging_loss=0.00684, over 15790.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09542, pruned_loss=0.01544, audio_tagging_loss=0.009203, over 3056261.81 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:56:53,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284400
2023-11-22 09:57:08,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1896033.3333333333, ans=0.0
2023-11-22 09:57:09,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.39 vs. limit=15.0
2023-11-22 09:57:37,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1896166.6666666667, ans=0.0
2023-11-22 09:57:52,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1896233.3333333333, ans=0.125
2023-11-22 09:57:55,016 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7900, loss[loss=0.06076, simple_loss=0.07425, pruned_loss=0.0113, audio_tagging_loss=0.01234, over 14416.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09417, pruned_loss=0.01517, audio_tagging_loss=0.009347, over 3049326.53 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:57:59,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284450
2023-11-22 09:58:06,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1896366.6666666667, ans=0.1
2023-11-22 09:58:36,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1896500.0, ans=10.0
2023-11-22 09:58:37,540 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.160e+01 8.848e+01 9.602e+01 1.825e+02, threshold=1.770e+02, percent-clipped=1.0
2023-11-22 09:58:40,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1896500.0, ans=0.07
2023-11-22 09:58:54,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1896566.6666666667, ans=0.125
2023-11-22 09:58:58,638 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 7950, loss[loss=0.07, simple_loss=0.09647, pruned_loss=0.01173, audio_tagging_loss=0.01003, over 14920.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09448, pruned_loss=0.01521, audio_tagging_loss=0.009479, over 3049173.88 frames. ], batch size: 54, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 09:59:02,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284500
2023-11-22 09:59:15,240 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:59:25,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1896766.6666666667, ans=0.0
2023-11-22 09:59:33,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1896766.6666666667, ans=0.125
2023-11-22 09:59:41,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1896833.3333333333, ans=0.125
2023-11-22 09:59:59,462 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.30 vs. limit=15.0
2023-11-22 10:00:02,901 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8000, loss[loss=0.07424, simple_loss=0.09846, pruned_loss=0.01475, audio_tagging_loss=0.01026, over 14638.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09418, pruned_loss=0.01511, audio_tagging_loss=0.009629, over 3051488.62 frames. ], batch size: 53, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:00:06,791 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284550
2023-11-22 10:00:06,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1896966.6666666667, ans=0.0
2023-11-22 10:00:15,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1897033.3333333333, ans=0.125
2023-11-22 10:00:19,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1897033.3333333333, ans=0.0
2023-11-22 10:00:30,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1897100.0, ans=0.125
2023-11-22 10:00:41,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.33 vs. limit=12.0
2023-11-22 10:00:47,709 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.187e+01 8.649e+01 9.156e+01 1.196e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 10:01:05,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.17 vs. limit=15.0
2023-11-22 10:01:06,855 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8050, loss[loss=0.08168, simple_loss=0.1105, pruned_loss=0.01487, audio_tagging_loss=0.01154, over 15306.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.09406, pruned_loss=0.01497, audio_tagging_loss=0.009666, over 3043417.77 frames. ], batch size: 54, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:01:11,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284600
2023-11-22 10:01:59,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1897566.6666666667, ans=0.1
2023-11-22 10:02:02,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1897566.6666666667, ans=0.125
2023-11-22 10:02:08,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1897566.6666666667, ans=0.1
2023-11-22 10:02:12,583 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8100, loss[loss=0.07504, simple_loss=0.09956, pruned_loss=0.0158, audio_tagging_loss=0.009458, over 16066.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09443, pruned_loss=0.01499, audio_tagging_loss=0.009481, over 3047897.79 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:02:16,448 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284650
2023-11-22 10:02:34,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1897700.0, ans=0.2
2023-11-22 10:02:34,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=12.21 vs. limit=15.0
2023-11-22 10:02:59,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.417e+01 8.877e+01 9.599e+01 1.117e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 10:02:59,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1897833.3333333333, ans=0.1
2023-11-22 10:03:04,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.98 vs. limit=15.0
2023-11-22 10:03:16,298 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8150, loss[loss=0.07737, simple_loss=0.09405, pruned_loss=0.01905, audio_tagging_loss=0.0113, over 14554.00 frames. ], tot_loss[loss=0.07153, simple_loss=0.09433, pruned_loss=0.01502, audio_tagging_loss=0.009348, over 3043287.05 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:03:20,578 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284700
2023-11-22 10:03:29,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1898033.3333333333, ans=0.1
2023-11-22 10:03:40,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1898033.3333333333, ans=0.125
2023-11-22 10:03:56,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1898166.6666666667, ans=0.125
2023-11-22 10:03:59,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1898166.6666666667, ans=0.0
2023-11-22 10:04:01,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.00 vs. limit=10.0
2023-11-22 10:04:15,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1898233.3333333333, ans=0.125
2023-11-22 10:04:18,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1898233.3333333333, ans=0.2
2023-11-22 10:04:20,963 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8200, loss[loss=0.06539, simple_loss=0.08735, pruned_loss=0.01105, audio_tagging_loss=0.01067, over 15450.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09396, pruned_loss=0.01484, audio_tagging_loss=0.009334, over 3053879.09 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:04:21,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1898300.0, ans=0.0
2023-11-22 10:04:22,287 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:04:23,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1898300.0, ans=0.1
2023-11-22 10:04:25,333 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284750
2023-11-22 10:04:25,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.35 vs. limit=15.0
2023-11-22 10:04:37,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1898366.6666666667, ans=0.125
2023-11-22 10:04:38,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1898366.6666666667, ans=0.125
2023-11-22 10:05:06,963 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.783e+01 8.106e+01 8.680e+01 9.269e+01 1.329e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-22 10:05:24,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1898633.3333333333, ans=0.2
2023-11-22 10:05:25,274 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8250, loss[loss=0.06935, simple_loss=0.08672, pruned_loss=0.01483, audio_tagging_loss=0.01116, over 15385.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09285, pruned_loss=0.01465, audio_tagging_loss=0.009327, over 3051518.63 frames. ], batch size: 62, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:05:29,058 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284800
2023-11-22 10:05:52,894 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.11 vs. limit=15.0
2023-11-22 10:05:57,696 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0
2023-11-22 10:06:05,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1898833.3333333333, ans=0.125
2023-11-22 10:06:14,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1898833.3333333333, ans=0.125
2023-11-22 10:06:29,558 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8300, loss[loss=0.0635, simple_loss=0.0784, pruned_loss=0.01337, audio_tagging_loss=0.01093, over 14461.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09272, pruned_loss=0.01465, audio_tagging_loss=0.009334, over 3052692.78 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:06:33,255 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284850
2023-11-22 10:06:35,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1898966.6666666667, ans=0.125
2023-11-22 10:06:38,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1898966.6666666667, ans=0.125
2023-11-22 10:06:39,169 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.21 vs. limit=15.0
2023-11-22 10:06:48,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1899033.3333333333, ans=0.0
2023-11-22 10:06:52,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1899033.3333333333, ans=0.125
2023-11-22 10:06:52,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1899033.3333333333, ans=0.2
2023-11-22 10:06:58,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1899100.0, ans=0.0
2023-11-22 10:07:15,158 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-22 10:07:15,708 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.988e+01 8.571e+01 9.748e+01 1.296e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-22 10:07:25,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1899233.3333333333, ans=0.95
2023-11-22 10:07:33,345 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8350, loss[loss=0.07134, simple_loss=0.09094, pruned_loss=0.01758, audio_tagging_loss=0.00829, over 14319.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09315, pruned_loss=0.0149, audio_tagging_loss=0.009207, over 3052180.81 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:07:37,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284900
2023-11-22 10:07:53,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1899366.6666666667, ans=0.0
2023-11-22 10:07:59,286 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:08:19,144 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.29 vs. limit=15.0
2023-11-22 10:08:19,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1899500.0, ans=0.125
2023-11-22 10:08:21,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1899500.0, ans=0.125
2023-11-22 10:08:35,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.54 vs. limit=15.0
2023-11-22 10:08:38,238 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8400, loss[loss=0.09291, simple_loss=0.1225, pruned_loss=0.02222, audio_tagging_loss=0.009425, over 15318.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09364, pruned_loss=0.01487, audio_tagging_loss=0.00918, over 3052382.12 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:08:40,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1899633.3333333333, ans=0.125
2023-11-22 10:08:41,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 284950
2023-11-22 10:08:44,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1899633.3333333333, ans=0.125
2023-11-22 10:09:11,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1899766.6666666667, ans=0.125
2023-11-22 10:09:24,807 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.199e+01 8.877e+01 9.641e+01 1.238e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 10:09:28,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1899900.0, ans=0.125
2023-11-22 10:09:35,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1899900.0, ans=0.125
2023-11-22 10:09:42,472 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8450, loss[loss=0.05452, simple_loss=0.06965, pruned_loss=0.007794, audio_tagging_loss=0.0119, over 14596.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09283, pruned_loss=0.01483, audio_tagging_loss=0.0093, over 3049609.54 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:09:46,284 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285000
2023-11-22 10:09:51,276 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.37 vs. limit=15.0
2023-11-22 10:09:56,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1900033.3333333333, ans=0.0
2023-11-22 10:09:57,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1900033.3333333333, ans=0.125
2023-11-22 10:10:09,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1900100.0, ans=0.05
2023-11-22 10:10:11,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1900100.0, ans=0.0
2023-11-22 10:10:26,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1900166.6666666667, ans=0.0
2023-11-22 10:10:47,748 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8500, loss[loss=0.06026, simple_loss=0.07722, pruned_loss=0.01356, audio_tagging_loss=0.008093, over 13919.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09345, pruned_loss=0.01509, audio_tagging_loss=0.009194, over 3053609.25 frames. ], batch size: 53, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:10:51,594 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285050
2023-11-22 10:10:55,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0
2023-11-22 10:11:23,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1900433.3333333333, ans=0.125
2023-11-22 10:11:32,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff2.min_abs, batch_count=1900500.0, ans=0.1
2023-11-22 10:11:33,986 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.621e+01 8.277e+01 9.051e+01 9.526e+01 1.493e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 10:11:34,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1900500.0, ans=0.1
2023-11-22 10:11:39,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1900566.6666666667, ans=0.125
2023-11-22 10:11:52,687 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8550, loss[loss=0.0601, simple_loss=0.08487, pruned_loss=0.01123, audio_tagging_loss=0.006436, over 14090.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09463, pruned_loss=0.01536, audio_tagging_loss=0.009213, over 3047818.53 frames. ], batch size: 54, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:11:54,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1900633.3333333333, ans=0.125
2023-11-22 10:11:56,413 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285100
2023-11-22 10:11:56,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1900633.3333333333, ans=0.125
2023-11-22 10:11:57,064 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.03 vs. limit=15.0
2023-11-22 10:12:09,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1900700.0, ans=0.1
2023-11-22 10:12:19,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1900766.6666666667, ans=0.125
2023-11-22 10:12:31,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1900833.3333333333, ans=0.0
2023-11-22 10:12:42,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1900900.0, ans=0.035
2023-11-22 10:12:55,962 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8600, loss[loss=0.06292, simple_loss=0.08559, pruned_loss=0.009823, audio_tagging_loss=0.0103, over 15438.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09383, pruned_loss=0.01502, audio_tagging_loss=0.00934, over 3039515.04 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:12:57,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1900966.6666666667, ans=0.0
2023-11-22 10:12:59,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285150
2023-11-22 10:13:41,802 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.761e+01 8.269e+01 8.846e+01 9.645e+01 1.157e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 10:13:43,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=15.0
2023-11-22 10:13:59,983 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8650, loss[loss=0.07074, simple_loss=0.08919, pruned_loss=0.01445, audio_tagging_loss=0.0117, over 15370.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09417, pruned_loss=0.01519, audio_tagging_loss=0.009442, over 3048521.19 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:14:03,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285200
2023-11-22 10:14:10,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1901300.0, ans=0.125
2023-11-22 10:14:18,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1901366.6666666667, ans=0.1
2023-11-22 10:14:29,585 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.46 vs. limit=15.0
2023-11-22 10:14:41,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1901500.0, ans=0.125
2023-11-22 10:14:42,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1901500.0, ans=0.0
2023-11-22 10:15:03,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1901566.6666666667, ans=0.125
2023-11-22 10:15:03,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1901566.6666666667, ans=0.0
2023-11-22 10:15:05,273 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8700, loss[loss=0.07262, simple_loss=0.1018, pruned_loss=0.01375, audio_tagging_loss=0.007971, over 15359.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09494, pruned_loss=0.01535, audio_tagging_loss=0.009453, over 3048867.01 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:15:05,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1901633.3333333333, ans=0.07
2023-11-22 10:15:09,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285250
2023-11-22 10:15:15,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1901633.3333333333, ans=0.0
2023-11-22 10:15:20,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1901700.0, ans=0.0
2023-11-22 10:15:33,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1901766.6666666667, ans=0.0
2023-11-22 10:15:40,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1901766.6666666667, ans=0.125
2023-11-22 10:15:40,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1901766.6666666667, ans=0.125
2023-11-22 10:15:43,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1901833.3333333333, ans=0.1
2023-11-22 10:15:51,786 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.447e+01 8.934e+01 9.629e+01 1.298e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 10:16:01,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1901900.0, ans=0.2
2023-11-22 10:16:09,454 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8750, loss[loss=0.07173, simple_loss=0.09261, pruned_loss=0.01574, audio_tagging_loss=0.009683, over 15858.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09516, pruned_loss=0.01544, audio_tagging_loss=0.009365, over 3050741.69 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:16:09,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1901966.6666666667, ans=0.125
2023-11-22 10:16:13,229 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285300
2023-11-22 10:16:22,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1902033.3333333333, ans=0.2
2023-11-22 10:16:35,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1902100.0, ans=0.125
2023-11-22 10:16:45,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.78 vs. limit=22.5
2023-11-22 10:17:13,270 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8800, loss[loss=0.07773, simple_loss=0.1054, pruned_loss=0.01539, audio_tagging_loss=0.009639, over 15856.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09634, pruned_loss=0.01564, audio_tagging_loss=0.009391, over 3049228.26 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:17:16,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285350
2023-11-22 10:17:33,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1902366.6666666667, ans=0.125
2023-11-22 10:17:49,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1902433.3333333333, ans=0.2
2023-11-22 10:17:49,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1902433.3333333333, ans=0.125
2023-11-22 10:17:59,347 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.363e+01 9.001e+01 9.837e+01 1.230e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 10:18:04,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1902566.6666666667, ans=0.1
2023-11-22 10:18:14,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1902566.6666666667, ans=0.0
2023-11-22 10:18:17,782 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8850, loss[loss=0.08567, simple_loss=0.1191, pruned_loss=0.01704, audio_tagging_loss=0.009087, over 15133.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09552, pruned_loss=0.01551, audio_tagging_loss=0.009505, over 3044955.64 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:18:21,586 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285400
2023-11-22 10:18:25,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1902633.3333333333, ans=0.2
2023-11-22 10:18:25,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1902633.3333333333, ans=0.125
2023-11-22 10:18:28,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1902633.3333333333, ans=0.025
2023-11-22 10:18:32,177 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:18:33,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1902700.0, ans=0.125
2023-11-22 10:18:56,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1902833.3333333333, ans=0.125
2023-11-22 10:19:00,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1902833.3333333333, ans=0.0
2023-11-22 10:19:13,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1902900.0, ans=0.125
2023-11-22 10:19:22,931 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8900, loss[loss=0.078, simple_loss=0.1062, pruned_loss=0.01728, audio_tagging_loss=0.007626, over 14962.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09567, pruned_loss=0.01554, audio_tagging_loss=0.009386, over 3050514.98 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:19:26,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285450
2023-11-22 10:19:31,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1902966.6666666667, ans=0.0
2023-11-22 10:19:46,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1903100.0, ans=0.125
2023-11-22 10:19:51,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1903100.0, ans=0.1
2023-11-22 10:19:53,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1903100.0, ans=0.125
2023-11-22 10:20:04,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1903166.6666666667, ans=0.09899494936611666
2023-11-22 10:20:10,563 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.009e+01 8.748e+01 9.224e+01 1.095e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 10:20:19,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1903233.3333333333, ans=0.1
2023-11-22 10:20:26,849 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 8950, loss[loss=0.08692, simple_loss=0.1187, pruned_loss=0.02133, audio_tagging_loss=0.006237, over 15949.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09459, pruned_loss=0.01541, audio_tagging_loss=0.009292, over 3045991.16 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:20:29,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1903300.0, ans=0.1
2023-11-22 10:20:30,533 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285500
2023-11-22 10:20:48,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.22 vs. limit=22.5
2023-11-22 10:21:07,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.31 vs. limit=15.0
2023-11-22 10:21:16,146 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.40 vs. limit=15.0
2023-11-22 10:21:17,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1903566.6666666667, ans=0.2
2023-11-22 10:21:24,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.25 vs. limit=15.0
2023-11-22 10:21:27,161 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.01 vs. limit=6.0
2023-11-22 10:21:29,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1903633.3333333333, ans=0.125
2023-11-22 10:21:30,146 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9000, loss[loss=0.09495, simple_loss=0.1319, pruned_loss=0.02287, audio_tagging_loss=0.006135, over 14709.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09489, pruned_loss=0.01559, audio_tagging_loss=0.009223, over 3040612.53 frames. ], batch size: 53, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:21:30,147 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 10:21:49,139 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.7934, 4.5645, 4.4307, 4.4981], device='cuda:2')
2023-11-22 10:21:58,966 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9360, 3.7141, 4.8747, 4.4610], device='cuda:2')
2023-11-22 10:22:12,123 INFO [train_asr.py:1253] (2/4) Epoch 24, validation: loss=0.06037, simple_loss=0.05165, pruned_loss=0.00517, audio_tagging_loss=0.02938, over 4681554.00 frames. 
2023-11-22 10:22:12,124 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 10:22:15,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285550
2023-11-22 10:22:17,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1903633.3333333333, ans=0.125
2023-11-22 10:22:34,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1903700.0, ans=0.0
2023-11-22 10:22:52,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1903833.3333333333, ans=0.0
2023-11-22 10:22:59,082 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.344e+01 8.279e+01 8.807e+01 9.755e+01 1.198e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 10:23:12,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1903900.0, ans=0.125
2023-11-22 10:23:15,770 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9050, loss[loss=0.0813, simple_loss=0.1048, pruned_loss=0.02117, audio_tagging_loss=0.007736, over 15442.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09468, pruned_loss=0.01553, audio_tagging_loss=0.009218, over 3048841.41 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:23:19,505 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285600
2023-11-22 10:23:26,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1903966.6666666667, ans=0.125
2023-11-22 10:23:48,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1904100.0, ans=0.1
2023-11-22 10:24:20,409 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9100, loss[loss=0.06055, simple_loss=0.07697, pruned_loss=0.01017, audio_tagging_loss=0.01189, over 16456.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09344, pruned_loss=0.01519, audio_tagging_loss=0.009281, over 3047098.16 frames. ], batch size: 63, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:24:24,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285650
2023-11-22 10:24:24,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1904300.0, ans=0.125
2023-11-22 10:24:29,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1904300.0, ans=0.1
2023-11-22 10:24:38,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.01 vs. limit=15.0
2023-11-22 10:24:51,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1904433.3333333333, ans=0.125
2023-11-22 10:24:53,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1904433.3333333333, ans=0.2
2023-11-22 10:24:56,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1904433.3333333333, ans=0.1
2023-11-22 10:25:03,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1904500.0, ans=0.1
2023-11-22 10:25:07,535 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.735e+01 8.029e+01 8.849e+01 9.580e+01 1.187e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 10:25:24,632 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9150, loss[loss=0.09473, simple_loss=0.121, pruned_loss=0.02625, audio_tagging_loss=0.007966, over 15718.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09454, pruned_loss=0.0154, audio_tagging_loss=0.009249, over 3045957.81 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:25:28,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285700
2023-11-22 10:25:47,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1904700.0, ans=0.125
2023-11-22 10:25:51,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1904766.6666666667, ans=0.125
2023-11-22 10:26:28,549 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9200, loss[loss=0.08123, simple_loss=0.11, pruned_loss=0.01697, audio_tagging_loss=0.009241, over 15780.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09395, pruned_loss=0.01521, audio_tagging_loss=0.00924, over 3043836.58 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:26:28,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1904966.6666666667, ans=0.1
2023-11-22 10:26:29,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0
2023-11-22 10:26:32,205 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285750
2023-11-22 10:26:32,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1904966.6666666667, ans=0.125
2023-11-22 10:26:39,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1905033.3333333333, ans=0.125
2023-11-22 10:26:42,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1905033.3333333333, ans=0.04949747468305833
2023-11-22 10:27:15,509 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=12.0
2023-11-22 10:27:15,867 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.192e+01 8.757e+01 9.519e+01 1.252e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 10:27:23,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.14 vs. limit=12.0
2023-11-22 10:27:32,498 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9250, loss[loss=0.07591, simple_loss=0.105, pruned_loss=0.01804, audio_tagging_loss=0.005366, over 15181.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09263, pruned_loss=0.01505, audio_tagging_loss=0.009296, over 3050171.01 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:27:32,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1905300.0, ans=0.0
2023-11-22 10:27:36,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285800
2023-11-22 10:27:40,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1905300.0, ans=0.125
2023-11-22 10:28:01,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1905433.3333333333, ans=0.125
2023-11-22 10:28:37,331 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9300, loss[loss=0.04476, simple_loss=0.04904, pruned_loss=0.007167, audio_tagging_loss=0.01307, over 15354.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09256, pruned_loss=0.01503, audio_tagging_loss=0.009354, over 3056629.33 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:28:38,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.89 vs. limit=15.0
2023-11-22 10:28:41,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285850
2023-11-22 10:28:48,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1905700.0, ans=0.0
2023-11-22 10:28:57,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1905700.0, ans=0.0
2023-11-22 10:28:58,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1905700.0, ans=0.125
2023-11-22 10:29:05,783 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0
2023-11-22 10:29:16,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1905833.3333333333, ans=0.125
2023-11-22 10:29:20,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1905833.3333333333, ans=0.125
2023-11-22 10:29:25,422 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 8.386e+01 8.892e+01 9.597e+01 1.135e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 10:29:28,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1905900.0, ans=0.5
2023-11-22 10:29:41,930 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9350, loss[loss=0.09201, simple_loss=0.1218, pruned_loss=0.02377, audio_tagging_loss=0.007332, over 14967.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09276, pruned_loss=0.01494, audio_tagging_loss=0.009253, over 3054443.71 frames. ], batch size: 54, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:29:45,659 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285900
2023-11-22 10:29:49,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1905966.6666666667, ans=0.0
2023-11-22 10:29:58,621 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.04 vs. limit=15.0
2023-11-22 10:30:45,406 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9400, loss[loss=0.08741, simple_loss=0.1254, pruned_loss=0.01775, audio_tagging_loss=0.006975, over 16621.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09338, pruned_loss=0.01502, audio_tagging_loss=0.009238, over 3050241.25 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:30:48,678 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:30:49,773 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 285950
2023-11-22 10:31:32,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.313e+01 8.349e+01 8.920e+01 9.542e+01 1.196e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 10:31:48,505 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:31:49,664 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9450, loss[loss=0.06335, simple_loss=0.08049, pruned_loss=0.01242, audio_tagging_loss=0.01069, over 14823.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09413, pruned_loss=0.01512, audio_tagging_loss=0.009303, over 3051457.22 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:31:54,011 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286000
2023-11-22 10:32:26,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1906766.6666666667, ans=0.0
2023-11-22 10:32:27,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1906833.3333333333, ans=0.125
2023-11-22 10:32:46,794 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.18 vs. limit=15.0
2023-11-22 10:32:54,478 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9500, loss[loss=0.07434, simple_loss=0.09614, pruned_loss=0.01716, audio_tagging_loss=0.009108, over 15814.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09444, pruned_loss=0.01527, audio_tagging_loss=0.009396, over 3053752.67 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:32:58,863 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286050
2023-11-22 10:32:59,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1906966.6666666667, ans=0.125
2023-11-22 10:33:07,906 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.27 vs. limit=12.0
2023-11-22 10:33:19,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=1907100.0, ans=0.05
2023-11-22 10:33:29,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1907100.0, ans=0.125
2023-11-22 10:33:34,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1907166.6666666667, ans=0.125
2023-11-22 10:33:42,361 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.223e+01 8.226e+01 8.759e+01 9.603e+01 1.524e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 10:33:42,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1907166.6666666667, ans=0.95
2023-11-22 10:33:45,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1907233.3333333333, ans=0.0
2023-11-22 10:33:58,985 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9550, loss[loss=0.08035, simple_loss=0.1043, pruned_loss=0.01772, audio_tagging_loss=0.01046, over 16645.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09376, pruned_loss=0.01516, audio_tagging_loss=0.009414, over 3056124.51 frames. ], batch size: 64, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:34:02,721 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286100
2023-11-22 10:34:30,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1907433.3333333333, ans=0.125
2023-11-22 10:35:04,203 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9600, loss[loss=0.05625, simple_loss=0.06336, pruned_loss=0.008289, audio_tagging_loss=0.01628, over 14422.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09296, pruned_loss=0.01503, audio_tagging_loss=0.009604, over 3056928.22 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:35:07,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286150
2023-11-22 10:35:11,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1907633.3333333333, ans=0.125
2023-11-22 10:35:31,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1907766.6666666667, ans=0.0
2023-11-22 10:35:39,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.74 vs. limit=10.0
2023-11-22 10:35:47,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1907833.3333333333, ans=0.0
2023-11-22 10:35:51,441 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.074e+01 8.745e+01 9.476e+01 1.191e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 10:35:56,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1907900.0, ans=0.125
2023-11-22 10:36:00,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1907900.0, ans=0.125
2023-11-22 10:36:07,697 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9650, loss[loss=0.07648, simple_loss=0.1074, pruned_loss=0.01599, audio_tagging_loss=0.006791, over 14907.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09348, pruned_loss=0.015, audio_tagging_loss=0.009543, over 3054713.97 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:36:12,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286200
2023-11-22 10:36:25,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1908033.3333333333, ans=0.0
2023-11-22 10:37:00,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1908233.3333333333, ans=0.125
2023-11-22 10:37:01,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=22.5
2023-11-22 10:37:06,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1908233.3333333333, ans=0.0
2023-11-22 10:37:12,274 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9700, loss[loss=0.08146, simple_loss=0.1025, pruned_loss=0.02126, audio_tagging_loss=0.008971, over 14889.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09397, pruned_loss=0.01519, audio_tagging_loss=0.009418, over 3046550.84 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:37:13,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1908300.0, ans=0.125
2023-11-22 10:37:16,054 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286250
2023-11-22 10:37:32,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1908366.6666666667, ans=0.1
2023-11-22 10:37:48,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1908433.3333333333, ans=0.125
2023-11-22 10:37:59,364 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.165e+01 7.968e+01 8.645e+01 9.511e+01 1.252e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 10:38:04,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1908566.6666666667, ans=0.0
2023-11-22 10:38:16,142 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9750, loss[loss=0.06607, simple_loss=0.08104, pruned_loss=0.01365, audio_tagging_loss=0.01191, over 15078.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09421, pruned_loss=0.01524, audio_tagging_loss=0.009329, over 3046911.44 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:38:20,572 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286300
2023-11-22 10:38:20,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1908633.3333333333, ans=0.5
2023-11-22 10:38:26,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1908633.3333333333, ans=0.125
2023-11-22 10:38:28,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.84 vs. limit=15.0
2023-11-22 10:38:30,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1908700.0, ans=0.0
2023-11-22 10:38:51,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1908766.6666666667, ans=0.125
2023-11-22 10:38:59,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1908833.3333333333, ans=0.125
2023-11-22 10:39:01,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1908833.3333333333, ans=0.125
2023-11-22 10:39:17,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1908900.0, ans=0.1
2023-11-22 10:39:20,635 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9800, loss[loss=0.06109, simple_loss=0.07784, pruned_loss=0.01409, audio_tagging_loss=0.008081, over 14934.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.09362, pruned_loss=0.01537, audio_tagging_loss=0.009446, over 3044539.70 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:39:24,324 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286350
2023-11-22 10:39:36,787 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:39:36,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1909033.3333333333, ans=0.125
2023-11-22 10:40:08,400 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.345e+01 8.979e+01 9.748e+01 1.178e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-22 10:40:15,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1909233.3333333333, ans=0.95
2023-11-22 10:40:18,746 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:40:25,478 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9850, loss[loss=0.07751, simple_loss=0.1071, pruned_loss=0.01413, audio_tagging_loss=0.009836, over 14784.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09402, pruned_loss=0.0153, audio_tagging_loss=0.009341, over 3045685.83 frames. ], batch size: 54, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:40:29,226 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286400
2023-11-22 10:40:50,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1909433.3333333333, ans=0.125
2023-11-22 10:40:57,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1909433.3333333333, ans=0.04949747468305833
2023-11-22 10:40:59,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.29 vs. limit=12.0
2023-11-22 10:41:00,982 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=12.0
2023-11-22 10:41:04,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1909500.0, ans=0.125
2023-11-22 10:41:07,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1909500.0, ans=0.0
2023-11-22 10:41:17,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1909566.6666666667, ans=0.125
2023-11-22 10:41:20,110 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:41:20,655 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.43 vs. limit=15.0
2023-11-22 10:41:30,099 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9900, loss[loss=0.09694, simple_loss=0.1363, pruned_loss=0.02368, audio_tagging_loss=0.005089, over 14809.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09422, pruned_loss=0.01537, audio_tagging_loss=0.00931, over 3043194.09 frames. ], batch size: 52, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:41:34,467 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286450
2023-11-22 10:41:43,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1909700.0, ans=0.2
2023-11-22 10:41:56,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1909766.6666666667, ans=0.125
2023-11-22 10:42:01,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1909766.6666666667, ans=0.125
2023-11-22 10:42:09,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1909833.3333333333, ans=0.0
2023-11-22 10:42:19,147 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.212e+01 8.837e+01 9.450e+01 1.215e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 10:42:20,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1909900.0, ans=0.125
2023-11-22 10:42:23,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1909900.0, ans=0.125
2023-11-22 10:42:34,522 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 9950, loss[loss=0.05197, simple_loss=0.06793, pruned_loss=0.008562, audio_tagging_loss=0.009447, over 14783.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09425, pruned_loss=0.01532, audio_tagging_loss=0.00926, over 3045698.69 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:42:38,301 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286500
2023-11-22 10:43:04,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1910100.0, ans=0.5
2023-11-22 10:43:08,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=22.78 vs. limit=15.0
2023-11-22 10:43:39,040 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10000, loss[loss=0.05412, simple_loss=0.07276, pruned_loss=0.009972, audio_tagging_loss=0.007767, over 14747.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.0942, pruned_loss=0.01522, audio_tagging_loss=0.009221, over 3043615.65 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:43:40,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1910300.0, ans=0.125
2023-11-22 10:43:42,861 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286550
2023-11-22 10:44:01,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.34 vs. limit=15.0
2023-11-22 10:44:08,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1910433.3333333333, ans=0.0
2023-11-22 10:44:26,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1910500.0, ans=0.07
2023-11-22 10:44:27,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.515e+01 8.076e+01 8.734e+01 9.444e+01 1.191e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 10:44:35,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1910566.6666666667, ans=0.125
2023-11-22 10:44:43,780 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10050, loss[loss=0.06829, simple_loss=0.09194, pruned_loss=0.0152, audio_tagging_loss=0.007122, over 15004.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09346, pruned_loss=0.01494, audio_tagging_loss=0.009192, over 3043051.24 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:44:47,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286600
2023-11-22 10:45:05,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1910700.0, ans=0.2
2023-11-22 10:45:48,197 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10100, loss[loss=0.09356, simple_loss=0.114, pruned_loss=0.02664, audio_tagging_loss=0.009917, over 15239.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.09441, pruned_loss=0.01526, audio_tagging_loss=0.009182, over 3047766.65 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:45:52,057 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286650
2023-11-22 10:46:32,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1911166.6666666667, ans=0.125
2023-11-22 10:46:37,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1911166.6666666667, ans=0.125
2023-11-22 10:46:38,285 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.221e+01 8.968e+01 9.727e+01 1.162e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 10:46:40,872 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:46:41,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1911233.3333333333, ans=0.0
2023-11-22 10:46:42,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1911233.3333333333, ans=0.0
2023-11-22 10:46:52,451 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10150, loss[loss=0.08877, simple_loss=0.1176, pruned_loss=0.02039, audio_tagging_loss=0.009559, over 15319.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.0945, pruned_loss=0.01532, audio_tagging_loss=0.009353, over 3041207.72 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:46:56,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286700
2023-11-22 10:47:23,817 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:47:43,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1911566.6666666667, ans=0.125
2023-11-22 10:47:44,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.68 vs. limit=15.0
2023-11-22 10:47:56,798 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10200, loss[loss=0.06934, simple_loss=0.07703, pruned_loss=0.01734, audio_tagging_loss=0.01349, over 15277.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09534, pruned_loss=0.01559, audio_tagging_loss=0.009469, over 3050024.08 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:48:00,545 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286750
2023-11-22 10:48:00,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1911633.3333333333, ans=0.125
2023-11-22 10:48:11,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1911700.0, ans=0.125
2023-11-22 10:48:11,384 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.58 vs. limit=22.5
2023-11-22 10:48:13,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1911700.0, ans=0.0
2023-11-22 10:48:20,844 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:48:38,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1911833.3333333333, ans=0.125
2023-11-22 10:48:46,801 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.248e+01 8.751e+01 9.441e+01 1.217e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 10:49:01,123 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10250, loss[loss=0.08684, simple_loss=0.1272, pruned_loss=0.01665, audio_tagging_loss=0.006587, over 15971.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09503, pruned_loss=0.01546, audio_tagging_loss=0.009431, over 3041940.76 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:49:04,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286800
2023-11-22 10:49:15,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.53 vs. limit=15.0
2023-11-22 10:49:17,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1912033.3333333333, ans=0.125
2023-11-22 10:49:38,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1912100.0, ans=0.125
2023-11-22 10:49:42,331 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.97 vs. limit=22.5
2023-11-22 10:49:56,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1912233.3333333333, ans=0.0
2023-11-22 10:50:02,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1912233.3333333333, ans=0.0
2023-11-22 10:50:05,639 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10300, loss[loss=0.05907, simple_loss=0.07825, pruned_loss=0.009633, audio_tagging_loss=0.01032, over 14823.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09511, pruned_loss=0.01549, audio_tagging_loss=0.009429, over 3040535.96 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:50:09,982 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286850
2023-11-22 10:50:11,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1912300.0, ans=0.2
2023-11-22 10:50:22,688 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.47 vs. limit=15.0
2023-11-22 10:50:29,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1912366.6666666667, ans=0.125
2023-11-22 10:50:34,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1912433.3333333333, ans=0.125
2023-11-22 10:50:46,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1912500.0, ans=0.125
2023-11-22 10:50:48,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1912500.0, ans=0.125
2023-11-22 10:50:51,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1912500.0, ans=0.125
2023-11-22 10:50:56,529 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.119e+01 8.625e+01 9.295e+01 1.228e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 10:50:58,033 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:51:09,894 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10350, loss[loss=0.07385, simple_loss=0.084, pruned_loss=0.01821, audio_tagging_loss=0.01365, over 14973.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.0952, pruned_loss=0.01544, audio_tagging_loss=0.009511, over 3040554.56 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:51:14,280 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286900
2023-11-22 10:51:23,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1912700.0, ans=0.125
2023-11-22 10:51:42,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1912766.6666666667, ans=0.125
2023-11-22 10:51:51,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1912833.3333333333, ans=0.125
2023-11-22 10:52:06,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1912900.0, ans=0.0
2023-11-22 10:52:06,488 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.90 vs. limit=22.5
2023-11-22 10:52:13,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1912900.0, ans=0.2
2023-11-22 10:52:15,704 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10400, loss[loss=0.06945, simple_loss=0.08415, pruned_loss=0.01493, audio_tagging_loss=0.01244, over 16253.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09524, pruned_loss=0.01546, audio_tagging_loss=0.009533, over 3032939.24 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:52:15,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1912966.6666666667, ans=0.2
2023-11-22 10:52:19,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 286950
2023-11-22 10:52:42,409 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.65 vs. limit=22.5
2023-11-22 10:52:47,100 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.23 vs. limit=15.0
2023-11-22 10:52:48,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1913100.0, ans=0.125
2023-11-22 10:53:05,479 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.131e+01 8.865e+01 9.715e+01 1.299e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 10:53:07,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1913233.3333333333, ans=0.125
2023-11-22 10:53:18,877 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10450, loss[loss=0.0952, simple_loss=0.1261, pruned_loss=0.02369, audio_tagging_loss=0.008481, over 15133.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09458, pruned_loss=0.01536, audio_tagging_loss=0.009528, over 3037152.16 frames. ], batch size: 54, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:53:23,131 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287000
2023-11-22 10:53:23,784 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.44 vs. limit=15.0
2023-11-22 10:53:42,662 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:54:05,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1913500.0, ans=0.125
2023-11-22 10:54:14,324 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.39 vs. limit=15.0
2023-11-22 10:54:22,329 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10500, loss[loss=0.07937, simple_loss=0.1027, pruned_loss=0.01913, audio_tagging_loss=0.008869, over 14975.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09451, pruned_loss=0.01547, audio_tagging_loss=0.009441, over 3031481.27 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:54:26,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287050
2023-11-22 10:54:26,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1913633.3333333333, ans=0.0
2023-11-22 10:54:44,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1913700.0, ans=0.1
2023-11-22 10:54:45,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1913700.0, ans=0.05
2023-11-22 10:55:08,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1913833.3333333333, ans=0.125
2023-11-22 10:55:11,485 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.41 vs. limit=15.0
2023-11-22 10:55:12,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1913833.3333333333, ans=0.125
2023-11-22 10:55:13,013 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.123e+01 8.688e+01 9.437e+01 1.274e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 10:55:17,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1913900.0, ans=0.125
2023-11-22 10:55:17,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1913900.0, ans=0.0
2023-11-22 10:55:21,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1913900.0, ans=0.125
2023-11-22 10:55:28,273 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10550, loss[loss=0.05769, simple_loss=0.06876, pruned_loss=0.0107, audio_tagging_loss=0.0126, over 14249.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09378, pruned_loss=0.01523, audio_tagging_loss=0.009325, over 3033159.66 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:55:32,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287100
2023-11-22 10:55:33,243 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.74 vs. limit=15.0
2023-11-22 10:55:42,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1914033.3333333333, ans=0.5
2023-11-22 10:56:00,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1914100.0, ans=0.1
2023-11-22 10:56:17,215 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0
2023-11-22 10:56:19,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1914233.3333333333, ans=0.2
2023-11-22 10:56:22,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1914233.3333333333, ans=0.2
2023-11-22 10:56:33,347 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10600, loss[loss=0.07702, simple_loss=0.1149, pruned_loss=0.01297, audio_tagging_loss=0.006596, over 15099.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09368, pruned_loss=0.0152, audio_tagging_loss=0.00929, over 3030529.96 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:56:37,230 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287150
2023-11-22 10:56:37,732 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.70 vs. limit=12.0
2023-11-22 10:57:01,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=15.0
2023-11-22 10:57:07,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1914433.3333333333, ans=0.125
2023-11-22 10:57:23,282 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 8.242e+01 9.024e+01 9.598e+01 1.340e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-22 10:57:25,046 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:57:31,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1914566.6666666667, ans=0.05
2023-11-22 10:57:31,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1914566.6666666667, ans=0.125
2023-11-22 10:57:36,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1914633.3333333333, ans=0.125
2023-11-22 10:57:37,646 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10650, loss[loss=0.08566, simple_loss=0.1121, pruned_loss=0.02031, audio_tagging_loss=0.009307, over 16224.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09397, pruned_loss=0.01519, audio_tagging_loss=0.009177, over 3031854.64 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:57:41,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287200
2023-11-22 10:57:47,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1914633.3333333333, ans=0.1
2023-11-22 10:57:55,209 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.62 vs. limit=22.5
2023-11-22 10:58:03,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1914766.6666666667, ans=0.125
2023-11-22 10:58:06,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1914766.6666666667, ans=6.0
2023-11-22 10:58:22,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1914833.3333333333, ans=0.1
2023-11-22 10:58:26,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1914833.3333333333, ans=0.1
2023-11-22 10:58:42,756 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10700, loss[loss=0.06378, simple_loss=0.07169, pruned_loss=0.01655, audio_tagging_loss=0.01139, over 14587.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09401, pruned_loss=0.0151, audio_tagging_loss=0.009322, over 3028587.91 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:58:47,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287250
2023-11-22 10:59:21,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.00 vs. limit=22.5
2023-11-22 10:59:23,214 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.34 vs. limit=22.5
2023-11-22 10:59:28,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1915166.6666666667, ans=0.1
2023-11-22 10:59:33,546 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.175e+01 8.915e+01 9.385e+01 1.445e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-22 10:59:36,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1915233.3333333333, ans=0.1
2023-11-22 10:59:39,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1915233.3333333333, ans=0.125
2023-11-22 10:59:47,567 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10750, loss[loss=0.07365, simple_loss=0.1027, pruned_loss=0.01589, audio_tagging_loss=0.006437, over 16265.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09424, pruned_loss=0.01508, audio_tagging_loss=0.009279, over 3028359.78 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:59:51,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287300
2023-11-22 11:00:25,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1915500.0, ans=0.0
2023-11-22 11:00:46,480 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.55 vs. limit=6.0
2023-11-22 11:00:47,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1915566.6666666667, ans=0.125
2023-11-22 11:00:51,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.34 vs. limit=15.0
2023-11-22 11:00:51,827 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10800, loss[loss=0.07105, simple_loss=0.1011, pruned_loss=0.01165, audio_tagging_loss=0.00885, over 15638.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09381, pruned_loss=0.01497, audio_tagging_loss=0.009189, over 3039581.13 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:00:55,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287350
2023-11-22 11:01:02,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1915633.3333333333, ans=0.125
2023-11-22 11:01:21,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1915766.6666666667, ans=0.1
2023-11-22 11:01:34,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1915833.3333333333, ans=0.125
2023-11-22 11:01:40,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1915833.3333333333, ans=0.125
2023-11-22 11:01:42,413 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.103e+01 8.562e+01 9.437e+01 1.159e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-22 11:01:43,237 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.23 vs. limit=22.5
2023-11-22 11:01:56,480 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10850, loss[loss=0.09473, simple_loss=0.1257, pruned_loss=0.0237, audio_tagging_loss=0.008193, over 14886.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09337, pruned_loss=0.01495, audio_tagging_loss=0.009324, over 3039128.72 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:01:59,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1915966.6666666667, ans=0.125
2023-11-22 11:02:00,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287400
2023-11-22 11:02:15,381 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0
2023-11-22 11:02:30,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1916100.0, ans=0.1
2023-11-22 11:02:36,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1916166.6666666667, ans=0.125
2023-11-22 11:02:50,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1916233.3333333333, ans=0.0
2023-11-22 11:02:57,555 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:03:01,203 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10900, loss[loss=0.08635, simple_loss=0.1096, pruned_loss=0.02156, audio_tagging_loss=0.009986, over 14730.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09411, pruned_loss=0.01517, audio_tagging_loss=0.009349, over 3042355.66 frames. ], batch size: 53, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:03:04,958 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287450
2023-11-22 11:03:21,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1916366.6666666667, ans=0.0
2023-11-22 11:03:38,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1916433.3333333333, ans=0.125
2023-11-22 11:03:52,819 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.229e+01 8.892e+01 9.728e+01 1.443e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 11:03:54,621 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0
2023-11-22 11:04:06,238 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 10950, loss[loss=0.06418, simple_loss=0.07981, pruned_loss=0.01554, audio_tagging_loss=0.008738, over 14148.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.0949, pruned_loss=0.01523, audio_tagging_loss=0.009344, over 3037054.69 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:04:10,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287500
2023-11-22 11:04:17,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1916700.0, ans=0.125
2023-11-22 11:04:31,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1916766.6666666667, ans=0.0
2023-11-22 11:04:45,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1916833.3333333333, ans=0.0
2023-11-22 11:05:10,037 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11000, loss[loss=0.06228, simple_loss=0.07704, pruned_loss=0.01471, audio_tagging_loss=0.009044, over 15376.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.09452, pruned_loss=0.01504, audio_tagging_loss=0.009371, over 3038830.24 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:05:11,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1916966.6666666667, ans=0.0
2023-11-22 11:05:13,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287550
2023-11-22 11:05:17,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1916966.6666666667, ans=0.0
2023-11-22 11:05:17,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1916966.6666666667, ans=0.05
2023-11-22 11:05:21,791 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:05:24,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1917033.3333333333, ans=0.1
2023-11-22 11:05:32,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1917033.3333333333, ans=0.1
2023-11-22 11:05:34,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1917100.0, ans=0.125
2023-11-22 11:06:01,244 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.398e+01 8.039e+01 8.665e+01 9.378e+01 1.568e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 11:06:03,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1917233.3333333333, ans=0.125
2023-11-22 11:06:11,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1917233.3333333333, ans=0.125
2023-11-22 11:06:14,116 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11050, loss[loss=0.07134, simple_loss=0.09557, pruned_loss=0.01407, audio_tagging_loss=0.009488, over 14315.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09406, pruned_loss=0.01506, audio_tagging_loss=0.009521, over 3045553.13 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:06:14,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1917300.0, ans=0.0
2023-11-22 11:06:17,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287600
2023-11-22 11:06:20,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1917300.0, ans=0.1
2023-11-22 11:06:58,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1917500.0, ans=0.0
2023-11-22 11:06:59,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1917500.0, ans=0.125
2023-11-22 11:07:04,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1917566.6666666667, ans=0.125
2023-11-22 11:07:10,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1917566.6666666667, ans=0.125
2023-11-22 11:07:18,038 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11100, loss[loss=0.08794, simple_loss=0.1169, pruned_loss=0.02072, audio_tagging_loss=0.008753, over 16072.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09369, pruned_loss=0.01494, audio_tagging_loss=0.00964, over 3045530.83 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:07:22,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287650
2023-11-22 11:07:25,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.28 vs. limit=15.0
2023-11-22 11:07:46,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1917766.6666666667, ans=0.0
2023-11-22 11:07:49,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1917766.6666666667, ans=0.1
2023-11-22 11:07:49,776 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.76 vs. limit=15.0
2023-11-22 11:07:51,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1917766.6666666667, ans=0.0
2023-11-22 11:08:09,638 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.222e+01 9.043e+01 9.663e+01 1.186e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 11:08:21,647 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.48 vs. limit=12.0
2023-11-22 11:08:22,596 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11150, loss[loss=0.06704, simple_loss=0.08947, pruned_loss=0.01332, audio_tagging_loss=0.008987, over 14557.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09391, pruned_loss=0.01507, audio_tagging_loss=0.0097, over 3048191.84 frames. ], batch size: 53, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:08:26,280 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287700
2023-11-22 11:08:30,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1917966.6666666667, ans=0.0
2023-11-22 11:08:57,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1918100.0, ans=0.2
2023-11-22 11:09:01,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1918166.6666666667, ans=0.2
2023-11-22 11:09:02,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1918166.6666666667, ans=0.09899494936611666
2023-11-22 11:09:09,080 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.16 vs. limit=6.0
2023-11-22 11:09:26,696 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11200, loss[loss=0.06223, simple_loss=0.07862, pruned_loss=0.01356, audio_tagging_loss=0.009351, over 15593.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09303, pruned_loss=0.01486, audio_tagging_loss=0.009802, over 3056171.63 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:09:30,511 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287750
2023-11-22 11:09:42,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1918366.6666666667, ans=0.0
2023-11-22 11:10:01,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1918433.3333333333, ans=0.125
2023-11-22 11:10:17,758 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.119e+01 8.627e+01 9.513e+01 1.631e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 11:10:24,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1918566.6666666667, ans=0.5
2023-11-22 11:10:26,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1918566.6666666667, ans=0.0
2023-11-22 11:10:30,709 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11250, loss[loss=0.08032, simple_loss=0.1057, pruned_loss=0.01913, audio_tagging_loss=0.008341, over 16363.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09298, pruned_loss=0.01481, audio_tagging_loss=0.009795, over 3050514.68 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:10:34,209 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.63 vs. limit=15.0
2023-11-22 11:10:34,646 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287800
2023-11-22 11:10:46,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.43 vs. limit=15.0
2023-11-22 11:10:46,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1918700.0, ans=0.1
2023-11-22 11:11:13,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1918833.3333333333, ans=0.125
2023-11-22 11:11:17,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1918833.3333333333, ans=0.1
2023-11-22 11:11:35,610 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11300, loss[loss=0.07387, simple_loss=0.09277, pruned_loss=0.01839, audio_tagging_loss=0.009088, over 14540.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09387, pruned_loss=0.01492, audio_tagging_loss=0.009659, over 3053388.40 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:11:36,108 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.08 vs. limit=15.0
2023-11-22 11:11:39,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287850
2023-11-22 11:12:19,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1919166.6666666667, ans=0.125
2023-11-22 11:12:28,497 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.163e+01 8.307e+01 9.060e+01 9.651e+01 1.397e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-22 11:12:36,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1919233.3333333333, ans=0.2
2023-11-22 11:12:38,410 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.82 vs. limit=15.0
2023-11-22 11:12:39,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1919300.0, ans=0.0
2023-11-22 11:12:40,202 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11350, loss[loss=0.06489, simple_loss=0.08764, pruned_loss=0.01193, audio_tagging_loss=0.009137, over 15044.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.0941, pruned_loss=0.01498, audio_tagging_loss=0.009512, over 3051192.98 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:12:43,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1919300.0, ans=0.0
2023-11-22 11:12:44,141 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287900
2023-11-22 11:12:49,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1919300.0, ans=0.0
2023-11-22 11:13:20,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1919500.0, ans=0.125
2023-11-22 11:13:43,679 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11400, loss[loss=0.05629, simple_loss=0.06819, pruned_loss=0.00861, audio_tagging_loss=0.01359, over 15240.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09385, pruned_loss=0.01507, audio_tagging_loss=0.009464, over 3052120.19 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:13:43,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1919633.3333333333, ans=0.025
2023-11-22 11:13:47,958 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 287950
2023-11-22 11:13:49,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.40 vs. limit=12.0
2023-11-22 11:13:53,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1919633.3333333333, ans=0.125
2023-11-22 11:14:07,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1919700.0, ans=0.2
2023-11-22 11:14:14,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1919766.6666666667, ans=0.0
2023-11-22 11:14:25,503 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.33 vs. limit=6.0
2023-11-22 11:14:26,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1919833.3333333333, ans=0.1
2023-11-22 11:14:36,227 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.311e+01 8.086e+01 8.647e+01 9.538e+01 1.200e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 11:14:39,293 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.81 vs. limit=15.0
2023-11-22 11:14:43,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1919900.0, ans=0.0
2023-11-22 11:14:47,056 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11450, loss[loss=0.06775, simple_loss=0.1001, pruned_loss=0.01029, audio_tagging_loss=0.007434, over 14920.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.0944, pruned_loss=0.01517, audio_tagging_loss=0.009379, over 3047924.53 frames. ], batch size: 53, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:14:47,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1919966.6666666667, ans=0.1
2023-11-22 11:14:50,781 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288000
2023-11-22 11:15:07,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.88 vs. limit=15.0
2023-11-22 11:15:13,898 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=5.62 vs. limit=15.0
2023-11-22 11:15:17,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1920100.0, ans=0.125
2023-11-22 11:15:31,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1920166.6666666667, ans=0.125
2023-11-22 11:15:48,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1920233.3333333333, ans=0.5
2023-11-22 11:15:52,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1920233.3333333333, ans=0.125
2023-11-22 11:15:54,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1920300.0, ans=0.0
2023-11-22 11:15:55,751 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11500, loss[loss=0.06618, simple_loss=0.09107, pruned_loss=0.009817, audio_tagging_loss=0.01082, over 15885.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09495, pruned_loss=0.01518, audio_tagging_loss=0.009361, over 3055938.77 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:15:58,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1920300.0, ans=0.125
2023-11-22 11:15:59,506 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288050
2023-11-22 11:16:04,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1920300.0, ans=0.125
2023-11-22 11:16:07,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1920366.6666666667, ans=0.125
2023-11-22 11:16:07,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1920366.6666666667, ans=0.025
2023-11-22 11:16:16,378 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.15 vs. limit=22.5
2023-11-22 11:16:44,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1920500.0, ans=0.125
2023-11-22 11:16:47,947 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.254e+01 8.955e+01 9.456e+01 2.559e+02, threshold=1.791e+02, percent-clipped=1.0
2023-11-22 11:16:55,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.57 vs. limit=15.0
2023-11-22 11:16:59,007 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11550, loss[loss=0.0632, simple_loss=0.08293, pruned_loss=0.01058, audio_tagging_loss=0.01116, over 14263.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09434, pruned_loss=0.01512, audio_tagging_loss=0.009328, over 3047744.07 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:16:59,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-22 11:17:02,682 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288100
2023-11-22 11:17:15,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1920700.0, ans=0.125
2023-11-22 11:17:24,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1920766.6666666667, ans=0.0
2023-11-22 11:17:37,682 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:18:02,514 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11600, loss[loss=0.05038, simple_loss=0.0597, pruned_loss=0.009825, audio_tagging_loss=0.01071, over 14911.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09426, pruned_loss=0.01507, audio_tagging_loss=0.009211, over 3046810.23 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:18:06,166 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288150
2023-11-22 11:18:09,508 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.16 vs. limit=15.0
2023-11-22 11:18:26,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1921033.3333333333, ans=0.0
2023-11-22 11:18:55,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1921233.3333333333, ans=0.125
2023-11-22 11:18:56,249 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.284e+01 8.414e+01 8.937e+01 9.626e+01 1.842e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-22 11:18:56,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.74 vs. limit=12.0
2023-11-22 11:19:05,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1921300.0, ans=0.125
2023-11-22 11:19:07,313 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11650, loss[loss=0.0749, simple_loss=0.1006, pruned_loss=0.01567, audio_tagging_loss=0.008939, over 15135.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09413, pruned_loss=0.0152, audio_tagging_loss=0.009281, over 3044967.27 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:19:11,732 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288200
2023-11-22 11:19:16,437 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.88 vs. limit=10.0
2023-11-22 11:19:28,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1921366.6666666667, ans=0.125
2023-11-22 11:19:35,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1921433.3333333333, ans=0.125
2023-11-22 11:19:39,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1921433.3333333333, ans=0.09899494936611666
2023-11-22 11:19:44,869 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.01 vs. limit=15.0
2023-11-22 11:19:46,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.76 vs. limit=12.0
2023-11-22 11:19:51,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1921500.0, ans=0.2
2023-11-22 11:20:01,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1921566.6666666667, ans=0.125
2023-11-22 11:20:11,339 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11700, loss[loss=0.1123, simple_loss=0.1507, pruned_loss=0.02981, audio_tagging_loss=0.007113, over 15081.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09394, pruned_loss=0.01515, audio_tagging_loss=0.009398, over 3046136.51 frames. ], batch size: 58, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:20:15,004 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288250
2023-11-22 11:20:21,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1921633.3333333333, ans=0.125
2023-11-22 11:20:32,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1921700.0, ans=0.2
2023-11-22 11:20:35,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0
2023-11-22 11:21:00,965 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.23 vs. limit=10.0
2023-11-22 11:21:05,058 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.380e+01 8.961e+01 9.495e+01 1.206e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 11:21:12,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1921900.0, ans=0.2
2023-11-22 11:21:15,693 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11750, loss[loss=0.05079, simple_loss=0.06614, pruned_loss=0.007408, audio_tagging_loss=0.01031, over 15054.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09331, pruned_loss=0.01518, audio_tagging_loss=0.009419, over 3039251.67 frames. ], batch size: 58, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:21:17,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1921966.6666666667, ans=0.1
2023-11-22 11:21:19,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288300
2023-11-22 11:21:28,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1922033.3333333333, ans=0.125
2023-11-22 11:21:38,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1922033.3333333333, ans=0.1
2023-11-22 11:21:43,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1922100.0, ans=0.0
2023-11-22 11:21:47,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1922100.0, ans=0.125
2023-11-22 11:22:05,538 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0
2023-11-22 11:22:16,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1922233.3333333333, ans=0.2
2023-11-22 11:22:20,067 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11800, loss[loss=0.0745, simple_loss=0.09655, pruned_loss=0.01817, audio_tagging_loss=0.00806, over 16175.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09281, pruned_loss=0.01521, audio_tagging_loss=0.009534, over 3048484.89 frames. ], batch size: 58, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:22:24,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288350
2023-11-22 11:22:39,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1922366.6666666667, ans=0.05
2023-11-22 11:22:46,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1922433.3333333333, ans=0.1
2023-11-22 11:22:47,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1922433.3333333333, ans=0.125
2023-11-22 11:22:47,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1922433.3333333333, ans=0.125
2023-11-22 11:22:56,933 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.51 vs. limit=10.0
2023-11-22 11:23:14,677 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.163e+01 8.779e+01 9.476e+01 1.167e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 11:23:20,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1922566.6666666667, ans=0.125
2023-11-22 11:23:25,259 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11850, loss[loss=0.07916, simple_loss=0.1037, pruned_loss=0.01742, audio_tagging_loss=0.009914, over 14877.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.0932, pruned_loss=0.01533, audio_tagging_loss=0.009501, over 3039592.08 frames. ], batch size: 57, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:23:29,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288400
2023-11-22 11:23:33,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1922633.3333333333, ans=0.2
2023-11-22 11:23:42,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1922700.0, ans=0.0
2023-11-22 11:24:09,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1922833.3333333333, ans=0.0
2023-11-22 11:24:27,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1922900.0, ans=0.0
2023-11-22 11:24:29,691 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11900, loss[loss=0.08187, simple_loss=0.1042, pruned_loss=0.01843, audio_tagging_loss=0.01135, over 15582.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09353, pruned_loss=0.01544, audio_tagging_loss=0.009596, over 3045985.98 frames. ], batch size: 56, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:24:33,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288450
2023-11-22 11:25:17,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1923166.6666666667, ans=0.0
2023-11-22 11:25:23,683 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.115e+01 8.834e+01 9.649e+01 1.264e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 11:25:31,152 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:25:31,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1923233.3333333333, ans=0.1
2023-11-22 11:25:33,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1923300.0, ans=0.125
2023-11-22 11:25:34,085 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 11950, loss[loss=0.08083, simple_loss=0.09351, pruned_loss=0.02039, audio_tagging_loss=0.01368, over 14869.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09266, pruned_loss=0.01526, audio_tagging_loss=0.009696, over 3044401.96 frames. ], batch size: 58, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:25:34,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1923300.0, ans=0.125
2023-11-22 11:25:37,858 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288500
2023-11-22 11:25:41,457 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.20 vs. limit=15.0
2023-11-22 11:25:49,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1923366.6666666667, ans=0.0
2023-11-22 11:26:14,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.91 vs. limit=10.0
2023-11-22 11:26:35,557 INFO [train_asr.py:1221] (2/4) Epoch 24, batch 12000, loss[loss=0.07093, simple_loss=0.09405, pruned_loss=0.01324, audio_tagging_loss=0.01066, over 16668.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.0928, pruned_loss=0.01516, audio_tagging_loss=0.009841, over 3045464.46 frames. ], batch size: 63, lr: 2.81e-03, grad_scale: 32.0
2023-11-22 11:26:35,558 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 11:27:10,472 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9210, 3.7668, 4.8965, 4.4575], device='cuda:2')
2023-11-22 11:27:17,218 INFO [train_asr.py:1253] (2/4) Epoch 24, validation: loss=0.05896, simple_loss=0.05166, pruned_loss=0.00516, audio_tagging_loss=0.02797, over 4681554.00 frames. 
2023-11-22 11:27:17,219 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 11:27:20,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288550
2023-11-22 11:27:31,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1923700.0, ans=0.05
2023-11-22 11:28:19,666 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 0, loss[loss=0.06973, simple_loss=0.06888, pruned_loss=0.01167, audio_tagging_loss=0.02362, over 14872.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.06888, pruned_loss=0.01167, audio_tagging_loss=0.02362, over 14872.00 frames. ], batch size: 58, lr: 2.76e-03, grad_scale: 32.0
2023-11-22 11:28:19,666 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 11:28:55,811 INFO [train_asr.py:1253] (2/4) Epoch 25, validation: loss=0.05903, simple_loss=0.05164, pruned_loss=0.005146, audio_tagging_loss=0.02807, over 4681554.00 frames. 
2023-11-22 11:28:55,812 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 11:29:18,963 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.611e+01 9.501e+01 1.042e+02 1.380e+02, threshold=1.900e+02, percent-clipped=0.0
2023-11-22 11:29:34,039 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288600
2023-11-22 11:29:58,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1924060.0, ans=0.125
2023-11-22 11:30:00,633 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 50, loss[loss=0.07074, simple_loss=0.0813, pruned_loss=0.01329, audio_tagging_loss=0.01679, over 14794.00 frames. ], tot_loss[loss=0.08155, simple_loss=0.09724, pruned_loss=0.01569, audio_tagging_loss=0.01724, over 689254.48 frames. ], batch size: 55, lr: 2.76e-03, grad_scale: 32.0
2023-11-22 11:30:04,909 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.55 vs. limit=6.0
2023-11-22 11:30:14,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1924193.3333333333, ans=0.1
2023-11-22 11:30:20,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.88 vs. limit=15.0
2023-11-22 11:30:38,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288650
2023-11-22 11:30:57,472 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:31:04,557 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 100, loss[loss=0.07907, simple_loss=0.1011, pruned_loss=0.01179, audio_tagging_loss=0.01673, over 16244.00 frames. ], tot_loss[loss=0.0811, simple_loss=0.09761, pruned_loss=0.01567, audio_tagging_loss=0.01663, over 1216542.17 frames. ], batch size: 62, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:31:23,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.65 vs. limit=15.0
2023-11-22 11:31:28,375 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.104e+01 8.752e+01 9.299e+01 1.007e+02 1.330e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-22 11:31:42,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288700
2023-11-22 11:32:03,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.77 vs. limit=6.0
2023-11-22 11:32:10,039 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 150, loss[loss=0.09256, simple_loss=0.119, pruned_loss=0.02171, audio_tagging_loss=0.01133, over 15491.00 frames. ], tot_loss[loss=0.0791, simple_loss=0.09688, pruned_loss=0.01554, audio_tagging_loss=0.01512, over 1623248.13 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:32:24,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1924860.0, ans=0.0
2023-11-22 11:32:42,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1924926.6666666667, ans=0.1
2023-11-22 11:32:47,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288750
2023-11-22 11:32:49,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1924993.3333333333, ans=0.125
2023-11-22 11:33:11,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1925060.0, ans=10.0
2023-11-22 11:33:14,437 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 200, loss[loss=0.0688, simple_loss=0.09184, pruned_loss=0.01412, audio_tagging_loss=0.00876, over 15390.00 frames. ], tot_loss[loss=0.07731, simple_loss=0.09712, pruned_loss=0.01543, audio_tagging_loss=0.01332, over 1936065.40 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:33:25,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1925126.6666666667, ans=0.1
2023-11-22 11:33:37,541 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.935e+01 8.185e+01 8.728e+01 9.456e+01 1.257e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 11:33:39,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1925260.0, ans=0.2
2023-11-22 11:33:40,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1925260.0, ans=0.125
2023-11-22 11:33:42,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1925260.0, ans=0.125
2023-11-22 11:33:44,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1925260.0, ans=0.125
2023-11-22 11:33:51,619 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288800
2023-11-22 11:33:54,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1925326.6666666667, ans=0.125
2023-11-22 11:34:03,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1925326.6666666667, ans=0.5
2023-11-22 11:34:13,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1925393.3333333333, ans=0.0
2023-11-22 11:34:18,353 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 250, loss[loss=0.07761, simple_loss=0.1081, pruned_loss=0.01521, audio_tagging_loss=0.008331, over 16886.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09736, pruned_loss=0.0154, audio_tagging_loss=0.01199, over 2182023.07 frames. ], batch size: 64, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:34:27,804 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:34:48,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1925593.3333333333, ans=0.125
2023-11-22 11:34:52,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1925593.3333333333, ans=0.125
2023-11-22 11:34:54,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1925593.3333333333, ans=0.1
2023-11-22 11:34:54,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1925593.3333333333, ans=0.1
2023-11-22 11:34:55,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288850
2023-11-22 11:35:13,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1925726.6666666667, ans=0.0
2023-11-22 11:35:22,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1925793.3333333333, ans=0.125
2023-11-22 11:35:22,899 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 300, loss[loss=0.05605, simple_loss=0.06905, pruned_loss=0.01115, audio_tagging_loss=0.01038, over 15340.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09621, pruned_loss=0.01522, audio_tagging_loss=0.01116, over 2378707.21 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:35:26,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=1925793.3333333333, ans=0.025
2023-11-22 11:35:45,377 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.207e+01 8.755e+01 9.551e+01 1.256e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 11:35:57,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1925926.6666666667, ans=0.1
2023-11-22 11:35:59,597 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288900
2023-11-22 11:36:05,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1925993.3333333333, ans=0.1
2023-11-22 11:36:14,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1926060.0, ans=0.0
2023-11-22 11:36:15,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1926060.0, ans=0.125
2023-11-22 11:36:27,376 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 350, loss[loss=0.1065, simple_loss=0.1451, pruned_loss=0.02715, audio_tagging_loss=0.006795, over 15398.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.096, pruned_loss=0.01511, audio_tagging_loss=0.01065, over 2527372.99 frames. ], batch size: 53, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:36:31,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1926126.6666666667, ans=0.1
2023-11-22 11:36:45,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1926193.3333333333, ans=0.125
2023-11-22 11:37:05,080 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 288950
2023-11-22 11:37:08,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1926326.6666666667, ans=0.0
2023-11-22 11:37:20,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1926393.3333333333, ans=0.125
2023-11-22 11:37:20,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1926393.3333333333, ans=0.0
2023-11-22 11:37:32,210 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 400, loss[loss=0.07441, simple_loss=0.104, pruned_loss=0.01708, audio_tagging_loss=0.005311, over 14239.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09561, pruned_loss=0.01521, audio_tagging_loss=0.0103, over 2642215.25 frames. ], batch size: 53, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:37:42,858 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.57 vs. limit=22.5
2023-11-22 11:37:44,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1926526.6666666667, ans=0.0
2023-11-22 11:37:48,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1926526.6666666667, ans=0.0
2023-11-22 11:37:49,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1926526.6666666667, ans=0.125
2023-11-22 11:37:53,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1926526.6666666667, ans=0.0
2023-11-22 11:37:55,743 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.456e+01 8.000e+01 8.654e+01 9.629e+01 1.287e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 11:38:03,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1926593.3333333333, ans=0.2
2023-11-22 11:38:09,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289000
2023-11-22 11:38:15,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.56 vs. limit=15.0
2023-11-22 11:38:29,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1926726.6666666667, ans=0.125
2023-11-22 11:38:37,048 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 450, loss[loss=0.05293, simple_loss=0.0605, pruned_loss=0.01227, audio_tagging_loss=0.01041, over 14246.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09474, pruned_loss=0.01512, audio_tagging_loss=0.009991, over 2722946.31 frames. ], batch size: 55, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:39:02,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1926926.6666666667, ans=0.0
2023-11-22 11:39:14,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289050
2023-11-22 11:39:24,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1926993.3333333333, ans=0.125
2023-11-22 11:39:24,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1926993.3333333333, ans=0.125
2023-11-22 11:39:42,475 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 500, loss[loss=0.07573, simple_loss=0.1024, pruned_loss=0.01606, audio_tagging_loss=0.008496, over 15478.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09539, pruned_loss=0.01517, audio_tagging_loss=0.009808, over 2789562.54 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:40:01,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1927193.3333333333, ans=0.0
2023-11-22 11:40:04,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1927193.3333333333, ans=0.1
2023-11-22 11:40:05,717 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.152e+01 8.293e+01 8.940e+01 9.677e+01 1.305e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 11:40:19,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289100
2023-11-22 11:40:32,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1927326.6666666667, ans=0.0
2023-11-22 11:40:47,559 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 550, loss[loss=0.07588, simple_loss=0.1036, pruned_loss=0.01631, audio_tagging_loss=0.007755, over 15452.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09409, pruned_loss=0.01484, audio_tagging_loss=0.009709, over 2840768.34 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:41:06,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1927526.6666666667, ans=0.0
2023-11-22 11:41:10,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1927526.6666666667, ans=0.0
2023-11-22 11:41:12,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1927593.3333333333, ans=0.0
2023-11-22 11:41:24,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289150
2023-11-22 11:41:40,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1927726.6666666667, ans=0.2
2023-11-22 11:41:51,756 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 600, loss[loss=0.08448, simple_loss=0.1103, pruned_loss=0.01887, audio_tagging_loss=0.01045, over 14860.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09417, pruned_loss=0.01484, audio_tagging_loss=0.009689, over 2888730.96 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:41:51,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1927793.3333333333, ans=0.0
2023-11-22 11:42:13,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1927860.0, ans=0.2
2023-11-22 11:42:16,622 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.617e+01 7.918e+01 8.734e+01 9.306e+01 1.583e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 11:42:29,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289200
2023-11-22 11:42:36,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1927993.3333333333, ans=0.09899494936611666
2023-11-22 11:42:49,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1928060.0, ans=0.125
2023-11-22 11:42:53,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.36 vs. limit=15.0
2023-11-22 11:42:57,719 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 650, loss[loss=0.0795, simple_loss=0.1052, pruned_loss=0.01797, audio_tagging_loss=0.008921, over 16310.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09487, pruned_loss=0.01487, audio_tagging_loss=0.009617, over 2929705.51 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:43:00,866 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.82 vs. limit=15.0
2023-11-22 11:43:05,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1928126.6666666667, ans=0.1
2023-11-22 11:43:11,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1928193.3333333333, ans=0.125
2023-11-22 11:43:22,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.08 vs. limit=22.5
2023-11-22 11:43:22,857 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.96 vs. limit=8.0
2023-11-22 11:43:31,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.76 vs. limit=15.0
2023-11-22 11:43:34,918 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289250
2023-11-22 11:43:47,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1928393.3333333333, ans=0.125
2023-11-22 11:43:50,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1928393.3333333333, ans=0.125
2023-11-22 11:44:01,466 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 700, loss[loss=0.05755, simple_loss=0.07922, pruned_loss=0.01004, audio_tagging_loss=0.007896, over 16133.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09519, pruned_loss=0.01499, audio_tagging_loss=0.009478, over 2950839.11 frames. ], batch size: 61, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:44:25,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.97 vs. limit=12.0
2023-11-22 11:44:26,161 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.213e+01 8.759e+01 9.279e+01 1.127e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 11:44:32,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1928593.3333333333, ans=0.125
2023-11-22 11:44:39,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289300
2023-11-22 11:44:42,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1928660.0, ans=0.125
2023-11-22 11:44:42,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1928660.0, ans=0.1
2023-11-22 11:44:45,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1928660.0, ans=0.0
2023-11-22 11:44:59,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1928726.6666666667, ans=0.125
2023-11-22 11:45:01,489 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.56 vs. limit=15.0
2023-11-22 11:45:05,546 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 750, loss[loss=0.06117, simple_loss=0.07388, pruned_loss=0.01051, audio_tagging_loss=0.01372, over 15036.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09472, pruned_loss=0.01501, audio_tagging_loss=0.009565, over 2969468.24 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:45:25,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1928860.0, ans=0.2
2023-11-22 11:45:32,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.64 vs. limit=22.5
2023-11-22 11:45:43,744 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289350
2023-11-22 11:45:48,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1928993.3333333333, ans=0.125
2023-11-22 11:46:10,233 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 800, loss[loss=0.06436, simple_loss=0.08033, pruned_loss=0.01319, audio_tagging_loss=0.01101, over 15073.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09499, pruned_loss=0.01517, audio_tagging_loss=0.009543, over 2987422.67 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:46:17,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1929126.6666666667, ans=0.125
2023-11-22 11:46:36,819 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.118e+01 8.155e+01 8.807e+01 9.499e+01 1.198e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 11:46:39,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1929260.0, ans=0.125
2023-11-22 11:46:48,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289400
2023-11-22 11:46:50,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1929326.6666666667, ans=0.1
2023-11-22 11:46:59,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1929326.6666666667, ans=0.1
2023-11-22 11:47:12,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1929393.3333333333, ans=0.0
2023-11-22 11:47:16,265 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 850, loss[loss=0.06947, simple_loss=0.09202, pruned_loss=0.01424, audio_tagging_loss=0.009218, over 15713.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09549, pruned_loss=0.01539, audio_tagging_loss=0.009522, over 3006939.87 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:47:34,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1929526.6666666667, ans=0.1
2023-11-22 11:47:36,267 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:47:40,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1929526.6666666667, ans=0.1
2023-11-22 11:47:54,053 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289450
2023-11-22 11:48:01,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1929660.0, ans=0.125
2023-11-22 11:48:03,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1929660.0, ans=0.0
2023-11-22 11:48:21,277 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 900, loss[loss=0.06396, simple_loss=0.08316, pruned_loss=0.0112, audio_tagging_loss=0.01117, over 14607.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09524, pruned_loss=0.01539, audio_tagging_loss=0.009518, over 3014683.70 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:48:29,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1929793.3333333333, ans=0.2
2023-11-22 11:48:33,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1929860.0, ans=0.125
2023-11-22 11:48:46,853 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.150e+01 8.654e+01 9.309e+01 1.377e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 11:48:58,869 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289500
2023-11-22 11:49:05,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1929993.3333333333, ans=0.125
2023-11-22 11:49:05,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1929993.3333333333, ans=0.2
2023-11-22 11:49:26,012 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 950, loss[loss=0.06585, simple_loss=0.09557, pruned_loss=0.01115, audio_tagging_loss=0.006906, over 14987.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09479, pruned_loss=0.01533, audio_tagging_loss=0.009498, over 3026844.66 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:49:39,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1930193.3333333333, ans=0.0
2023-11-22 11:49:54,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1930260.0, ans=0.125
2023-11-22 11:50:04,406 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289550
2023-11-22 11:50:14,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.71 vs. limit=15.0
2023-11-22 11:50:18,308 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.21 vs. limit=22.5
2023-11-22 11:50:21,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1930393.3333333333, ans=0.2
2023-11-22 11:50:22,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1930393.3333333333, ans=0.0
2023-11-22 11:50:24,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1930393.3333333333, ans=0.0
2023-11-22 11:50:27,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1930393.3333333333, ans=0.125
2023-11-22 11:50:30,821 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1000, loss[loss=0.07977, simple_loss=0.1087, pruned_loss=0.01742, audio_tagging_loss=0.008017, over 15149.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09447, pruned_loss=0.01513, audio_tagging_loss=0.009399, over 3029705.09 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:50:32,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1930460.0, ans=0.125
2023-11-22 11:50:51,085 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:50:57,563 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.182e+01 8.192e+01 8.953e+01 9.947e+01 1.294e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 11:50:58,883 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:51:01,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1930593.3333333333, ans=0.125
2023-11-22 11:51:08,590 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289600
2023-11-22 11:51:34,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.44 vs. limit=15.0
2023-11-22 11:51:36,491 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1050, loss[loss=0.09903, simple_loss=0.1358, pruned_loss=0.02569, audio_tagging_loss=0.005414, over 14968.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09416, pruned_loss=0.01515, audio_tagging_loss=0.009374, over 3032591.62 frames. ], batch size: 53, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:51:38,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1930793.3333333333, ans=0.125
2023-11-22 11:51:47,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1930793.3333333333, ans=0.125
2023-11-22 11:52:13,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289650
2023-11-22 11:52:19,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1930993.3333333333, ans=0.2
2023-11-22 11:52:40,618 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1100, loss[loss=0.0713, simple_loss=0.0857, pruned_loss=0.01651, audio_tagging_loss=0.01194, over 13644.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09296, pruned_loss=0.01488, audio_tagging_loss=0.009303, over 3032398.44 frames. ], batch size: 53, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:52:44,282 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:53:03,884 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.53 vs. limit=15.0
2023-11-22 11:53:06,984 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.014e+01 8.517e+01 9.464e+01 1.257e+02, threshold=1.703e+02, percent-clipped=0.0
2023-11-22 11:53:13,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1931260.0, ans=0.0
2023-11-22 11:53:15,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1931260.0, ans=0.1
2023-11-22 11:53:18,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289700
2023-11-22 11:53:45,303 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1150, loss[loss=0.0584, simple_loss=0.06787, pruned_loss=0.01457, audio_tagging_loss=0.009888, over 14738.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09281, pruned_loss=0.01499, audio_tagging_loss=0.009398, over 3033845.62 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:54:03,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1931526.6666666667, ans=0.125
2023-11-22 11:54:22,865 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289750
2023-11-22 11:54:28,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1931660.0, ans=0.0
2023-11-22 11:54:41,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.23 vs. limit=15.0
2023-11-22 11:54:42,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1931726.6666666667, ans=0.2
2023-11-22 11:54:45,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1931726.6666666667, ans=0.0
2023-11-22 11:54:47,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1931726.6666666667, ans=0.1
2023-11-22 11:54:50,871 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1200, loss[loss=0.05119, simple_loss=0.06443, pruned_loss=0.008426, audio_tagging_loss=0.01055, over 15085.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09138, pruned_loss=0.01476, audio_tagging_loss=0.009407, over 3031314.91 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:54:51,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1931793.3333333333, ans=0.125
2023-11-22 11:55:07,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1931860.0, ans=0.1
2023-11-22 11:55:10,333 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:55:11,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1931860.0, ans=0.125
2023-11-22 11:55:16,755 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.526e+01 8.025e+01 8.580e+01 9.215e+01 1.708e+02, threshold=1.716e+02, percent-clipped=1.0
2023-11-22 11:55:28,151 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289800
2023-11-22 11:55:40,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.45 vs. limit=22.5
2023-11-22 11:55:43,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1932060.0, ans=0.1
2023-11-22 11:55:54,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1932060.0, ans=0.125
2023-11-22 11:55:56,224 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1250, loss[loss=0.07684, simple_loss=0.09861, pruned_loss=0.01758, audio_tagging_loss=0.00996, over 15433.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09164, pruned_loss=0.01474, audio_tagging_loss=0.00943, over 3040122.27 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:55:57,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1932126.6666666667, ans=0.125
2023-11-22 11:55:59,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1932126.6666666667, ans=0.0
2023-11-22 11:56:33,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289850
2023-11-22 11:56:36,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1932326.6666666667, ans=0.2
2023-11-22 11:56:46,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1932393.3333333333, ans=0.125
2023-11-22 11:56:49,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1932393.3333333333, ans=0.125
2023-11-22 11:56:58,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1932393.3333333333, ans=0.0
2023-11-22 11:57:00,753 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1300, loss[loss=0.07976, simple_loss=0.1032, pruned_loss=0.01954, audio_tagging_loss=0.008596, over 15639.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09201, pruned_loss=0.01461, audio_tagging_loss=0.009422, over 3042382.77 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:57:01,105 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.278e-02
2023-11-22 11:57:08,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1932460.0, ans=0.2
2023-11-22 11:57:26,642 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.940e+01 8.056e+01 8.885e+01 9.522e+01 1.296e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 11:57:38,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289900
2023-11-22 11:57:42,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1932660.0, ans=0.0
2023-11-22 11:57:51,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1932726.6666666667, ans=0.2
2023-11-22 11:57:57,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1932726.6666666667, ans=0.125
2023-11-22 11:58:05,448 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1350, loss[loss=0.073, simple_loss=0.08726, pruned_loss=0.01792, audio_tagging_loss=0.01144, over 14979.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09162, pruned_loss=0.0145, audio_tagging_loss=0.00942, over 3044785.67 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:58:41,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1932926.6666666667, ans=0.125
2023-11-22 11:58:42,938 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 289950
2023-11-22 11:58:52,196 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:59:03,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1933060.0, ans=0.0
2023-11-22 11:59:05,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1933060.0, ans=0.0
2023-11-22 11:59:09,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1933126.6666666667, ans=0.09899494936611666
2023-11-22 11:59:10,469 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1400, loss[loss=0.09836, simple_loss=0.139, pruned_loss=0.02256, audio_tagging_loss=0.00628, over 15990.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09224, pruned_loss=0.01478, audio_tagging_loss=0.009386, over 3044326.55 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:59:13,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1933126.6666666667, ans=0.0
2023-11-22 11:59:17,113 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.65 vs. limit=10.0
2023-11-22 11:59:36,929 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.595e+01 8.260e+01 8.672e+01 9.990e+01 1.427e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-22 11:59:47,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290000
2023-11-22 12:00:15,344 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1450, loss[loss=0.07489, simple_loss=0.09294, pruned_loss=0.01822, audio_tagging_loss=0.0102, over 13808.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09289, pruned_loss=0.01499, audio_tagging_loss=0.009476, over 3045500.52 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:00:53,409 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290050
2023-11-22 12:01:17,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1933726.6666666667, ans=0.0
2023-11-22 12:01:20,153 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1500, loss[loss=0.05868, simple_loss=0.07429, pruned_loss=0.01118, audio_tagging_loss=0.01036, over 14464.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09441, pruned_loss=0.01523, audio_tagging_loss=0.009482, over 3041506.04 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:01:32,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1933860.0, ans=0.125
2023-11-22 12:01:43,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1933860.0, ans=0.125
2023-11-22 12:01:45,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1933926.6666666667, ans=0.1
2023-11-22 12:01:46,716 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.126e+01 8.732e+01 9.513e+01 1.269e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 12:01:57,401 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290100
2023-11-22 12:02:08,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1933993.3333333333, ans=0.2
2023-11-22 12:02:15,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1934060.0, ans=0.125
2023-11-22 12:02:19,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1934060.0, ans=0.0
2023-11-22 12:02:24,722 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1550, loss[loss=0.0656, simple_loss=0.08566, pruned_loss=0.01515, audio_tagging_loss=0.007628, over 14564.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09481, pruned_loss=0.01535, audio_tagging_loss=0.009544, over 3044232.31 frames. ], batch size: 55, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:02:29,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1934126.6666666667, ans=0.125
2023-11-22 12:02:29,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1934126.6666666667, ans=0.0
2023-11-22 12:02:30,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1934126.6666666667, ans=0.0
2023-11-22 12:02:35,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1934126.6666666667, ans=0.0
2023-11-22 12:02:37,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1934193.3333333333, ans=0.0
2023-11-22 12:02:43,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1934193.3333333333, ans=0.125
2023-11-22 12:02:47,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1934193.3333333333, ans=0.0
2023-11-22 12:03:02,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290150
2023-11-22 12:03:11,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1934326.6666666667, ans=0.125
2023-11-22 12:03:20,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1934393.3333333333, ans=0.125
2023-11-22 12:03:28,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=1934393.3333333333, ans=0.1
2023-11-22 12:03:30,909 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1600, loss[loss=0.06287, simple_loss=0.07849, pruned_loss=0.01136, audio_tagging_loss=0.01227, over 15037.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09479, pruned_loss=0.01521, audio_tagging_loss=0.009555, over 3043499.65 frames. ], batch size: 61, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 12:03:45,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1934526.6666666667, ans=0.125
2023-11-22 12:03:55,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1934593.3333333333, ans=0.125
2023-11-22 12:03:56,092 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.51 vs. limit=15.0
2023-11-22 12:03:58,921 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.200e+01 8.897e+01 9.742e+01 1.180e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 12:04:08,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290200
2023-11-22 12:04:25,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1934726.6666666667, ans=0.125
2023-11-22 12:04:28,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1934726.6666666667, ans=0.025
2023-11-22 12:04:29,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1934726.6666666667, ans=0.0
2023-11-22 12:04:33,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1934726.6666666667, ans=0.1
2023-11-22 12:04:35,535 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1650, loss[loss=0.08598, simple_loss=0.1123, pruned_loss=0.01901, audio_tagging_loss=0.01083, over 16249.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09497, pruned_loss=0.01533, audio_tagging_loss=0.009504, over 3053045.33 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:04:46,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1934793.3333333333, ans=0.0
2023-11-22 12:05:08,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1934926.6666666667, ans=0.125
2023-11-22 12:05:11,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1934926.6666666667, ans=0.1
2023-11-22 12:05:13,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290250
2023-11-22 12:05:14,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1934993.3333333333, ans=0.125
2023-11-22 12:05:16,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1934993.3333333333, ans=0.05
2023-11-22 12:05:22,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1934993.3333333333, ans=0.125
2023-11-22 12:05:24,560 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.95 vs. limit=15.0
2023-11-22 12:05:31,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1935060.0, ans=0.125
2023-11-22 12:05:40,393 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1700, loss[loss=0.05872, simple_loss=0.07864, pruned_loss=0.01284, audio_tagging_loss=0.006554, over 15331.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09359, pruned_loss=0.01514, audio_tagging_loss=0.009635, over 3049346.59 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:05:48,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1935126.6666666667, ans=0.025
2023-11-22 12:06:10,634 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.333e+01 9.072e+01 9.807e+01 1.367e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-22 12:06:12,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1935260.0, ans=0.1
2023-11-22 12:06:18,707 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290300
2023-11-22 12:06:18,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1935326.6666666667, ans=0.0
2023-11-22 12:06:22,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1935326.6666666667, ans=0.05
2023-11-22 12:06:23,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1935326.6666666667, ans=0.2
2023-11-22 12:06:35,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1935393.3333333333, ans=0.125
2023-11-22 12:06:45,795 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1750, loss[loss=0.05979, simple_loss=0.08683, pruned_loss=0.009979, audio_tagging_loss=0.006397, over 15056.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09338, pruned_loss=0.01509, audio_tagging_loss=0.009556, over 3049888.69 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:06:48,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1935460.0, ans=0.2
2023-11-22 12:06:50,676 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=14.60 vs. limit=15.0
2023-11-22 12:07:08,453 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:07:23,611 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290350
2023-11-22 12:07:24,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1935660.0, ans=0.2
2023-11-22 12:07:26,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1935660.0, ans=0.125
2023-11-22 12:07:33,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1935660.0, ans=0.2
2023-11-22 12:07:44,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1935726.6666666667, ans=0.125
2023-11-22 12:07:50,645 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1800, loss[loss=0.09121, simple_loss=0.1225, pruned_loss=0.02299, audio_tagging_loss=0.006973, over 17029.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09349, pruned_loss=0.01498, audio_tagging_loss=0.009479, over 3056429.59 frames. ], batch size: 63, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:08:19,854 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.690e+01 8.156e+01 8.799e+01 9.735e+01 1.149e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 12:08:28,524 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290400
2023-11-22 12:08:55,122 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1850, loss[loss=0.0921, simple_loss=0.1266, pruned_loss=0.02036, audio_tagging_loss=0.008444, over 16015.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09384, pruned_loss=0.01499, audio_tagging_loss=0.009353, over 3051947.29 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:08:56,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1936126.6666666667, ans=0.1
2023-11-22 12:09:07,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1936193.3333333333, ans=0.125
2023-11-22 12:09:33,437 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290450
2023-11-22 12:09:34,961 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:09:45,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1936393.3333333333, ans=0.125
2023-11-22 12:09:56,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1936393.3333333333, ans=0.0
2023-11-22 12:09:59,887 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1900, loss[loss=0.05684, simple_loss=0.07481, pruned_loss=0.01026, audio_tagging_loss=0.009173, over 15031.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09265, pruned_loss=0.0147, audio_tagging_loss=0.009325, over 3050046.24 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:10:01,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1936460.0, ans=0.125
2023-11-22 12:10:15,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1936526.6666666667, ans=0.2
2023-11-22 12:10:16,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.18 vs. limit=22.5
2023-11-22 12:10:22,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1936526.6666666667, ans=0.0
2023-11-22 12:10:29,142 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.157e+01 8.748e+01 9.500e+01 1.354e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 12:10:37,239 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290500
2023-11-22 12:11:04,302 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 1950, loss[loss=0.08984, simple_loss=0.1249, pruned_loss=0.02045, audio_tagging_loss=0.006949, over 15883.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09303, pruned_loss=0.01484, audio_tagging_loss=0.009231, over 3046327.48 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:11:08,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1936793.3333333333, ans=0.0
2023-11-22 12:11:14,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.56 vs. limit=15.0
2023-11-22 12:11:38,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.63 vs. limit=10.0
2023-11-22 12:11:41,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290550
2023-11-22 12:12:08,544 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2000, loss[loss=0.06678, simple_loss=0.08253, pruned_loss=0.01651, audio_tagging_loss=0.009014, over 16073.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09239, pruned_loss=0.01472, audio_tagging_loss=0.009312, over 3050703.40 frames. ], batch size: 63, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:12:14,685 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.54 vs. limit=15.0
2023-11-22 12:12:20,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1937193.3333333333, ans=0.07
2023-11-22 12:12:24,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1937193.3333333333, ans=0.125
2023-11-22 12:12:26,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1937193.3333333333, ans=0.2
2023-11-22 12:12:38,528 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.037e+01 8.579e+01 9.384e+01 1.253e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 12:12:40,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1937260.0, ans=0.1
2023-11-22 12:12:44,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1937260.0, ans=0.125
2023-11-22 12:12:45,918 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290600
2023-11-22 12:12:50,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1937326.6666666667, ans=0.0
2023-11-22 12:12:52,017 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:12:55,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1937326.6666666667, ans=0.125
2023-11-22 12:12:56,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1937326.6666666667, ans=0.1
2023-11-22 12:13:02,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.51 vs. limit=5.0
2023-11-22 12:13:08,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0
2023-11-22 12:13:13,306 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2050, loss[loss=0.1032, simple_loss=0.1345, pruned_loss=0.02765, audio_tagging_loss=0.008241, over 15036.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09264, pruned_loss=0.01473, audio_tagging_loss=0.009289, over 3042597.08 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:13:23,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1937460.0, ans=0.2
2023-11-22 12:13:30,326 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.60 vs. limit=15.0
2023-11-22 12:13:50,724 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290650
2023-11-22 12:14:18,215 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2100, loss[loss=0.07089, simple_loss=0.08433, pruned_loss=0.01646, audio_tagging_loss=0.01227, over 16161.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09254, pruned_loss=0.01467, audio_tagging_loss=0.009347, over 3052454.90 frames. ], batch size: 60, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:14:27,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.43 vs. limit=22.5
2023-11-22 12:14:32,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1937860.0, ans=0.0
2023-11-22 12:14:37,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1937860.0, ans=0.1
2023-11-22 12:14:44,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1937926.6666666667, ans=0.0
2023-11-22 12:14:47,390 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.119e+01 8.164e+01 8.808e+01 9.351e+01 1.143e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 12:14:47,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1937926.6666666667, ans=10.0
2023-11-22 12:14:48,199 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.39 vs. limit=15.0
2023-11-22 12:14:54,711 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290700
2023-11-22 12:14:54,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1937993.3333333333, ans=0.1
2023-11-22 12:15:08,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1938060.0, ans=0.125
2023-11-22 12:15:16,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1938060.0, ans=0.0
2023-11-22 12:15:17,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1938060.0, ans=0.07
2023-11-22 12:15:18,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1938060.0, ans=0.125
2023-11-22 12:15:22,337 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2150, loss[loss=0.08429, simple_loss=0.1168, pruned_loss=0.01906, audio_tagging_loss=0.006843, over 16162.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09297, pruned_loss=0.01496, audio_tagging_loss=0.009347, over 3054839.32 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:15:37,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1938193.3333333333, ans=0.1
2023-11-22 12:15:56,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1938260.0, ans=0.2
2023-11-22 12:15:59,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290750
2023-11-22 12:16:01,968 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:16:20,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1938393.3333333333, ans=0.0
2023-11-22 12:16:23,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1938393.3333333333, ans=0.2
2023-11-22 12:16:26,548 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2200, loss[loss=0.07642, simple_loss=0.09867, pruned_loss=0.01703, audio_tagging_loss=0.01006, over 15845.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09384, pruned_loss=0.01505, audio_tagging_loss=0.009386, over 3051835.72 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:16:30,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1938460.0, ans=0.0
2023-11-22 12:16:56,426 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.385e+01 8.783e+01 9.663e+01 1.140e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 12:17:02,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1938593.3333333333, ans=0.125
2023-11-22 12:17:02,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1938593.3333333333, ans=0.025
2023-11-22 12:17:03,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290800
2023-11-22 12:17:23,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.04 vs. limit=22.5
2023-11-22 12:17:30,282 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-22 12:17:30,930 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2250, loss[loss=0.07891, simple_loss=0.09851, pruned_loss=0.01976, audio_tagging_loss=0.009896, over 16078.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09431, pruned_loss=0.01532, audio_tagging_loss=0.009321, over 3054224.69 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:17:32,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1938793.3333333333, ans=0.125
2023-11-22 12:17:47,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1938860.0, ans=0.0
2023-11-22 12:18:08,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290850
2023-11-22 12:18:36,069 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2300, loss[loss=0.08736, simple_loss=0.1129, pruned_loss=0.02154, audio_tagging_loss=0.009391, over 14850.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09454, pruned_loss=0.01527, audio_tagging_loss=0.009323, over 3054716.83 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:18:40,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1939126.6666666667, ans=0.2
2023-11-22 12:18:55,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1939193.3333333333, ans=0.125
2023-11-22 12:18:57,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1939193.3333333333, ans=0.0
2023-11-22 12:19:00,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1939260.0, ans=0.125
2023-11-22 12:19:06,739 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.734e+01 8.285e+01 8.843e+01 9.474e+01 1.631e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 12:19:06,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1939260.0, ans=0.125
2023-11-22 12:19:12,968 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290900
2023-11-22 12:19:27,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1939393.3333333333, ans=0.125
2023-11-22 12:19:28,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1939393.3333333333, ans=0.0
2023-11-22 12:19:31,964 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:19:39,818 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2350, loss[loss=0.07673, simple_loss=0.1103, pruned_loss=0.01398, audio_tagging_loss=0.007613, over 15014.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09443, pruned_loss=0.01502, audio_tagging_loss=0.009325, over 3058444.70 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:19:43,014 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.29 vs. limit=12.0
2023-11-22 12:19:59,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1939526.6666666667, ans=0.125
2023-11-22 12:20:17,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 290950
2023-11-22 12:20:30,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1939726.6666666667, ans=0.09899494936611666
2023-11-22 12:20:38,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1939726.6666666667, ans=0.125
2023-11-22 12:20:44,657 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2400, loss[loss=0.08546, simple_loss=0.1156, pruned_loss=0.02048, audio_tagging_loss=0.007161, over 15846.00 frames. ], tot_loss[loss=0.07219, simple_loss=0.09512, pruned_loss=0.01523, audio_tagging_loss=0.0094, over 3063900.99 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:21:15,530 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.176e+01 8.394e+01 8.930e+01 9.736e+01 4.109e+02, threshold=1.786e+02, percent-clipped=1.0
2023-11-22 12:21:21,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291000
2023-11-22 12:21:46,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1940060.0, ans=0.1
2023-11-22 12:21:50,317 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2450, loss[loss=0.08137, simple_loss=0.1029, pruned_loss=0.02042, audio_tagging_loss=0.009521, over 13859.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.0951, pruned_loss=0.01528, audio_tagging_loss=0.009501, over 3067146.31 frames. ], batch size: 53, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:21:56,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1940126.6666666667, ans=0.09899494936611666
2023-11-22 12:22:03,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1940193.3333333333, ans=0.125
2023-11-22 12:22:07,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1940193.3333333333, ans=0.1
2023-11-22 12:22:11,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1940193.3333333333, ans=0.125
2023-11-22 12:22:24,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1940260.0, ans=0.2
2023-11-22 12:22:28,619 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291050
2023-11-22 12:22:29,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1940326.6666666667, ans=0.125
2023-11-22 12:22:56,580 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2500, loss[loss=0.06397, simple_loss=0.08534, pruned_loss=0.01271, audio_tagging_loss=0.008583, over 14458.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09469, pruned_loss=0.01521, audio_tagging_loss=0.009584, over 3062266.40 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:23:26,947 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.685e+01 8.038e+01 8.739e+01 9.351e+01 1.232e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 12:23:31,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1940593.3333333333, ans=0.125
2023-11-22 12:23:33,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291100
2023-11-22 12:23:42,427 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.79 vs. limit=15.0
2023-11-22 12:23:45,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1940660.0, ans=0.0
2023-11-22 12:23:52,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1940726.6666666667, ans=0.125
2023-11-22 12:24:01,986 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2550, loss[loss=0.04725, simple_loss=0.06445, pruned_loss=0.008512, audio_tagging_loss=0.006514, over 15468.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09466, pruned_loss=0.01527, audio_tagging_loss=0.009515, over 3053696.64 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:24:13,756 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=22.5
2023-11-22 12:24:18,206 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.96 vs. limit=15.0
2023-11-22 12:24:40,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291150
2023-11-22 12:24:49,216 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-22 12:24:52,444 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.54 vs. limit=22.5
2023-11-22 12:24:57,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1941060.0, ans=0.2
2023-11-22 12:25:06,889 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2600, loss[loss=0.06601, simple_loss=0.09016, pruned_loss=0.01167, audio_tagging_loss=0.009255, over 14506.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09477, pruned_loss=0.01536, audio_tagging_loss=0.009419, over 3046414.29 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:25:07,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1941126.6666666667, ans=0.0
2023-11-22 12:25:10,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1941126.6666666667, ans=0.125
2023-11-22 12:25:19,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1941193.3333333333, ans=0.125
2023-11-22 12:25:23,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1941193.3333333333, ans=0.125
2023-11-22 12:25:30,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1941193.3333333333, ans=0.5
2023-11-22 12:25:39,208 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.159e+01 9.048e+01 9.727e+01 1.323e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 12:25:46,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291200
2023-11-22 12:25:46,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1941326.6666666667, ans=0.125
2023-11-22 12:25:46,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1941326.6666666667, ans=0.125
2023-11-22 12:26:14,632 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2650, loss[loss=0.09208, simple_loss=0.1238, pruned_loss=0.02369, audio_tagging_loss=0.006506, over 15679.00 frames. ], tot_loss[loss=0.07219, simple_loss=0.09487, pruned_loss=0.01548, audio_tagging_loss=0.009276, over 3050379.55 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:26:18,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1941460.0, ans=0.125
2023-11-22 12:26:25,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1941460.0, ans=0.125
2023-11-22 12:26:28,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1941526.6666666667, ans=0.125
2023-11-22 12:26:30,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1941526.6666666667, ans=0.0
2023-11-22 12:26:49,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1941593.3333333333, ans=0.0
2023-11-22 12:26:53,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291250
2023-11-22 12:27:06,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1941726.6666666667, ans=0.0
2023-11-22 12:27:20,075 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2700, loss[loss=0.06415, simple_loss=0.08943, pruned_loss=0.01184, audio_tagging_loss=0.007596, over 15467.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09471, pruned_loss=0.01544, audio_tagging_loss=0.009276, over 3053655.86 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:27:50,981 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.955e+01 8.601e+01 9.459e+01 1.243e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-22 12:27:58,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291300
2023-11-22 12:28:17,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1942060.0, ans=0.125
2023-11-22 12:28:21,870 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=15.0
2023-11-22 12:28:25,086 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2750, loss[loss=0.06726, simple_loss=0.09242, pruned_loss=0.0123, audio_tagging_loss=0.008753, over 15701.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09365, pruned_loss=0.01512, audio_tagging_loss=0.009409, over 3046600.05 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:28:40,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.35 vs. limit=12.0
2023-11-22 12:28:45,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1942193.3333333333, ans=0.0
2023-11-22 12:28:49,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1942193.3333333333, ans=0.0
2023-11-22 12:28:49,889 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.30 vs. limit=15.0
2023-11-22 12:29:03,584 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291350
2023-11-22 12:29:10,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1942326.6666666667, ans=0.125
2023-11-22 12:29:18,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1942393.3333333333, ans=0.125
2023-11-22 12:29:21,441 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:29:30,142 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2800, loss[loss=0.06463, simple_loss=0.08952, pruned_loss=0.01127, audio_tagging_loss=0.008604, over 14554.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.093, pruned_loss=0.01488, audio_tagging_loss=0.009286, over 3045225.07 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:29:46,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1942526.6666666667, ans=0.2
2023-11-22 12:30:01,639 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.537e+01 8.249e+01 8.938e+01 9.573e+01 1.382e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 12:30:07,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291400
2023-11-22 12:30:26,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1942726.6666666667, ans=0.125
2023-11-22 12:30:27,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1942726.6666666667, ans=0.0
2023-11-22 12:30:36,187 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2850, loss[loss=0.07021, simple_loss=0.08874, pruned_loss=0.01726, audio_tagging_loss=0.008578, over 14589.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.0929, pruned_loss=0.01482, audio_tagging_loss=0.009324, over 3047439.30 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:31:04,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1942926.6666666667, ans=0.035
2023-11-22 12:31:13,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291450
2023-11-22 12:31:21,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1942993.3333333333, ans=0.1
2023-11-22 12:31:25,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1942993.3333333333, ans=0.125
2023-11-22 12:31:30,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1943060.0, ans=0.0
2023-11-22 12:31:34,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1943060.0, ans=0.125
2023-11-22 12:31:40,801 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2900, loss[loss=0.07599, simple_loss=0.09621, pruned_loss=0.0177, audio_tagging_loss=0.01018, over 13979.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09408, pruned_loss=0.01514, audio_tagging_loss=0.009274, over 3046221.79 frames. ], batch size: 52, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:31:45,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.23 vs. limit=15.0
2023-11-22 12:31:50,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.35 vs. limit=22.5
2023-11-22 12:31:54,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1943193.3333333333, ans=0.015
2023-11-22 12:32:02,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1943193.3333333333, ans=0.0
2023-11-22 12:32:05,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1943260.0, ans=0.125
2023-11-22 12:32:06,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=1943260.0, ans=0.02
2023-11-22 12:32:11,960 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.290e+01 8.892e+01 9.564e+01 1.220e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 12:32:14,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1943260.0, ans=0.0
2023-11-22 12:32:19,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291500
2023-11-22 12:32:28,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1943326.6666666667, ans=0.05
2023-11-22 12:32:45,619 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 2950, loss[loss=0.07499, simple_loss=0.1048, pruned_loss=0.01529, audio_tagging_loss=0.007283, over 15237.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09433, pruned_loss=0.01508, audio_tagging_loss=0.009262, over 3046270.57 frames. ], batch size: 53, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:33:02,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1943526.6666666667, ans=0.125
2023-11-22 12:33:10,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1943526.6666666667, ans=0.125
2023-11-22 12:33:15,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1943593.3333333333, ans=0.1
2023-11-22 12:33:23,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291550
2023-11-22 12:33:47,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1943726.6666666667, ans=0.07
2023-11-22 12:33:49,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1943793.3333333333, ans=0.125
2023-11-22 12:33:50,765 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3000, loss[loss=0.05833, simple_loss=0.07069, pruned_loss=0.01211, audio_tagging_loss=0.01088, over 14560.00 frames. ], tot_loss[loss=0.07164, simple_loss=0.09417, pruned_loss=0.0152, audio_tagging_loss=0.009364, over 3051913.32 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:33:50,766 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 12:34:30,490 INFO [train_asr.py:1253] (2/4) Epoch 25, validation: loss=0.05876, simple_loss=0.05157, pruned_loss=0.005103, audio_tagging_loss=0.02788, over 4681554.00 frames. 
2023-11-22 12:34:30,491 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 12:34:34,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1943793.3333333333, ans=0.5
2023-11-22 12:34:39,147 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.26 vs. limit=15.0
2023-11-22 12:34:44,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1943860.0, ans=0.125
2023-11-22 12:34:57,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1943926.6666666667, ans=0.5
2023-11-22 12:35:01,167 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.344e+01 8.945e+01 9.581e+01 1.218e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-22 12:35:08,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291600
2023-11-22 12:35:21,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.96 vs. limit=12.0
2023-11-22 12:35:35,433 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3050, loss[loss=0.08063, simple_loss=0.1038, pruned_loss=0.0196, audio_tagging_loss=0.009119, over 15404.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09313, pruned_loss=0.015, audio_tagging_loss=0.009544, over 3054409.19 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:35:40,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1944126.6666666667, ans=0.0
2023-11-22 12:35:45,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1944126.6666666667, ans=0.0
2023-11-22 12:35:46,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1944126.6666666667, ans=0.125
2023-11-22 12:35:48,939 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.59 vs. limit=15.0
2023-11-22 12:36:11,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1944260.0, ans=0.2
2023-11-22 12:36:13,308 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:36:13,362 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291650
2023-11-22 12:36:23,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1944326.6666666667, ans=0.0
2023-11-22 12:36:39,981 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3100, loss[loss=0.07361, simple_loss=0.09931, pruned_loss=0.01375, audio_tagging_loss=0.01021, over 14956.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09303, pruned_loss=0.01486, audio_tagging_loss=0.009566, over 3043635.64 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:36:41,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1944460.0, ans=10.0
2023-11-22 12:36:43,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1944460.0, ans=0.125
2023-11-22 12:36:49,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1944460.0, ans=0.0
2023-11-22 12:37:05,130 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-22 12:37:10,659 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.355e+01 8.957e+01 9.791e+01 1.144e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 12:37:17,524 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291700
2023-11-22 12:37:19,266 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.30 vs. limit=15.0
2023-11-22 12:37:31,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1944726.6666666667, ans=0.125
2023-11-22 12:37:44,665 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3150, loss[loss=0.07014, simple_loss=0.09191, pruned_loss=0.0146, audio_tagging_loss=0.009578, over 14843.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.0944, pruned_loss=0.01521, audio_tagging_loss=0.00956, over 3042346.27 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:37:59,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1944860.0, ans=0.2
2023-11-22 12:38:19,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1944926.6666666667, ans=0.125
2023-11-22 12:38:21,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291750
2023-11-22 12:38:23,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1944993.3333333333, ans=0.07
2023-11-22 12:38:49,290 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3200, loss[loss=0.08137, simple_loss=0.118, pruned_loss=0.01538, audio_tagging_loss=0.006996, over 16133.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.0948, pruned_loss=0.01528, audio_tagging_loss=0.00955, over 3051728.22 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:38:49,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1945126.6666666667, ans=0.05
2023-11-22 12:38:52,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1945126.6666666667, ans=0.2
2023-11-22 12:39:03,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1945193.3333333333, ans=0.125
2023-11-22 12:39:08,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1945193.3333333333, ans=0.1
2023-11-22 12:39:20,212 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.187e+01 8.718e+01 9.345e+01 1.176e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 12:39:26,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1945260.0, ans=0.125
2023-11-22 12:39:27,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291800
2023-11-22 12:39:28,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1945326.6666666667, ans=0.125
2023-11-22 12:39:41,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1945393.3333333333, ans=0.07
2023-11-22 12:39:45,809 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:39:47,570 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.53 vs. limit=15.0
2023-11-22 12:39:53,931 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3250, loss[loss=0.07542, simple_loss=0.09953, pruned_loss=0.01739, audio_tagging_loss=0.008263, over 16082.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09504, pruned_loss=0.01523, audio_tagging_loss=0.009589, over 3053324.61 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:40:24,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.26 vs. limit=15.0
2023-11-22 12:40:29,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1945593.3333333333, ans=0.2
2023-11-22 12:40:30,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1945593.3333333333, ans=0.2
2023-11-22 12:40:31,192 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291850
2023-11-22 12:40:58,266 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3300, loss[loss=0.07256, simple_loss=0.09623, pruned_loss=0.01323, audio_tagging_loss=0.01122, over 15977.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09562, pruned_loss=0.01516, audio_tagging_loss=0.009586, over 3055724.02 frames. ], batch size: 61, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:41:29,363 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.525e+01 8.169e+01 8.738e+01 9.487e+01 1.174e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 12:41:35,666 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291900
2023-11-22 12:41:42,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1945993.3333333333, ans=0.125
2023-11-22 12:42:03,258 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3350, loss[loss=0.06753, simple_loss=0.09198, pruned_loss=0.01292, audio_tagging_loss=0.008624, over 16678.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09539, pruned_loss=0.01499, audio_tagging_loss=0.009513, over 3057645.19 frames. ], batch size: 63, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:42:22,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1946193.3333333333, ans=0.1
2023-11-22 12:42:40,610 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 291950
2023-11-22 12:42:42,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1946326.6666666667, ans=0.0
2023-11-22 12:42:47,314 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-22 12:42:51,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.29 vs. limit=15.0
2023-11-22 12:43:08,066 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3400, loss[loss=0.07696, simple_loss=0.1016, pruned_loss=0.01751, audio_tagging_loss=0.008658, over 17294.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09476, pruned_loss=0.01491, audio_tagging_loss=0.009489, over 3060604.20 frames. ], batch size: 64, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:43:17,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1946460.0, ans=0.125
2023-11-22 12:43:38,750 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.227e+01 8.817e+01 9.429e+01 1.318e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 12:43:41,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1946593.3333333333, ans=0.035
2023-11-22 12:43:45,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292000
2023-11-22 12:43:50,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1946660.0, ans=0.0
2023-11-22 12:43:56,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1946660.0, ans=0.0
2023-11-22 12:44:05,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1946726.6666666667, ans=0.0
2023-11-22 12:44:14,905 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.56 vs. limit=10.0
2023-11-22 12:44:15,282 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3450, loss[loss=0.08828, simple_loss=0.1203, pruned_loss=0.02207, audio_tagging_loss=0.006059, over 16221.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09455, pruned_loss=0.01492, audio_tagging_loss=0.009391, over 3059392.44 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:44:23,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.59 vs. limit=15.0
2023-11-22 12:44:24,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1946793.3333333333, ans=0.125
2023-11-22 12:44:25,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1946793.3333333333, ans=0.125
2023-11-22 12:44:51,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292050
2023-11-22 12:44:54,602 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:44:55,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1946993.3333333333, ans=0.1
2023-11-22 12:45:11,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.30 vs. limit=12.0
2023-11-22 12:45:19,188 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3500, loss[loss=0.06669, simple_loss=0.08809, pruned_loss=0.01378, audio_tagging_loss=0.008864, over 14245.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09422, pruned_loss=0.0149, audio_tagging_loss=0.009257, over 3063099.78 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:45:19,770 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.58 vs. limit=15.0
2023-11-22 12:45:33,869 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.08 vs. limit=12.0
2023-11-22 12:45:43,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1947260.0, ans=0.125
2023-11-22 12:45:47,212 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.72 vs. limit=15.0
2023-11-22 12:45:51,049 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:45:52,128 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 7.990e+01 8.840e+01 9.495e+01 1.155e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 12:45:55,983 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292100
2023-11-22 12:46:15,061 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.16 vs. limit=22.5
2023-11-22 12:46:17,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1947393.3333333333, ans=0.0
2023-11-22 12:46:20,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1947393.3333333333, ans=0.1
2023-11-22 12:46:23,358 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3550, loss[loss=0.0698, simple_loss=0.09449, pruned_loss=0.01471, audio_tagging_loss=0.00785, over 15948.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09388, pruned_loss=0.01477, audio_tagging_loss=0.009226, over 3059043.19 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:46:37,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1947526.6666666667, ans=0.0
2023-11-22 12:46:37,912 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.42 vs. limit=15.0
2023-11-22 12:46:44,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1947526.6666666667, ans=0.125
2023-11-22 12:46:56,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1947593.3333333333, ans=0.125
2023-11-22 12:47:00,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292150
2023-11-22 12:47:09,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.10 vs. limit=15.0
2023-11-22 12:47:18,651 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.08 vs. limit=15.0
2023-11-22 12:47:27,278 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3600, loss[loss=0.07673, simple_loss=0.0977, pruned_loss=0.01528, audio_tagging_loss=0.0126, over 14913.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09326, pruned_loss=0.01471, audio_tagging_loss=0.009177, over 3057667.61 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:47:38,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1947860.0, ans=0.0
2023-11-22 12:47:46,718 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.77 vs. limit=15.0
2023-11-22 12:47:48,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1947860.0, ans=0.0
2023-11-22 12:47:55,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1947926.6666666667, ans=0.0
2023-11-22 12:47:57,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1947926.6666666667, ans=0.0
2023-11-22 12:48:00,885 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 8.103e+01 8.938e+01 1.008e+02 1.423e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 12:48:04,088 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.78 vs. limit=6.0
2023-11-22 12:48:04,701 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292200
2023-11-22 12:48:05,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1947993.3333333333, ans=0.125
2023-11-22 12:48:31,935 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3650, loss[loss=0.08097, simple_loss=0.1089, pruned_loss=0.01821, audio_tagging_loss=0.008291, over 14834.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09335, pruned_loss=0.01478, audio_tagging_loss=0.009119, over 3052257.26 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:48:37,444 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.37 vs. limit=15.0
2023-11-22 12:48:49,950 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.75 vs. limit=15.0
2023-11-22 12:48:55,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1948193.3333333333, ans=10.0
2023-11-22 12:49:09,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292250
2023-11-22 12:49:21,636 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.88 vs. limit=15.0
2023-11-22 12:49:37,037 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3700, loss[loss=0.07544, simple_loss=0.09214, pruned_loss=0.01705, audio_tagging_loss=0.01232, over 15402.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09375, pruned_loss=0.01503, audio_tagging_loss=0.009145, over 3055756.05 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:50:02,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1948593.3333333333, ans=0.125
2023-11-22 12:50:10,186 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.131e+01 8.890e+01 9.560e+01 1.123e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 12:50:13,981 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292300
2023-11-22 12:50:41,534 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3750, loss[loss=0.05635, simple_loss=0.07011, pruned_loss=0.009284, audio_tagging_loss=0.01201, over 14391.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09362, pruned_loss=0.01511, audio_tagging_loss=0.009365, over 3052050.80 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:51:00,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1948860.0, ans=0.125
2023-11-22 12:51:19,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292350
2023-11-22 12:51:25,162 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:51:45,487 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3800, loss[loss=0.08155, simple_loss=0.1123, pruned_loss=0.01781, audio_tagging_loss=0.007592, over 16356.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09446, pruned_loss=0.01521, audio_tagging_loss=0.009369, over 3055302.58 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:52:06,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1949193.3333333333, ans=0.1
2023-11-22 12:52:21,348 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.073e+01 8.669e+01 9.455e+01 1.608e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-22 12:52:23,909 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292400
2023-11-22 12:52:24,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1949326.6666666667, ans=0.125
2023-11-22 12:52:50,532 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3850, loss[loss=0.05568, simple_loss=0.06975, pruned_loss=0.009138, audio_tagging_loss=0.01166, over 14942.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09491, pruned_loss=0.01527, audio_tagging_loss=0.009356, over 3052513.66 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:52:50,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1949460.0, ans=0.1
2023-11-22 12:53:02,754 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.39 vs. limit=15.0
2023-11-22 12:53:05,179 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.55 vs. limit=15.0
2023-11-22 12:53:27,524 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.89 vs. limit=15.0
2023-11-22 12:53:28,226 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292450
2023-11-22 12:53:34,865 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.55 vs. limit=15.0
2023-11-22 12:53:46,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1949726.6666666667, ans=0.2
2023-11-22 12:53:55,290 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3900, loss[loss=0.07032, simple_loss=0.0904, pruned_loss=0.01456, audio_tagging_loss=0.01056, over 16206.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09456, pruned_loss=0.01523, audio_tagging_loss=0.009387, over 3046450.06 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:54:22,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1949926.6666666667, ans=0.125
2023-11-22 12:54:26,678 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.09 vs. limit=15.0
2023-11-22 12:54:30,354 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.059e+01 8.183e+01 8.833e+01 9.463e+01 1.290e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 12:54:33,689 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292500
2023-11-22 12:54:57,001 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.19 vs. limit=22.5
2023-11-22 12:55:00,233 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 3950, loss[loss=0.09799, simple_loss=0.127, pruned_loss=0.02315, audio_tagging_loss=0.01131, over 14456.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.0944, pruned_loss=0.01508, audio_tagging_loss=0.009491, over 3051684.75 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:55:07,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1950126.6666666667, ans=0.0
2023-11-22 12:55:20,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1950193.3333333333, ans=0.2
2023-11-22 12:55:27,397 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.21 vs. limit=22.5
2023-11-22 12:55:38,532 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292550
2023-11-22 12:55:45,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1950326.6666666667, ans=0.1
2023-11-22 12:55:47,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1950326.6666666667, ans=0.0
2023-11-22 12:55:59,638 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.21 vs. limit=15.0
2023-11-22 12:56:04,970 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4000, loss[loss=0.06792, simple_loss=0.08767, pruned_loss=0.01453, audio_tagging_loss=0.009551, over 14794.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09356, pruned_loss=0.01512, audio_tagging_loss=0.009513, over 3049888.26 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:56:20,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1950526.6666666667, ans=0.0
2023-11-22 12:56:34,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.23 vs. limit=10.0
2023-11-22 12:56:37,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=1950593.3333333333, ans=0.02
2023-11-22 12:56:40,611 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.359e+01 8.523e+01 9.261e+01 9.950e+01 1.651e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-22 12:56:43,800 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292600
2023-11-22 12:56:58,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1950726.6666666667, ans=0.1
2023-11-22 12:57:02,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1950726.6666666667, ans=0.125
2023-11-22 12:57:11,644 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4050, loss[loss=0.07091, simple_loss=0.09369, pruned_loss=0.01625, audio_tagging_loss=0.007811, over 14536.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09366, pruned_loss=0.01501, audio_tagging_loss=0.009556, over 3041996.94 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:57:14,238 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:57:18,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1950793.3333333333, ans=0.2
2023-11-22 12:57:18,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1950793.3333333333, ans=0.125
2023-11-22 12:57:49,351 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292650
2023-11-22 12:57:52,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1950993.3333333333, ans=0.0
2023-11-22 12:57:57,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1950993.3333333333, ans=0.125
2023-11-22 12:58:00,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1950993.3333333333, ans=0.1
2023-11-22 12:58:17,015 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4100, loss[loss=0.05954, simple_loss=0.0779, pruned_loss=0.01273, audio_tagging_loss=0.007857, over 14485.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09398, pruned_loss=0.01498, audio_tagging_loss=0.009522, over 3037680.85 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:58:24,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1951126.6666666667, ans=0.1
2023-11-22 12:58:33,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1951193.3333333333, ans=0.1
2023-11-22 12:58:41,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1951260.0, ans=0.1
2023-11-22 12:58:42,053 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=22.5
2023-11-22 12:58:52,133 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.488e+01 8.924e+01 9.565e+01 1.327e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 12:58:54,707 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292700
2023-11-22 12:58:58,686 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.59 vs. limit=22.5
2023-11-22 12:59:14,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1951393.3333333333, ans=0.0
2023-11-22 12:59:22,198 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4150, loss[loss=0.07798, simple_loss=0.1023, pruned_loss=0.02081, audio_tagging_loss=0.006034, over 15649.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09512, pruned_loss=0.01502, audio_tagging_loss=0.00933, over 3045238.15 frames. ], batch size: 61, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:59:28,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1951460.0, ans=0.125
2023-11-22 12:59:34,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.18 vs. limit=15.0
2023-11-22 12:59:43,753 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.37 vs. limit=15.0
2023-11-22 12:59:49,416 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.87 vs. limit=22.5
2023-11-22 12:59:57,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1951593.3333333333, ans=0.09899494936611666
2023-11-22 13:00:00,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292750
2023-11-22 13:00:02,333 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.33 vs. limit=15.0
2023-11-22 13:00:08,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1951660.0, ans=0.125
2023-11-22 13:00:09,178 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:00:24,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1951726.6666666667, ans=0.125
2023-11-22 13:00:26,776 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4200, loss[loss=0.08716, simple_loss=0.1175, pruned_loss=0.02154, audio_tagging_loss=0.00685, over 15253.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.0947, pruned_loss=0.01515, audio_tagging_loss=0.009324, over 3044379.69 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 13:00:31,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.95 vs. limit=15.0
2023-11-22 13:00:54,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1951926.6666666667, ans=0.2
2023-11-22 13:01:02,476 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.208e+01 8.744e+01 9.952e+01 1.323e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 13:01:05,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292800
2023-11-22 13:01:05,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1951993.3333333333, ans=0.1
2023-11-22 13:01:11,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1951993.3333333333, ans=0.0
2023-11-22 13:01:33,210 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4250, loss[loss=0.05129, simple_loss=0.06536, pruned_loss=0.007649, audio_tagging_loss=0.01096, over 15931.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.0946, pruned_loss=0.01503, audio_tagging_loss=0.009291, over 3050312.16 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 13:01:45,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.40 vs. limit=22.5
2023-11-22 13:02:04,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1952260.0, ans=0.125
2023-11-22 13:02:09,962 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292850
2023-11-22 13:02:18,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1952326.6666666667, ans=0.1
2023-11-22 13:02:27,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1952393.3333333333, ans=0.125
2023-11-22 13:02:37,479 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4300, loss[loss=0.0709, simple_loss=0.09077, pruned_loss=0.01328, audio_tagging_loss=0.01223, over 15262.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09502, pruned_loss=0.0152, audio_tagging_loss=0.00922, over 3055616.15 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 13:02:50,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1952526.6666666667, ans=0.0
2023-11-22 13:02:55,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1952526.6666666667, ans=0.0
2023-11-22 13:03:12,144 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.605e+01 8.363e+01 9.136e+01 9.974e+01 1.277e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-22 13:03:14,755 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292900
2023-11-22 13:03:18,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1952660.0, ans=0.125
2023-11-22 13:03:23,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1952660.0, ans=0.125
2023-11-22 13:03:28,314 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.12 vs. limit=22.5
2023-11-22 13:03:41,706 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4350, loss[loss=0.07308, simple_loss=0.09777, pruned_loss=0.01383, audio_tagging_loss=0.01036, over 15250.00 frames. ], tot_loss[loss=0.07184, simple_loss=0.09465, pruned_loss=0.01531, audio_tagging_loss=0.009208, over 3049603.01 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:03:48,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1952793.3333333333, ans=0.125
2023-11-22 13:03:49,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1952793.3333333333, ans=0.2
2023-11-22 13:04:19,677 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 292950
2023-11-22 13:04:19,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1952993.3333333333, ans=0.0
2023-11-22 13:04:21,203 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:04:31,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1952993.3333333333, ans=0.125
2023-11-22 13:04:40,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1953060.0, ans=0.1
2023-11-22 13:04:46,738 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4400, loss[loss=0.06859, simple_loss=0.09161, pruned_loss=0.01452, audio_tagging_loss=0.008269, over 15589.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09521, pruned_loss=0.01551, audio_tagging_loss=0.009179, over 3047039.77 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:04:54,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1953126.6666666667, ans=0.125
2023-11-22 13:05:17,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1953260.0, ans=0.125
2023-11-22 13:05:18,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1953260.0, ans=0.125
2023-11-22 13:05:21,489 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.093e+01 8.881e+01 9.679e+01 1.352e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-22 13:05:21,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1953260.0, ans=0.125
2023-11-22 13:05:24,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293000
2023-11-22 13:05:42,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1953393.3333333333, ans=0.125
2023-11-22 13:05:51,665 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4450, loss[loss=0.08432, simple_loss=0.1164, pruned_loss=0.02046, audio_tagging_loss=0.005665, over 15272.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09549, pruned_loss=0.01545, audio_tagging_loss=0.009114, over 3047853.13 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:06:06,189 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.74 vs. limit=15.0
2023-11-22 13:06:10,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1953526.6666666667, ans=0.125
2023-11-22 13:06:10,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1953526.6666666667, ans=0.2
2023-11-22 13:06:27,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.14 vs. limit=15.0
2023-11-22 13:06:28,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293050
2023-11-22 13:06:31,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1953660.0, ans=0.1
2023-11-22 13:06:47,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1953726.6666666667, ans=0.125
2023-11-22 13:06:51,551 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:06:53,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1953726.6666666667, ans=0.125
2023-11-22 13:06:55,425 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4500, loss[loss=0.07057, simple_loss=0.09327, pruned_loss=0.01617, audio_tagging_loss=0.007762, over 15388.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09469, pruned_loss=0.01527, audio_tagging_loss=0.009151, over 3045319.87 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:06:56,081 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.00 vs. limit=15.0
2023-11-22 13:06:57,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1953793.3333333333, ans=0.0
2023-11-22 13:07:01,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1953793.3333333333, ans=0.125
2023-11-22 13:07:04,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1953793.3333333333, ans=22.5
2023-11-22 13:07:18,881 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.26 vs. limit=15.0
2023-11-22 13:07:22,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1953926.6666666667, ans=0.125
2023-11-22 13:07:29,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1953926.6666666667, ans=0.125
2023-11-22 13:07:31,538 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.144e+01 8.843e+01 9.621e+01 1.771e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 13:07:32,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293100
2023-11-22 13:07:51,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1954060.0, ans=0.07
2023-11-22 13:07:57,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1954126.6666666667, ans=0.125
2023-11-22 13:07:58,962 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4550, loss[loss=0.05068, simple_loss=0.06973, pruned_loss=0.00834, audio_tagging_loss=0.007471, over 15350.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09401, pruned_loss=0.0151, audio_tagging_loss=0.009205, over 3047784.80 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:08:03,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1954126.6666666667, ans=0.125
2023-11-22 13:08:18,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1954193.3333333333, ans=0.125
2023-11-22 13:08:32,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1954260.0, ans=0.2
2023-11-22 13:08:37,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293150
2023-11-22 13:08:48,005 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:08:48,576 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.11 vs. limit=15.0
2023-11-22 13:08:53,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1954393.3333333333, ans=0.125
2023-11-22 13:09:03,942 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4600, loss[loss=0.07345, simple_loss=0.09743, pruned_loss=0.01521, audio_tagging_loss=0.009526, over 15072.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09357, pruned_loss=0.01508, audio_tagging_loss=0.009352, over 3041723.20 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:09:09,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1954460.0, ans=0.2
2023-11-22 13:09:25,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1954526.6666666667, ans=0.125
2023-11-22 13:09:28,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1954593.3333333333, ans=0.05
2023-11-22 13:09:28,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1954593.3333333333, ans=0.0
2023-11-22 13:09:31,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0
2023-11-22 13:09:40,084 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 8.168e+01 8.767e+01 9.419e+01 1.196e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 13:09:41,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293200
2023-11-22 13:10:09,632 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4650, loss[loss=0.06818, simple_loss=0.0875, pruned_loss=0.01357, audio_tagging_loss=0.01086, over 15719.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09365, pruned_loss=0.0149, audio_tagging_loss=0.009351, over 3039734.37 frames. ], batch size: 61, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:10:14,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1954793.3333333333, ans=0.2
2023-11-22 13:10:17,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1954793.3333333333, ans=0.0
2023-11-22 13:10:32,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1954860.0, ans=0.125
2023-11-22 13:10:32,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.75 vs. limit=10.0
2023-11-22 13:10:35,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1954926.6666666667, ans=0.2
2023-11-22 13:10:46,878 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293250
2023-11-22 13:10:54,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.14 vs. limit=22.5
2023-11-22 13:11:04,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1955060.0, ans=0.125
2023-11-22 13:11:09,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1955060.0, ans=0.0
2023-11-22 13:11:13,269 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4700, loss[loss=0.0978, simple_loss=0.1341, pruned_loss=0.02068, audio_tagging_loss=0.01009, over 15371.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09434, pruned_loss=0.01505, audio_tagging_loss=0.00946, over 3045389.30 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:11:19,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1955126.6666666667, ans=0.2
2023-11-22 13:11:38,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1955260.0, ans=0.125
2023-11-22 13:11:38,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.91 vs. limit=6.0
2023-11-22 13:11:44,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1955260.0, ans=0.125
2023-11-22 13:11:49,579 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.061e+01 8.708e+01 9.431e+01 1.267e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 13:11:50,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293300
2023-11-22 13:12:03,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1955393.3333333333, ans=0.0
2023-11-22 13:12:12,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1955393.3333333333, ans=0.0
2023-11-22 13:12:16,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1955460.0, ans=0.125
2023-11-22 13:12:17,192 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4750, loss[loss=0.0816, simple_loss=0.1101, pruned_loss=0.01902, audio_tagging_loss=0.00755, over 15424.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09359, pruned_loss=0.01489, audio_tagging_loss=0.009447, over 3048342.22 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:12:25,006 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.00 vs. limit=15.0
2023-11-22 13:12:54,966 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293350
2023-11-22 13:12:56,998 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:12:59,874 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.12 vs. limit=15.0
2023-11-22 13:13:12,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1955726.6666666667, ans=0.1
2023-11-22 13:13:12,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1955726.6666666667, ans=0.125
2023-11-22 13:13:21,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.81 vs. limit=6.0
2023-11-22 13:13:23,046 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4800, loss[loss=0.08834, simple_loss=0.1166, pruned_loss=0.02157, audio_tagging_loss=0.008473, over 14927.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09355, pruned_loss=0.01486, audio_tagging_loss=0.009682, over 3042826.41 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:13:32,747 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:13:35,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.32 vs. limit=10.0
2023-11-22 13:13:41,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1955860.0, ans=0.125
2023-11-22 13:13:58,934 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 8.018e+01 8.726e+01 9.477e+01 1.215e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 13:13:59,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1955926.6666666667, ans=0.1
2023-11-22 13:14:00,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293400
2023-11-22 13:14:20,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1956060.0, ans=0.125
2023-11-22 13:14:20,504 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.68 vs. limit=15.0
2023-11-22 13:14:23,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1956060.0, ans=0.125
2023-11-22 13:14:28,252 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4850, loss[loss=0.0769, simple_loss=0.1005, pruned_loss=0.01538, audio_tagging_loss=0.01125, over 15255.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.0944, pruned_loss=0.01492, audio_tagging_loss=0.009685, over 3048101.90 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:14:33,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1956126.6666666667, ans=0.0
2023-11-22 13:15:05,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.38 vs. limit=15.0
2023-11-22 13:15:06,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293450
2023-11-22 13:15:20,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1956393.3333333333, ans=0.125
2023-11-22 13:15:27,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1956393.3333333333, ans=0.125
2023-11-22 13:15:33,152 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4900, loss[loss=0.09339, simple_loss=0.1273, pruned_loss=0.02317, audio_tagging_loss=0.006577, over 14394.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09388, pruned_loss=0.01498, audio_tagging_loss=0.009643, over 3045832.85 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:15:45,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1956526.6666666667, ans=0.07
2023-11-22 13:15:50,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1956526.6666666667, ans=0.1
2023-11-22 13:15:53,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1956526.6666666667, ans=0.1
2023-11-22 13:16:00,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1956593.3333333333, ans=0.2
2023-11-22 13:16:10,096 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.038e+01 8.107e+01 8.809e+01 9.600e+01 1.365e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 13:16:11,498 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293500
2023-11-22 13:16:29,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1956726.6666666667, ans=0.125
2023-11-22 13:16:33,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1956726.6666666667, ans=0.0
2023-11-22 13:16:38,729 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 4950, loss[loss=0.05193, simple_loss=0.06312, pruned_loss=0.01158, audio_tagging_loss=0.008786, over 13754.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09307, pruned_loss=0.01492, audio_tagging_loss=0.009481, over 3045883.83 frames. ], batch size: 52, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:16:45,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1956793.3333333333, ans=0.125
2023-11-22 13:16:46,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1956793.3333333333, ans=0.2
2023-11-22 13:16:53,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1956860.0, ans=0.125
2023-11-22 13:16:59,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1956860.0, ans=0.125
2023-11-22 13:17:05,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1956926.6666666667, ans=0.125
2023-11-22 13:17:07,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1956926.6666666667, ans=0.0
2023-11-22 13:17:14,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1956926.6666666667, ans=0.0
2023-11-22 13:17:16,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293550
2023-11-22 13:17:24,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1956993.3333333333, ans=0.09899494936611666
2023-11-22 13:17:28,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1956993.3333333333, ans=0.125
2023-11-22 13:17:38,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1957060.0, ans=0.0
2023-11-22 13:17:42,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1957060.0, ans=0.95
2023-11-22 13:17:44,344 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5000, loss[loss=0.09122, simple_loss=0.1083, pruned_loss=0.02414, audio_tagging_loss=0.01293, over 14663.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09445, pruned_loss=0.01515, audio_tagging_loss=0.009226, over 3046907.85 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:18:16,227 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0
2023-11-22 13:18:20,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.26 vs. limit=22.5
2023-11-22 13:18:20,731 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.208e+01 8.762e+01 9.402e+01 1.160e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 13:18:22,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293600
2023-11-22 13:18:49,653 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5050, loss[loss=0.06104, simple_loss=0.07634, pruned_loss=0.01098, audio_tagging_loss=0.01188, over 14620.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09354, pruned_loss=0.01501, audio_tagging_loss=0.009164, over 3047537.85 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:18:54,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1957460.0, ans=0.0
2023-11-22 13:19:21,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1957593.3333333333, ans=0.125
2023-11-22 13:19:27,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293650
2023-11-22 13:19:44,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1957726.6666666667, ans=0.0
2023-11-22 13:19:54,003 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5100, loss[loss=0.07213, simple_loss=0.09918, pruned_loss=0.01426, audio_tagging_loss=0.008281, over 15351.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09347, pruned_loss=0.01488, audio_tagging_loss=0.009143, over 3048706.72 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:19:59,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1957793.3333333333, ans=0.0
2023-11-22 13:20:02,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1957793.3333333333, ans=0.2
2023-11-22 13:20:10,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1957860.0, ans=0.125
2023-11-22 13:20:30,033 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.490e+01 8.194e+01 8.730e+01 9.544e+01 1.434e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 13:20:31,371 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293700
2023-11-22 13:20:46,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1958060.0, ans=0.125
2023-11-22 13:20:58,901 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5150, loss[loss=0.07301, simple_loss=0.098, pruned_loss=0.0142, audio_tagging_loss=0.009805, over 15700.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09437, pruned_loss=0.01498, audio_tagging_loss=0.009181, over 3054914.01 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:21:24,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1958260.0, ans=0.125
2023-11-22 13:21:35,840 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293750
2023-11-22 13:21:42,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1958326.6666666667, ans=0.125
2023-11-22 13:21:44,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1958326.6666666667, ans=0.0
2023-11-22 13:21:45,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1958326.6666666667, ans=0.0
2023-11-22 13:21:45,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1958326.6666666667, ans=0.0
2023-11-22 13:22:03,627 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5200, loss[loss=0.08782, simple_loss=0.1158, pruned_loss=0.02087, audio_tagging_loss=0.009048, over 15408.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09447, pruned_loss=0.01524, audio_tagging_loss=0.009203, over 3053546.92 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:22:06,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1958460.0, ans=0.0
2023-11-22 13:22:15,811 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.22 vs. limit=15.0
2023-11-22 13:22:23,114 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.03 vs. limit=15.0
2023-11-22 13:22:39,708 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.666e+01 8.256e+01 9.056e+01 9.741e+01 1.453e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 13:22:41,117 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293800
2023-11-22 13:22:49,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0
2023-11-22 13:23:08,255 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5250, loss[loss=0.05359, simple_loss=0.05672, pruned_loss=0.009621, audio_tagging_loss=0.01562, over 14851.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.0951, pruned_loss=0.01538, audio_tagging_loss=0.009193, over 3047390.01 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:23:46,130 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293850
2023-11-22 13:23:46,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1958993.3333333333, ans=0.0
2023-11-22 13:24:12,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.13 vs. limit=8.0
2023-11-22 13:24:13,213 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5300, loss[loss=0.06775, simple_loss=0.08237, pruned_loss=0.01482, audio_tagging_loss=0.01175, over 14884.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09542, pruned_loss=0.01545, audio_tagging_loss=0.009196, over 3044599.70 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:24:17,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1959126.6666666667, ans=0.0
2023-11-22 13:24:38,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1959260.0, ans=0.0
2023-11-22 13:24:47,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=1959260.0, ans=0.05
2023-11-22 13:24:49,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1959260.0, ans=0.0
2023-11-22 13:24:51,125 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.505e+01 9.104e+01 9.788e+01 1.254e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-22 13:24:51,292 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293900
2023-11-22 13:25:01,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.06 vs. limit=12.0
2023-11-22 13:25:18,116 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5350, loss[loss=0.06324, simple_loss=0.08617, pruned_loss=0.01144, audio_tagging_loss=0.008716, over 14912.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09574, pruned_loss=0.0156, audio_tagging_loss=0.009211, over 3049861.76 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:25:18,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1959460.0, ans=0.0
2023-11-22 13:25:30,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1959526.6666666667, ans=0.2
2023-11-22 13:25:31,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1959526.6666666667, ans=0.0
2023-11-22 13:25:44,484 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0
2023-11-22 13:25:44,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1959593.3333333333, ans=15.0
2023-11-22 13:25:55,574 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 293950
2023-11-22 13:25:56,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.00 vs. limit=10.0
2023-11-22 13:26:11,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1959726.6666666667, ans=0.125
2023-11-22 13:26:14,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1959726.6666666667, ans=0.05
2023-11-22 13:26:16,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1959726.6666666667, ans=0.125
2023-11-22 13:26:19,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1959726.6666666667, ans=0.125
2023-11-22 13:26:23,246 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5400, loss[loss=0.08704, simple_loss=0.1151, pruned_loss=0.02011, audio_tagging_loss=0.009392, over 14760.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09628, pruned_loss=0.01575, audio_tagging_loss=0.009259, over 3041390.26 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:26:45,160 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.62 vs. limit=15.0
2023-11-22 13:26:58,849 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.43 vs. limit=6.0
2023-11-22 13:27:00,604 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.190e+01 8.796e+01 9.372e+01 1.218e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 13:27:00,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294000
2023-11-22 13:27:01,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1959993.3333333333, ans=0.2
2023-11-22 13:27:04,087 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.34 vs. limit=22.5
2023-11-22 13:27:28,012 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5450, loss[loss=0.06418, simple_loss=0.08226, pruned_loss=0.01377, audio_tagging_loss=0.009284, over 15407.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09566, pruned_loss=0.01578, audio_tagging_loss=0.009312, over 3043715.63 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:27:31,191 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.21 vs. limit=15.0
2023-11-22 13:27:42,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.20 vs. limit=15.0
2023-11-22 13:27:51,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1960193.3333333333, ans=0.125
2023-11-22 13:28:06,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294050
2023-11-22 13:28:18,334 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.66 vs. limit=15.0
2023-11-22 13:28:19,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1960393.3333333333, ans=0.125
2023-11-22 13:28:32,363 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5500, loss[loss=0.08575, simple_loss=0.1119, pruned_loss=0.02151, audio_tagging_loss=0.008278, over 14617.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.09498, pruned_loss=0.01567, audio_tagging_loss=0.009351, over 3052473.11 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:28:42,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.55 vs. limit=15.0
2023-11-22 13:28:43,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1960460.0, ans=0.125
2023-11-22 13:28:49,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=1960526.6666666667, ans=15.0
2023-11-22 13:28:58,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1960593.3333333333, ans=0.05
2023-11-22 13:29:01,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1960593.3333333333, ans=0.2
2023-11-22 13:29:10,076 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.609e+01 8.387e+01 8.932e+01 9.464e+01 1.203e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 13:29:10,251 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294100
2023-11-22 13:29:24,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.06 vs. limit=15.0
2023-11-22 13:29:27,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1960726.6666666667, ans=0.0
2023-11-22 13:29:30,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1960726.6666666667, ans=0.2
2023-11-22 13:29:32,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1960726.6666666667, ans=0.125
2023-11-22 13:29:37,763 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5550, loss[loss=0.05383, simple_loss=0.06582, pruned_loss=0.009997, audio_tagging_loss=0.01092, over 15495.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09519, pruned_loss=0.01568, audio_tagging_loss=0.009363, over 3057009.82 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:29:41,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1960793.3333333333, ans=0.125
2023-11-22 13:29:44,513 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.40 vs. limit=15.0
2023-11-22 13:29:54,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1960860.0, ans=0.125
2023-11-22 13:29:56,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.35 vs. limit=15.0
2023-11-22 13:30:14,757 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294150
2023-11-22 13:30:16,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1960993.3333333333, ans=0.1
2023-11-22 13:30:25,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1960993.3333333333, ans=0.1
2023-11-22 13:30:36,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1961060.0, ans=0.125
2023-11-22 13:30:39,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1961060.0, ans=0.125
2023-11-22 13:30:42,324 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5600, loss[loss=0.05551, simple_loss=0.07937, pruned_loss=0.009129, audio_tagging_loss=0.006696, over 14040.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09457, pruned_loss=0.01548, audio_tagging_loss=0.009469, over 3055893.72 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:30:57,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1961193.3333333333, ans=0.125
2023-11-22 13:31:19,920 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294200
2023-11-22 13:31:20,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1961326.6666666667, ans=0.0
2023-11-22 13:31:21,012 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.047e+01 8.683e+01 9.396e+01 1.171e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 13:31:22,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1961326.6666666667, ans=0.125
2023-11-22 13:31:26,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1961326.6666666667, ans=0.5
2023-11-22 13:31:28,719 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:31:46,082 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.61 vs. limit=15.0
2023-11-22 13:31:46,619 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5650, loss[loss=0.06506, simple_loss=0.08501, pruned_loss=0.01136, audio_tagging_loss=0.0112, over 14366.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09434, pruned_loss=0.0153, audio_tagging_loss=0.009467, over 3046637.08 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:31:47,605 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.07 vs. limit=8.0
2023-11-22 13:31:58,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1961526.6666666667, ans=0.2
2023-11-22 13:32:07,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1961526.6666666667, ans=0.125
2023-11-22 13:32:24,578 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294250
2023-11-22 13:32:34,904 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.26 vs. limit=6.0
2023-11-22 13:32:37,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1961726.6666666667, ans=0.2
2023-11-22 13:32:38,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0
2023-11-22 13:32:48,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1961726.6666666667, ans=0.125
2023-11-22 13:32:51,463 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5700, loss[loss=0.04776, simple_loss=0.05661, pruned_loss=0.008863, audio_tagging_loss=0.01059, over 14673.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.0939, pruned_loss=0.01525, audio_tagging_loss=0.009431, over 3052727.43 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:32:56,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1961793.3333333333, ans=0.04949747468305833
2023-11-22 13:32:57,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1961793.3333333333, ans=0.1
2023-11-22 13:33:11,395 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:33:28,448 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294300
2023-11-22 13:33:29,497 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.416e+01 9.012e+01 9.956e+01 1.838e+02, threshold=1.802e+02, percent-clipped=1.0
2023-11-22 13:33:39,029 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:33:45,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1962060.0, ans=0.07
2023-11-22 13:33:54,344 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.81 vs. limit=15.0
2023-11-22 13:33:55,983 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5750, loss[loss=0.08154, simple_loss=0.1157, pruned_loss=0.01941, audio_tagging_loss=0.004272, over 15846.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09368, pruned_loss=0.01512, audio_tagging_loss=0.009325, over 3052330.20 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:34:05,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1962126.6666666667, ans=0.125
2023-11-22 13:34:05,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1962126.6666666667, ans=0.125
2023-11-22 13:34:13,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1962193.3333333333, ans=0.125
2023-11-22 13:34:22,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1962260.0, ans=0.0
2023-11-22 13:34:23,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1962260.0, ans=0.0
2023-11-22 13:34:25,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1962260.0, ans=0.0
2023-11-22 13:34:33,960 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294350
2023-11-22 13:34:39,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1962326.6666666667, ans=0.125
2023-11-22 13:34:53,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1962393.3333333333, ans=0.125
2023-11-22 13:35:00,428 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5800, loss[loss=0.06928, simple_loss=0.09188, pruned_loss=0.0159, audio_tagging_loss=0.007436, over 15033.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09425, pruned_loss=0.0152, audio_tagging_loss=0.009231, over 3052100.46 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:35:03,598 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.78 vs. limit=12.0
2023-11-22 13:35:14,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1962526.6666666667, ans=0.1
2023-11-22 13:35:38,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294400
2023-11-22 13:35:39,729 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.175e+01 8.400e+01 8.933e+01 9.518e+01 1.972e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-22 13:35:42,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1962660.0, ans=0.125
2023-11-22 13:35:55,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1962726.6666666667, ans=0.125
2023-11-22 13:36:01,162 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.34 vs. limit=10.0
2023-11-22 13:36:05,428 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5850, loss[loss=0.07339, simple_loss=0.09061, pruned_loss=0.01967, audio_tagging_loss=0.008423, over 15170.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09349, pruned_loss=0.01494, audio_tagging_loss=0.009168, over 3048531.44 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:36:12,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1962793.3333333333, ans=0.2
2023-11-22 13:36:41,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1962926.6666666667, ans=0.125
2023-11-22 13:36:42,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294450
2023-11-22 13:36:45,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1962993.3333333333, ans=0.1
2023-11-22 13:37:04,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1963060.0, ans=0.125
2023-11-22 13:37:10,377 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5900, loss[loss=0.0554, simple_loss=0.06745, pruned_loss=0.009317, audio_tagging_loss=0.01236, over 14812.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09314, pruned_loss=0.01495, audio_tagging_loss=0.009146, over 3045270.54 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:37:26,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.94 vs. limit=15.0
2023-11-22 13:37:38,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1963260.0, ans=0.0
2023-11-22 13:37:48,076 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294500
2023-11-22 13:37:49,035 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.759e+01 8.196e+01 8.876e+01 9.676e+01 1.604e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 13:37:51,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1963326.6666666667, ans=0.1
2023-11-22 13:38:08,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0
2023-11-22 13:38:11,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1963393.3333333333, ans=0.0
2023-11-22 13:38:14,682 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 5950, loss[loss=0.08383, simple_loss=0.1118, pruned_loss=0.01936, audio_tagging_loss=0.008566, over 15476.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09349, pruned_loss=0.01499, audio_tagging_loss=0.009175, over 3043353.01 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:38:33,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1963526.6666666667, ans=0.125
2023-11-22 13:38:38,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1963526.6666666667, ans=0.05
2023-11-22 13:38:46,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1963593.3333333333, ans=0.05
2023-11-22 13:38:52,677 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294550
2023-11-22 13:38:54,419 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.79 vs. limit=15.0
2023-11-22 13:39:05,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1963726.6666666667, ans=0.07
2023-11-22 13:39:16,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1963726.6666666667, ans=0.2
2023-11-22 13:39:19,044 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6000, loss[loss=0.04821, simple_loss=0.0639, pruned_loss=0.006078, audio_tagging_loss=0.01019, over 15294.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09341, pruned_loss=0.01487, audio_tagging_loss=0.009189, over 3045271.44 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:39:19,045 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 13:39:56,541 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.5010, 3.1139, 2.8753, 2.7799, 3.3183, 3.5008, 3.1888, 3.6175],
       device='cuda:2')
2023-11-22 13:40:01,011 INFO [train_asr.py:1253] (2/4) Epoch 25, validation: loss=0.05896, simple_loss=0.05155, pruned_loss=0.00512, audio_tagging_loss=0.02806, over 4681554.00 frames. 
2023-11-22 13:40:01,011 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 13:40:13,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1963860.0, ans=0.0
2023-11-22 13:40:25,698 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:40:38,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294600
2023-11-22 13:40:39,174 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.185e+01 8.770e+01 9.296e+01 1.374e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 13:40:43,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1963993.3333333333, ans=0.2
2023-11-22 13:40:46,813 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:40:47,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1963993.3333333333, ans=0.2
2023-11-22 13:40:49,306 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.66 vs. limit=22.5
2023-11-22 13:41:04,627 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6050, loss[loss=0.06493, simple_loss=0.08848, pruned_loss=0.01223, audio_tagging_loss=0.008455, over 15537.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09417, pruned_loss=0.01504, audio_tagging_loss=0.009155, over 3060930.91 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:41:14,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1964126.6666666667, ans=0.0
2023-11-22 13:41:28,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1964193.3333333333, ans=0.1
2023-11-22 13:41:37,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1964260.0, ans=0.125
2023-11-22 13:41:42,854 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294650
2023-11-22 13:41:55,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1964393.3333333333, ans=0.2
2023-11-22 13:42:09,281 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6100, loss[loss=0.06377, simple_loss=0.08439, pruned_loss=0.01057, audio_tagging_loss=0.01101, over 15204.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09384, pruned_loss=0.01502, audio_tagging_loss=0.009103, over 3055017.79 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:42:16,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1964460.0, ans=0.0
2023-11-22 13:42:46,715 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294700
2023-11-22 13:42:47,801 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.151e+01 8.939e+01 9.703e+01 1.163e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 13:42:58,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1964660.0, ans=0.1
2023-11-22 13:43:13,331 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6150, loss[loss=0.0828, simple_loss=0.111, pruned_loss=0.01923, audio_tagging_loss=0.008061, over 16485.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09354, pruned_loss=0.015, audio_tagging_loss=0.009163, over 3049940.38 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:43:20,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.58 vs. limit=15.0
2023-11-22 13:43:22,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1964793.3333333333, ans=0.125
2023-11-22 13:43:22,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1964793.3333333333, ans=10.0
2023-11-22 13:43:42,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1964926.6666666667, ans=0.0
2023-11-22 13:43:50,861 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294750
2023-11-22 13:43:57,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=1964993.3333333333, ans=0.02
2023-11-22 13:43:57,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1964993.3333333333, ans=0.125
2023-11-22 13:44:18,491 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6200, loss[loss=0.0686, simple_loss=0.09301, pruned_loss=0.0139, audio_tagging_loss=0.008189, over 15034.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09362, pruned_loss=0.01498, audio_tagging_loss=0.009196, over 3039177.50 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:44:28,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1965126.6666666667, ans=0.125
2023-11-22 13:44:31,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1965193.3333333333, ans=0.125
2023-11-22 13:44:36,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.07 vs. limit=15.0
2023-11-22 13:44:42,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1965193.3333333333, ans=0.0
2023-11-22 13:44:42,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1965193.3333333333, ans=0.0
2023-11-22 13:44:45,065 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0
2023-11-22 13:44:45,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1965260.0, ans=0.0
2023-11-22 13:44:56,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294800
2023-11-22 13:44:57,267 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 7.998e+01 8.582e+01 9.232e+01 1.087e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 13:45:04,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1965326.6666666667, ans=0.0
2023-11-22 13:45:18,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1965393.3333333333, ans=0.0
2023-11-22 13:45:23,729 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6250, loss[loss=0.06937, simple_loss=0.08495, pruned_loss=0.01528, audio_tagging_loss=0.01161, over 14256.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.0934, pruned_loss=0.01496, audio_tagging_loss=0.009307, over 3042103.41 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:45:32,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1965460.0, ans=0.125
2023-11-22 13:46:01,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294850
2023-11-22 13:46:03,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1965660.0, ans=0.125
2023-11-22 13:46:09,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.10 vs. limit=8.0
2023-11-22 13:46:28,187 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6300, loss[loss=0.06654, simple_loss=0.09075, pruned_loss=0.0114, audio_tagging_loss=0.009762, over 15062.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09453, pruned_loss=0.01522, audio_tagging_loss=0.00938, over 3045478.72 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:46:36,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1965793.3333333333, ans=0.035
2023-11-22 13:46:47,355 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.56 vs. limit=15.0
2023-11-22 13:46:56,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1965926.6666666667, ans=0.0
2023-11-22 13:47:01,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1965926.6666666667, ans=0.0
2023-11-22 13:47:03,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1965926.6666666667, ans=0.125
2023-11-22 13:47:05,597 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294900
2023-11-22 13:47:07,899 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.416e+01 8.996e+01 9.619e+01 1.207e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 13:47:13,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1965993.3333333333, ans=0.2
2023-11-22 13:47:17,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.79 vs. limit=15.0
2023-11-22 13:47:24,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1966060.0, ans=0.1
2023-11-22 13:47:32,555 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6350, loss[loss=0.05535, simple_loss=0.06489, pruned_loss=0.01019, audio_tagging_loss=0.01271, over 14756.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09446, pruned_loss=0.01518, audio_tagging_loss=0.009597, over 3051174.76 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:47:44,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1966193.3333333333, ans=10.0
2023-11-22 13:47:56,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1966193.3333333333, ans=0.125
2023-11-22 13:48:10,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 294950
2023-11-22 13:48:38,023 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6400, loss[loss=0.08524, simple_loss=0.1173, pruned_loss=0.0179, audio_tagging_loss=0.008706, over 16129.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09436, pruned_loss=0.01514, audio_tagging_loss=0.009579, over 3052327.63 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:49:04,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1966593.3333333333, ans=0.125
2023-11-22 13:49:15,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295000
2023-11-22 13:49:15,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1966660.0, ans=0.125
2023-11-22 13:49:17,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.103e+01 8.926e+01 9.580e+01 1.211e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 13:49:43,148 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6450, loss[loss=0.08301, simple_loss=0.1094, pruned_loss=0.02061, audio_tagging_loss=0.007693, over 15000.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09424, pruned_loss=0.01511, audio_tagging_loss=0.009598, over 3041616.86 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:49:43,854 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0
2023-11-22 13:49:48,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1966793.3333333333, ans=0.0
2023-11-22 13:50:02,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1966860.0, ans=0.125
2023-11-22 13:50:20,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295050
2023-11-22 13:50:32,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1966993.3333333333, ans=0.125
2023-11-22 13:50:35,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1967060.0, ans=0.125
2023-11-22 13:50:47,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1967126.6666666667, ans=0.2
2023-11-22 13:50:47,826 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6500, loss[loss=0.08125, simple_loss=0.1061, pruned_loss=0.01957, audio_tagging_loss=0.008641, over 15419.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.09543, pruned_loss=0.01537, audio_tagging_loss=0.009382, over 3043267.90 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:51:07,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1967193.3333333333, ans=0.1
2023-11-22 13:51:18,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1967260.0, ans=0.1
2023-11-22 13:51:23,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.25 vs. limit=15.0
2023-11-22 13:51:25,054 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295100
2023-11-22 13:51:26,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1967326.6666666667, ans=0.125
2023-11-22 13:51:27,958 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.137e+01 8.907e+01 9.781e+01 1.377e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 13:51:51,242 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6550, loss[loss=0.08044, simple_loss=0.1109, pruned_loss=0.01757, audio_tagging_loss=0.007419, over 15798.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.0946, pruned_loss=0.0151, audio_tagging_loss=0.009274, over 3044120.89 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:52:08,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1967526.6666666667, ans=0.125
2023-11-22 13:52:29,184 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295150
2023-11-22 13:52:51,355 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:52:56,801 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6600, loss[loss=0.08259, simple_loss=0.1176, pruned_loss=0.01787, audio_tagging_loss=0.005942, over 15263.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09441, pruned_loss=0.01503, audio_tagging_loss=0.0091, over 3041661.61 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:53:13,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1967860.0, ans=0.0
2023-11-22 13:53:22,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1967926.6666666667, ans=0.125
2023-11-22 13:53:34,433 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295200
2023-11-22 13:53:37,131 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.628e+01 8.017e+01 8.525e+01 9.445e+01 1.151e+02, threshold=1.705e+02, percent-clipped=0.0
2023-11-22 13:53:42,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1967993.3333333333, ans=0.2
2023-11-22 13:53:45,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1967993.3333333333, ans=0.0
2023-11-22 13:53:52,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1968060.0, ans=0.09899494936611666
2023-11-22 13:53:59,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1968060.0, ans=0.125
2023-11-22 13:54:02,034 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6650, loss[loss=0.04791, simple_loss=0.05585, pruned_loss=0.009198, audio_tagging_loss=0.01079, over 16249.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09258, pruned_loss=0.01477, audio_tagging_loss=0.009154, over 3045564.78 frames. ], batch size: 65, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:54:08,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.22 vs. limit=6.0
2023-11-22 13:54:11,161 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.57 vs. limit=22.5
2023-11-22 13:54:18,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1968193.3333333333, ans=0.2
2023-11-22 13:54:29,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1968260.0, ans=0.125
2023-11-22 13:54:34,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1968260.0, ans=0.125
2023-11-22 13:54:39,583 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295250
2023-11-22 13:54:55,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1968393.3333333333, ans=0.125
2023-11-22 13:54:57,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1968393.3333333333, ans=0.025
2023-11-22 13:55:04,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1968460.0, ans=0.0
2023-11-22 13:55:05,826 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6700, loss[loss=0.0641, simple_loss=0.08419, pruned_loss=0.01251, audio_tagging_loss=0.009496, over 15157.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09294, pruned_loss=0.01496, audio_tagging_loss=0.009106, over 3039519.24 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:55:38,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1968593.3333333333, ans=0.125
2023-11-22 13:55:38,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.87 vs. limit=10.0
2023-11-22 13:55:44,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295300
2023-11-22 13:55:46,361 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.891e+01 8.344e+01 8.911e+01 9.672e+01 1.240e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 13:55:46,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1968660.0, ans=0.0
2023-11-22 13:55:47,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1968660.0, ans=0.0
2023-11-22 13:55:59,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1968726.6666666667, ans=0.1
2023-11-22 13:56:02,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.14 vs. limit=15.0
2023-11-22 13:56:11,665 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6750, loss[loss=0.07521, simple_loss=0.1038, pruned_loss=0.01616, audio_tagging_loss=0.007143, over 15739.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09303, pruned_loss=0.01494, audio_tagging_loss=0.009189, over 3037721.10 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:56:27,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1968860.0, ans=0.125
2023-11-22 13:56:34,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1968860.0, ans=0.0
2023-11-22 13:56:36,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.70 vs. limit=6.0
2023-11-22 13:56:46,951 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.76 vs. limit=12.0
2023-11-22 13:56:47,804 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295350
2023-11-22 13:56:52,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1968993.3333333333, ans=0.0
2023-11-22 13:56:55,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.39 vs. limit=22.5
2023-11-22 13:57:15,711 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6800, loss[loss=0.06873, simple_loss=0.09547, pruned_loss=0.01317, audio_tagging_loss=0.007821, over 14462.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09292, pruned_loss=0.01497, audio_tagging_loss=0.009198, over 3029543.11 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:57:24,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1969126.6666666667, ans=0.125
2023-11-22 13:57:30,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1969193.3333333333, ans=0.0
2023-11-22 13:57:31,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1969193.3333333333, ans=0.125
2023-11-22 13:57:53,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295400
2023-11-22 13:57:56,106 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.834e+01 8.041e+01 8.623e+01 9.531e+01 1.254e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 13:58:20,155 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6850, loss[loss=0.1225, simple_loss=0.1695, pruned_loss=0.03005, audio_tagging_loss=0.007685, over 16209.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09312, pruned_loss=0.01497, audio_tagging_loss=0.009149, over 3033367.80 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:58:22,164 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.42 vs. limit=15.0
2023-11-22 13:58:49,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1969593.3333333333, ans=0.0
2023-11-22 13:58:51,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1969593.3333333333, ans=0.125
2023-11-22 13:58:51,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1969593.3333333333, ans=0.1
2023-11-22 13:58:53,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1969593.3333333333, ans=0.1
2023-11-22 13:58:53,635 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.76 vs. limit=15.0
2023-11-22 13:58:57,995 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295450
2023-11-22 13:59:21,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.31 vs. limit=15.0
2023-11-22 13:59:24,610 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6900, loss[loss=0.04557, simple_loss=0.05611, pruned_loss=0.006989, audio_tagging_loss=0.01052, over 14670.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09416, pruned_loss=0.01504, audio_tagging_loss=0.009152, over 3036429.82 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:59:32,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.62 vs. limit=22.5
2023-11-22 13:59:41,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1969860.0, ans=0.125
2023-11-22 13:59:43,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1969860.0, ans=0.2
2023-11-22 14:00:02,067 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295500
2023-11-22 14:00:04,521 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.094e+01 8.795e+01 9.359e+01 1.337e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 14:00:05,182 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.21 vs. limit=22.5
2023-11-22 14:00:15,057 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:00:23,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1970060.0, ans=0.125
2023-11-22 14:00:30,099 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 6950, loss[loss=0.06205, simple_loss=0.07774, pruned_loss=0.01314, audio_tagging_loss=0.01004, over 16233.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.0942, pruned_loss=0.01501, audio_tagging_loss=0.009173, over 3040779.89 frames. ], batch size: 63, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:00:35,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1970126.6666666667, ans=0.125
2023-11-22 14:01:02,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1970260.0, ans=0.125
2023-11-22 14:01:07,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295550
2023-11-22 14:01:09,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1970326.6666666667, ans=0.125
2023-11-22 14:01:15,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1970326.6666666667, ans=0.125
2023-11-22 14:01:19,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1970326.6666666667, ans=0.125
2023-11-22 14:01:24,458 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.60 vs. limit=15.0
2023-11-22 14:01:33,777 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7000, loss[loss=0.07899, simple_loss=0.104, pruned_loss=0.01725, audio_tagging_loss=0.009731, over 15459.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09435, pruned_loss=0.01508, audio_tagging_loss=0.009186, over 3032754.40 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:01:35,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1970460.0, ans=0.125
2023-11-22 14:01:54,671 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.77 vs. limit=15.0
2023-11-22 14:02:12,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295600
2023-11-22 14:02:12,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1970660.0, ans=0.1
2023-11-22 14:02:14,864 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.282e+01 8.833e+01 9.554e+01 1.299e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 14:02:30,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1970726.6666666667, ans=0.125
2023-11-22 14:02:33,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1970726.6666666667, ans=0.125
2023-11-22 14:02:38,748 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7050, loss[loss=0.07173, simple_loss=0.09245, pruned_loss=0.01629, audio_tagging_loss=0.009214, over 14492.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09407, pruned_loss=0.0149, audio_tagging_loss=0.009183, over 3041299.95 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:03:01,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1970860.0, ans=0.125
2023-11-22 14:03:05,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1970926.6666666667, ans=0.0
2023-11-22 14:03:07,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1970926.6666666667, ans=0.0
2023-11-22 14:03:16,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295650
2023-11-22 14:03:18,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1970993.3333333333, ans=0.09899494936611666
2023-11-22 14:03:27,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1970993.3333333333, ans=0.125
2023-11-22 14:03:43,567 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7100, loss[loss=0.08229, simple_loss=0.1099, pruned_loss=0.01939, audio_tagging_loss=0.007963, over 14527.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09399, pruned_loss=0.01493, audio_tagging_loss=0.009344, over 3038922.66 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:03:56,485 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.68 vs. limit=22.5
2023-11-22 14:04:18,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1971260.0, ans=0.125
2023-11-22 14:04:19,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1971260.0, ans=0.2
2023-11-22 14:04:21,106 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295700
2023-11-22 14:04:21,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1971326.6666666667, ans=0.0
2023-11-22 14:04:23,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1971326.6666666667, ans=0.125
2023-11-22 14:04:24,041 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.394e+01 8.961e+01 9.557e+01 1.218e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 14:04:35,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1971393.3333333333, ans=0.125
2023-11-22 14:04:47,945 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7150, loss[loss=0.06405, simple_loss=0.08763, pruned_loss=0.0122, audio_tagging_loss=0.008035, over 14913.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.0946, pruned_loss=0.01505, audio_tagging_loss=0.009392, over 3046764.32 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:04:49,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1971460.0, ans=0.125
2023-11-22 14:04:59,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1971526.6666666667, ans=0.0
2023-11-22 14:05:11,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1971526.6666666667, ans=15.0
2023-11-22 14:05:15,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1971593.3333333333, ans=0.125
2023-11-22 14:05:16,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1971593.3333333333, ans=0.2
2023-11-22 14:05:18,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1971593.3333333333, ans=0.125
2023-11-22 14:05:21,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1971593.3333333333, ans=0.125
2023-11-22 14:05:26,280 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295750
2023-11-22 14:05:41,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1971726.6666666667, ans=0.0
2023-11-22 14:05:41,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1971726.6666666667, ans=0.1
2023-11-22 14:05:52,776 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7200, loss[loss=0.08697, simple_loss=0.1126, pruned_loss=0.01954, audio_tagging_loss=0.01113, over 14604.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09519, pruned_loss=0.01506, audio_tagging_loss=0.009393, over 3043450.71 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:05:55,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1971793.3333333333, ans=0.0
2023-11-22 14:06:11,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1971860.0, ans=0.125
2023-11-22 14:06:13,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1971860.0, ans=0.0
2023-11-22 14:06:30,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295800
2023-11-22 14:06:32,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.22 vs. limit=15.0
2023-11-22 14:06:33,409 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.672e+01 8.039e+01 8.548e+01 9.116e+01 1.085e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 14:06:43,690 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.96 vs. limit=15.0
2023-11-22 14:06:58,277 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7250, loss[loss=0.06789, simple_loss=0.07842, pruned_loss=0.02099, audio_tagging_loss=0.007689, over 14007.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09449, pruned_loss=0.01489, audio_tagging_loss=0.009454, over 3042284.46 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:07:07,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1972126.6666666667, ans=0.0
2023-11-22 14:07:09,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1972126.6666666667, ans=0.0
2023-11-22 14:07:20,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1972193.3333333333, ans=15.0
2023-11-22 14:07:20,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1972193.3333333333, ans=0.125
2023-11-22 14:07:34,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.02 vs. limit=22.5
2023-11-22 14:07:36,048 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295850
2023-11-22 14:07:39,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1972326.6666666667, ans=0.2
2023-11-22 14:07:55,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1972393.3333333333, ans=0.0
2023-11-22 14:08:03,637 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7300, loss[loss=0.08112, simple_loss=0.1073, pruned_loss=0.01932, audio_tagging_loss=0.008155, over 14898.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09496, pruned_loss=0.01509, audio_tagging_loss=0.009341, over 3046180.46 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:08:03,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1972460.0, ans=0.1
2023-11-22 14:08:09,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.42 vs. limit=15.0
2023-11-22 14:08:15,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1972526.6666666667, ans=0.1
2023-11-22 14:08:28,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1972593.3333333333, ans=0.2
2023-11-22 14:08:38,879 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.26 vs. limit=15.0
2023-11-22 14:08:40,756 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295900
2023-11-22 14:08:44,844 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.276e+01 8.894e+01 9.525e+01 1.269e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 14:09:04,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1972726.6666666667, ans=0.2
2023-11-22 14:09:08,249 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7350, loss[loss=0.0604, simple_loss=0.07122, pruned_loss=0.01444, audio_tagging_loss=0.01035, over 13592.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09403, pruned_loss=0.015, audio_tagging_loss=0.00921, over 3039189.07 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:09:10,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1972793.3333333333, ans=0.0
2023-11-22 14:09:43,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1972926.6666666667, ans=0.1
2023-11-22 14:09:43,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1972926.6666666667, ans=0.0
2023-11-22 14:09:45,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 295950
2023-11-22 14:09:57,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1972993.3333333333, ans=0.1
2023-11-22 14:10:02,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1973060.0, ans=0.0
2023-11-22 14:10:12,118 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7400, loss[loss=0.05336, simple_loss=0.07152, pruned_loss=0.008168, audio_tagging_loss=0.009436, over 14151.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09319, pruned_loss=0.01487, audio_tagging_loss=0.00915, over 3033253.67 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:10:25,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1973193.3333333333, ans=0.125
2023-11-22 14:10:38,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1973260.0, ans=0.1
2023-11-22 14:10:39,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1973260.0, ans=0.125
2023-11-22 14:10:39,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1973260.0, ans=0.1
2023-11-22 14:10:40,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1973260.0, ans=0.125
2023-11-22 14:10:44,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1973260.0, ans=0.0
2023-11-22 14:10:49,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296000
2023-11-22 14:10:56,588 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.500e+01 8.081e+01 8.804e+01 9.585e+01 1.209e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 14:10:58,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1973326.6666666667, ans=0.125
2023-11-22 14:11:04,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1973326.6666666667, ans=0.2
2023-11-22 14:11:11,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1973393.3333333333, ans=0.125
2023-11-22 14:11:19,950 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7450, loss[loss=0.06324, simple_loss=0.0842, pruned_loss=0.01265, audio_tagging_loss=0.008488, over 14558.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09301, pruned_loss=0.0148, audio_tagging_loss=0.009107, over 3038056.85 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:11:37,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1973526.6666666667, ans=0.125
2023-11-22 14:11:55,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1973593.3333333333, ans=0.2
2023-11-22 14:11:57,578 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296050
2023-11-22 14:11:57,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1973660.0, ans=0.0
2023-11-22 14:12:11,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1973726.6666666667, ans=0.125
2023-11-22 14:12:21,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1973726.6666666667, ans=0.125
2023-11-22 14:12:24,569 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7500, loss[loss=0.08152, simple_loss=0.1195, pruned_loss=0.01554, audio_tagging_loss=0.006233, over 15500.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09426, pruned_loss=0.01511, audio_tagging_loss=0.009034, over 3046795.68 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:12:32,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1973793.3333333333, ans=0.2
2023-11-22 14:12:43,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.46 vs. limit=15.0
2023-11-22 14:12:47,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1973860.0, ans=0.0
2023-11-22 14:12:55,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1973926.6666666667, ans=0.1
2023-11-22 14:12:57,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1973926.6666666667, ans=0.0
2023-11-22 14:13:02,134 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296100
2023-11-22 14:13:04,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1973993.3333333333, ans=0.0
2023-11-22 14:13:06,436 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.270e+01 8.171e+01 8.790e+01 9.530e+01 1.185e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 14:13:29,559 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7550, loss[loss=0.08312, simple_loss=0.1138, pruned_loss=0.01854, audio_tagging_loss=0.007687, over 15168.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.0941, pruned_loss=0.01504, audio_tagging_loss=0.009042, over 3048436.08 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:13:32,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1974126.6666666667, ans=10.0
2023-11-22 14:13:37,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1974126.6666666667, ans=0.0
2023-11-22 14:13:45,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.11 vs. limit=15.0
2023-11-22 14:13:51,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1974193.3333333333, ans=0.125
2023-11-22 14:13:55,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1974260.0, ans=0.04949747468305833
2023-11-22 14:13:58,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1974260.0, ans=0.125
2023-11-22 14:14:00,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1974260.0, ans=0.0
2023-11-22 14:14:07,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296150
2023-11-22 14:14:16,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1974326.6666666667, ans=0.1
2023-11-22 14:14:24,761 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:14:33,782 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7600, loss[loss=0.06922, simple_loss=0.09227, pruned_loss=0.01422, audio_tagging_loss=0.008865, over 15128.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09338, pruned_loss=0.01491, audio_tagging_loss=0.009057, over 3048666.78 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:14:34,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1974460.0, ans=0.1
2023-11-22 14:14:44,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.96 vs. limit=22.5
2023-11-22 14:14:48,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1974526.6666666667, ans=0.1
2023-11-22 14:15:11,923 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296200
2023-11-22 14:15:13,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1974660.0, ans=0.125
2023-11-22 14:15:15,770 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.584e+01 8.244e+01 8.772e+01 9.413e+01 1.274e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 14:15:22,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.00 vs. limit=10.0
2023-11-22 14:15:28,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.89 vs. limit=12.0
2023-11-22 14:15:29,577 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.50 vs. limit=15.0
2023-11-22 14:15:35,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1974726.6666666667, ans=0.125
2023-11-22 14:15:39,245 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7650, loss[loss=0.06787, simple_loss=0.09061, pruned_loss=0.01259, audio_tagging_loss=0.009981, over 15497.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09389, pruned_loss=0.01502, audio_tagging_loss=0.009095, over 3045140.59 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:15:43,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1974793.3333333333, ans=0.125
2023-11-22 14:15:51,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1974860.0, ans=0.2
2023-11-22 14:15:58,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1974860.0, ans=0.0
2023-11-22 14:16:12,992 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=15.0
2023-11-22 14:16:17,216 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296250
2023-11-22 14:16:25,811 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:16:32,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1975060.0, ans=0.125
2023-11-22 14:16:38,622 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.13 vs. limit=15.0
2023-11-22 14:16:44,132 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7700, loss[loss=0.06145, simple_loss=0.08357, pruned_loss=0.01036, audio_tagging_loss=0.0093, over 14324.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09311, pruned_loss=0.01491, audio_tagging_loss=0.009175, over 3042168.13 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:17:15,431 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.20 vs. limit=15.0
2023-11-22 14:17:21,568 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296300
2023-11-22 14:17:25,648 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.979e+01 8.266e+01 8.985e+01 9.649e+01 1.326e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 14:17:32,645 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.77 vs. limit=22.5
2023-11-22 14:17:41,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=12.0
2023-11-22 14:17:42,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1975393.3333333333, ans=0.0
2023-11-22 14:17:48,832 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7750, loss[loss=0.05798, simple_loss=0.0713, pruned_loss=0.01259, audio_tagging_loss=0.009742, over 15626.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09397, pruned_loss=0.01526, audio_tagging_loss=0.009171, over 3034316.37 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:17:54,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1975460.0, ans=0.1
2023-11-22 14:17:56,962 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.57 vs. limit=6.0
2023-11-22 14:18:02,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1975526.6666666667, ans=0.07
2023-11-22 14:18:24,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1975593.3333333333, ans=0.1
2023-11-22 14:18:24,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1975593.3333333333, ans=0.0
2023-11-22 14:18:26,582 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296350
2023-11-22 14:18:26,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1975660.0, ans=0.0
2023-11-22 14:18:34,458 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.39 vs. limit=12.0
2023-11-22 14:18:52,672 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7800, loss[loss=0.06405, simple_loss=0.08424, pruned_loss=0.01323, audio_tagging_loss=0.008698, over 14651.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09367, pruned_loss=0.01527, audio_tagging_loss=0.009188, over 3032704.43 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:19:15,908 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:19:19,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1975926.6666666667, ans=0.125
2023-11-22 14:19:29,307 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.91 vs. limit=15.0
2023-11-22 14:19:30,985 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296400
2023-11-22 14:19:36,046 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 7.916e+01 8.644e+01 9.515e+01 1.205e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 14:19:52,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1976060.0, ans=0.1
2023-11-22 14:19:58,146 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7850, loss[loss=0.06351, simple_loss=0.08507, pruned_loss=0.01306, audio_tagging_loss=0.007915, over 15845.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09327, pruned_loss=0.01511, audio_tagging_loss=0.009259, over 3034019.50 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:19:58,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.75 vs. limit=15.0
2023-11-22 14:20:05,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1976126.6666666667, ans=0.0
2023-11-22 14:20:14,510 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:20:17,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.06 vs. limit=15.0
2023-11-22 14:20:18,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.96 vs. limit=12.0
2023-11-22 14:20:24,745 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.01 vs. limit=15.0
2023-11-22 14:20:34,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296450
2023-11-22 14:20:44,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1976326.6666666667, ans=0.1
2023-11-22 14:21:02,739 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7900, loss[loss=0.08241, simple_loss=0.1169, pruned_loss=0.01489, audio_tagging_loss=0.009046, over 15457.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09304, pruned_loss=0.01504, audio_tagging_loss=0.009494, over 3033472.16 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:21:14,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1976526.6666666667, ans=0.0
2023-11-22 14:21:15,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1976526.6666666667, ans=0.1
2023-11-22 14:21:26,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1976593.3333333333, ans=0.0
2023-11-22 14:21:39,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.64 vs. limit=15.0
2023-11-22 14:21:40,347 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296500
2023-11-22 14:21:42,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1976660.0, ans=0.07
2023-11-22 14:21:44,985 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.490e+01 8.875e+01 9.857e+01 1.314e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 14:21:54,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1976726.6666666667, ans=0.125
2023-11-22 14:21:55,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1976726.6666666667, ans=0.2
2023-11-22 14:21:56,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1976726.6666666667, ans=0.2
2023-11-22 14:22:00,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1976726.6666666667, ans=0.1
2023-11-22 14:22:06,367 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 7950, loss[loss=0.07659, simple_loss=0.1049, pruned_loss=0.01537, audio_tagging_loss=0.008767, over 15823.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09319, pruned_loss=0.01516, audio_tagging_loss=0.009608, over 3047823.78 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 8.0
2023-11-22 14:22:14,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.15 vs. limit=15.0
2023-11-22 14:22:23,718 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:22:44,789 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296550
2023-11-22 14:23:10,971 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8000, loss[loss=0.06388, simple_loss=0.08071, pruned_loss=0.01317, audio_tagging_loss=0.01035, over 14968.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09309, pruned_loss=0.01512, audio_tagging_loss=0.009708, over 3049604.77 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:23:11,620 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.87 vs. limit=15.0
2023-11-22 14:23:19,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1977126.6666666667, ans=0.09899494936611666
2023-11-22 14:23:22,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1977126.6666666667, ans=0.2
2023-11-22 14:23:45,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.55 vs. limit=15.0
2023-11-22 14:23:48,285 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296600
2023-11-22 14:23:55,292 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.359e+01 8.469e+01 8.936e+01 9.726e+01 1.360e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 14:24:14,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.65 vs. limit=15.0
2023-11-22 14:24:16,312 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8050, loss[loss=0.07838, simple_loss=0.08826, pruned_loss=0.0221, audio_tagging_loss=0.01215, over 14689.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09268, pruned_loss=0.0151, audio_tagging_loss=0.009835, over 3046862.21 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:24:16,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1977460.0, ans=0.0
2023-11-22 14:24:29,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1977526.6666666667, ans=0.0
2023-11-22 14:24:45,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1977593.3333333333, ans=0.0
2023-11-22 14:24:51,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1977593.3333333333, ans=0.125
2023-11-22 14:24:52,942 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296650
2023-11-22 14:25:09,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.62 vs. limit=6.0
2023-11-22 14:25:14,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1977726.6666666667, ans=0.125
2023-11-22 14:25:20,010 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8100, loss[loss=0.08145, simple_loss=0.1092, pruned_loss=0.01762, audio_tagging_loss=0.009232, over 15107.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.09357, pruned_loss=0.0152, audio_tagging_loss=0.009644, over 3042091.25 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:25:22,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1977793.3333333333, ans=0.125
2023-11-22 14:25:22,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1977793.3333333333, ans=0.0
2023-11-22 14:25:47,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1977926.6666666667, ans=0.1
2023-11-22 14:25:57,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296700
2023-11-22 14:26:03,383 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.691e+01 8.263e+01 8.996e+01 9.612e+01 1.203e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 14:26:06,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1977993.3333333333, ans=0.2
2023-11-22 14:26:12,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1978060.0, ans=0.0
2023-11-22 14:26:15,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1978060.0, ans=0.0
2023-11-22 14:26:23,602 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8150, loss[loss=0.06306, simple_loss=0.08089, pruned_loss=0.01223, audio_tagging_loss=0.01038, over 15628.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09435, pruned_loss=0.01521, audio_tagging_loss=0.009496, over 3045810.79 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:26:26,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1978126.6666666667, ans=0.125
2023-11-22 14:26:27,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1978126.6666666667, ans=0.125
2023-11-22 14:26:31,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1978126.6666666667, ans=0.1
2023-11-22 14:26:32,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1978126.6666666667, ans=0.05
2023-11-22 14:26:43,184 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.35 vs. limit=15.0
2023-11-22 14:26:46,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1978193.3333333333, ans=0.07
2023-11-22 14:26:53,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1978260.0, ans=0.0
2023-11-22 14:26:58,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1978260.0, ans=0.0
2023-11-22 14:27:00,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296750
2023-11-22 14:27:27,874 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8200, loss[loss=0.07201, simple_loss=0.09486, pruned_loss=0.01352, audio_tagging_loss=0.01105, over 14167.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09506, pruned_loss=0.01531, audio_tagging_loss=0.009228, over 3048783.95 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:27:29,132 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:27:50,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1978526.6666666667, ans=0.125
2023-11-22 14:28:04,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296800
2023-11-22 14:28:05,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1978660.0, ans=0.1
2023-11-22 14:28:08,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1978660.0, ans=0.125
2023-11-22 14:28:10,697 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 7.904e+01 8.816e+01 9.431e+01 1.476e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 14:28:15,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1978660.0, ans=0.0
2023-11-22 14:28:25,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1978726.6666666667, ans=0.1
2023-11-22 14:28:30,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1978726.6666666667, ans=0.2
2023-11-22 14:28:32,339 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8250, loss[loss=0.05145, simple_loss=0.06241, pruned_loss=0.008914, audio_tagging_loss=0.01133, over 15694.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09358, pruned_loss=0.01482, audio_tagging_loss=0.009199, over 3046993.92 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:28:33,016 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.43 vs. limit=6.0
2023-11-22 14:28:53,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1978860.0, ans=0.0
2023-11-22 14:29:05,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.71 vs. limit=15.0
2023-11-22 14:29:09,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296850
2023-11-22 14:29:14,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1978993.3333333333, ans=0.125
2023-11-22 14:29:27,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1979060.0, ans=0.125
2023-11-22 14:29:30,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1979060.0, ans=0.125
2023-11-22 14:29:32,407 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.87 vs. limit=15.0
2023-11-22 14:29:35,950 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8300, loss[loss=0.05589, simple_loss=0.06901, pruned_loss=0.0118, audio_tagging_loss=0.009584, over 14709.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09245, pruned_loss=0.01458, audio_tagging_loss=0.009178, over 3047032.09 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 8.0
2023-11-22 14:29:46,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1979193.3333333333, ans=0.125
2023-11-22 14:29:51,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1979193.3333333333, ans=0.125
2023-11-22 14:29:55,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.58 vs. limit=15.0
2023-11-22 14:30:11,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.42 vs. limit=10.0
2023-11-22 14:30:13,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296900
2023-11-22 14:30:20,390 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.061e+01 8.620e+01 9.639e+01 1.216e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-22 14:30:22,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1979326.6666666667, ans=0.0
2023-11-22 14:30:23,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1979326.6666666667, ans=0.05
2023-11-22 14:30:39,596 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8350, loss[loss=0.08457, simple_loss=0.1054, pruned_loss=0.0228, audio_tagging_loss=0.009068, over 15997.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09281, pruned_loss=0.01471, audio_tagging_loss=0.009175, over 3045676.28 frames. ], batch size: 60, lr: 2.72e-03, grad_scale: 8.0
2023-11-22 14:30:43,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1979460.0, ans=0.2
2023-11-22 14:30:56,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1979526.6666666667, ans=0.0
2023-11-22 14:31:06,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1979593.3333333333, ans=0.125
2023-11-22 14:31:14,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1979593.3333333333, ans=0.0
2023-11-22 14:31:15,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1979593.3333333333, ans=0.0
2023-11-22 14:31:17,028 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 296950
2023-11-22 14:31:38,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1979726.6666666667, ans=0.0
2023-11-22 14:31:43,416 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8400, loss[loss=0.0747, simple_loss=0.09862, pruned_loss=0.01679, audio_tagging_loss=0.008597, over 15078.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09369, pruned_loss=0.01495, audio_tagging_loss=0.009163, over 3055141.58 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:31:43,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1979793.3333333333, ans=0.1
2023-11-22 14:31:46,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1979793.3333333333, ans=0.125
2023-11-22 14:32:20,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297000
2023-11-22 14:32:24,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1979993.3333333333, ans=0.125
2023-11-22 14:32:29,034 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.093e+01 8.742e+01 9.671e+01 1.399e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 14:32:47,266 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8450, loss[loss=0.06894, simple_loss=0.1001, pruned_loss=0.01096, audio_tagging_loss=0.007953, over 15058.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09299, pruned_loss=0.01479, audio_tagging_loss=0.009248, over 3051576.40 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:32:55,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.73 vs. limit=15.0
2023-11-22 14:32:56,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1980126.6666666667, ans=0.0
2023-11-22 14:32:57,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1980126.6666666667, ans=0.2
2023-11-22 14:33:05,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1980193.3333333333, ans=0.5
2023-11-22 14:33:15,259 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.66 vs. limit=15.0
2023-11-22 14:33:25,267 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297050
2023-11-22 14:33:34,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1980326.6666666667, ans=0.0
2023-11-22 14:33:49,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1980393.3333333333, ans=0.2
2023-11-22 14:33:52,049 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8500, loss[loss=0.06637, simple_loss=0.08024, pruned_loss=0.0159, audio_tagging_loss=0.01035, over 15055.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09284, pruned_loss=0.01486, audio_tagging_loss=0.009233, over 3055133.63 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:34:03,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1980526.6666666667, ans=0.125
2023-11-22 14:34:05,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1980526.6666666667, ans=0.125
2023-11-22 14:34:15,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1980526.6666666667, ans=0.1
2023-11-22 14:34:16,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1980593.3333333333, ans=0.0
2023-11-22 14:34:20,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1980593.3333333333, ans=0.09899494936611666
2023-11-22 14:34:20,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1980593.3333333333, ans=0.125
2023-11-22 14:34:23,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.12 vs. limit=22.5
2023-11-22 14:34:28,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0
2023-11-22 14:34:29,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297100
2023-11-22 14:34:36,238 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.259e+01 8.972e+01 9.645e+01 1.298e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 14:34:41,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1980726.6666666667, ans=0.125
2023-11-22 14:34:55,711 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8550, loss[loss=0.08123, simple_loss=0.1101, pruned_loss=0.01708, audio_tagging_loss=0.009099, over 15143.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.0938, pruned_loss=0.01483, audio_tagging_loss=0.009242, over 3058314.64 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:35:33,166 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297150
2023-11-22 14:35:57,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1981060.0, ans=0.125
2023-11-22 14:35:59,851 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8600, loss[loss=0.07483, simple_loss=0.1004, pruned_loss=0.0185, audio_tagging_loss=0.006149, over 15302.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09499, pruned_loss=0.01517, audio_tagging_loss=0.009208, over 3055888.87 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:36:16,649 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.71 vs. limit=15.0
2023-11-22 14:36:18,780 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0
2023-11-22 14:36:37,022 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297200
2023-11-22 14:36:37,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1981326.6666666667, ans=0.125
2023-11-22 14:36:45,262 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 8.334e+01 9.004e+01 9.545e+01 1.536e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 14:36:46,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1981326.6666666667, ans=0.025
2023-11-22 14:36:53,411 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.57 vs. limit=15.0
2023-11-22 14:36:57,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1981393.3333333333, ans=0.125
2023-11-22 14:37:00,078 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.80 vs. limit=10.0
2023-11-22 14:37:04,297 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8650, loss[loss=0.07446, simple_loss=0.09381, pruned_loss=0.01633, audio_tagging_loss=0.01123, over 16047.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.0939, pruned_loss=0.01505, audio_tagging_loss=0.009334, over 3050304.56 frames. ], batch size: 64, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:37:13,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1981460.0, ans=0.2
2023-11-22 14:37:15,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.06 vs. limit=12.0
2023-11-22 14:37:18,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1981526.6666666667, ans=0.025
2023-11-22 14:37:29,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1981593.3333333333, ans=0.125
2023-11-22 14:37:36,559 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:37:41,390 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297250
2023-11-22 14:37:48,530 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0
2023-11-22 14:37:52,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.80 vs. limit=15.0
2023-11-22 14:38:06,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1981726.6666666667, ans=0.0
2023-11-22 14:38:07,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1981793.3333333333, ans=0.04949747468305833
2023-11-22 14:38:08,738 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8700, loss[loss=0.07472, simple_loss=0.09863, pruned_loss=0.01678, audio_tagging_loss=0.008629, over 15778.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09407, pruned_loss=0.01497, audio_tagging_loss=0.009343, over 3049648.28 frames. ], batch size: 59, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:38:13,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1981793.3333333333, ans=0.125
2023-11-22 14:38:24,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1981860.0, ans=0.2
2023-11-22 14:38:32,296 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.47 vs. limit=12.0
2023-11-22 14:38:41,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1981926.6666666667, ans=0.125
2023-11-22 14:38:45,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297300
2023-11-22 14:38:53,651 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.337e+01 8.929e+01 9.655e+01 1.210e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 14:38:56,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1981993.3333333333, ans=0.125
2023-11-22 14:39:06,248 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:39:07,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1982060.0, ans=0.2
2023-11-22 14:39:12,686 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8750, loss[loss=0.08575, simple_loss=0.1128, pruned_loss=0.01902, audio_tagging_loss=0.0103, over 14752.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09459, pruned_loss=0.01512, audio_tagging_loss=0.009373, over 3048133.20 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:39:25,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.79 vs. limit=15.0
2023-11-22 14:39:29,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1982193.3333333333, ans=0.2
2023-11-22 14:39:38,769 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.33 vs. limit=22.5
2023-11-22 14:39:49,729 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297350
2023-11-22 14:39:54,089 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.25 vs. limit=15.0
2023-11-22 14:40:07,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1982393.3333333333, ans=0.0
2023-11-22 14:40:16,886 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8800, loss[loss=0.06008, simple_loss=0.07571, pruned_loss=0.01369, audio_tagging_loss=0.008533, over 15657.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09455, pruned_loss=0.01516, audio_tagging_loss=0.00948, over 3047678.64 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:40:23,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1982460.0, ans=0.0
2023-11-22 14:40:26,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1982460.0, ans=0.125
2023-11-22 14:40:27,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1982460.0, ans=0.0
2023-11-22 14:40:28,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1982526.6666666667, ans=0.07
2023-11-22 14:40:33,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.96 vs. limit=15.0
2023-11-22 14:40:53,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297400
2023-11-22 14:40:58,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1982660.0, ans=0.125
2023-11-22 14:41:04,592 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.228e+01 8.972e+01 9.857e+01 1.350e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 14:41:17,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=1982726.6666666667, ans=0.02
2023-11-22 14:41:21,469 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8850, loss[loss=0.06558, simple_loss=0.09066, pruned_loss=0.01268, audio_tagging_loss=0.007572, over 15094.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09549, pruned_loss=0.01533, audio_tagging_loss=0.009424, over 3051383.16 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:41:33,605 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:41:47,460 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.02 vs. limit=15.0
2023-11-22 14:41:52,226 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:41:53,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.41 vs. limit=22.5
2023-11-22 14:41:56,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1982926.6666666667, ans=0.0
2023-11-22 14:41:58,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297450
2023-11-22 14:41:59,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1982993.3333333333, ans=0.125
2023-11-22 14:42:03,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1982993.3333333333, ans=0.1
2023-11-22 14:42:03,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1982993.3333333333, ans=0.0
2023-11-22 14:42:20,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1983060.0, ans=0.0
2023-11-22 14:42:24,239 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8900, loss[loss=0.07175, simple_loss=0.0908, pruned_loss=0.01409, audio_tagging_loss=0.01225, over 15808.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09558, pruned_loss=0.01545, audio_tagging_loss=0.009344, over 3053267.15 frames. ], batch size: 61, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:42:51,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1983260.0, ans=0.0
2023-11-22 14:42:55,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1983260.0, ans=0.2
2023-11-22 14:43:01,950 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297500
2023-11-22 14:43:10,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1983326.6666666667, ans=0.125
2023-11-22 14:43:11,664 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.250e+01 8.702e+01 9.316e+01 1.378e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 14:43:25,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1983393.3333333333, ans=0.04949747468305833
2023-11-22 14:43:28,589 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 8950, loss[loss=0.05239, simple_loss=0.06329, pruned_loss=0.009005, audio_tagging_loss=0.01174, over 16249.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09626, pruned_loss=0.0155, audio_tagging_loss=0.009183, over 3051202.28 frames. ], batch size: 62, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:43:30,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1983460.0, ans=0.125
2023-11-22 14:43:31,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1983460.0, ans=0.0
2023-11-22 14:43:54,154 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:43:55,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1983593.3333333333, ans=0.0
2023-11-22 14:44:02,909 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.95 vs. limit=15.0
2023-11-22 14:44:04,732 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297550
2023-11-22 14:44:09,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1983660.0, ans=0.125
2023-11-22 14:44:14,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1983660.0, ans=0.125
2023-11-22 14:44:14,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1983660.0, ans=0.0
2023-11-22 14:44:22,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1983726.6666666667, ans=0.0
2023-11-22 14:44:29,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1983726.6666666667, ans=0.2
2023-11-22 14:44:31,326 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9000, loss[loss=0.08604, simple_loss=0.1223, pruned_loss=0.01913, audio_tagging_loss=0.00576, over 15056.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.097, pruned_loss=0.01569, audio_tagging_loss=0.009112, over 3050749.12 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:44:31,327 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 14:45:11,818 INFO [train_asr.py:1253] (2/4) Epoch 25, validation: loss=0.06003, simple_loss=0.05148, pruned_loss=0.00513, audio_tagging_loss=0.02916, over 4681554.00 frames. 
2023-11-22 14:45:11,819 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 14:45:49,638 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297600
2023-11-22 14:45:51,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1983993.3333333333, ans=0.0
2023-11-22 14:45:59,589 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.485e+01 9.145e+01 9.654e+01 1.276e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-22 14:46:16,682 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9050, loss[loss=0.06259, simple_loss=0.0756, pruned_loss=0.01239, audio_tagging_loss=0.0124, over 16483.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09657, pruned_loss=0.01564, audio_tagging_loss=0.009043, over 3050211.16 frames. ], batch size: 63, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:46:49,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1984260.0, ans=0.0
2023-11-22 14:46:49,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1984260.0, ans=0.125
2023-11-22 14:46:53,076 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297650
2023-11-22 14:47:09,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1984393.3333333333, ans=0.0
2023-11-22 14:47:17,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1984393.3333333333, ans=0.0
2023-11-22 14:47:21,057 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9100, loss[loss=0.0638, simple_loss=0.0853, pruned_loss=0.01194, audio_tagging_loss=0.00921, over 14786.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09604, pruned_loss=0.01536, audio_tagging_loss=0.009184, over 3055845.24 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:47:32,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1984526.6666666667, ans=0.125
2023-11-22 14:47:36,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1984526.6666666667, ans=0.0
2023-11-22 14:47:43,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1984526.6666666667, ans=0.125
2023-11-22 14:47:57,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297700
2023-11-22 14:48:07,992 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.161e+01 8.752e+01 9.286e+01 2.425e+02, threshold=1.750e+02, percent-clipped=1.0
2023-11-22 14:48:24,404 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9150, loss[loss=0.06173, simple_loss=0.08546, pruned_loss=0.01059, audio_tagging_loss=0.008416, over 14691.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09522, pruned_loss=0.01508, audio_tagging_loss=0.009182, over 3053933.25 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:48:48,629 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.50 vs. limit=15.0
2023-11-22 14:49:02,438 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297750
2023-11-22 14:49:14,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1985060.0, ans=0.125
2023-11-22 14:49:28,671 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9200, loss[loss=0.06799, simple_loss=0.08964, pruned_loss=0.0158, audio_tagging_loss=0.007371, over 16173.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09562, pruned_loss=0.0153, audio_tagging_loss=0.009054, over 3057579.40 frames. ], batch size: 63, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:49:42,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.72 vs. limit=15.0
2023-11-22 14:49:48,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1985193.3333333333, ans=0.125
2023-11-22 14:49:50,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1985193.3333333333, ans=0.1
2023-11-22 14:50:05,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297800
2023-11-22 14:50:16,264 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.283e+01 9.032e+01 9.826e+01 1.218e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 14:50:23,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=15.0
2023-11-22 14:50:30,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1985393.3333333333, ans=0.0
2023-11-22 14:50:31,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1985393.3333333333, ans=0.125
2023-11-22 14:50:33,428 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9250, loss[loss=0.06138, simple_loss=0.07292, pruned_loss=0.01322, audio_tagging_loss=0.01171, over 15280.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09622, pruned_loss=0.01518, audio_tagging_loss=0.009091, over 3057020.37 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:50:39,324 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:50:41,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1985460.0, ans=0.125
2023-11-22 14:50:51,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1985526.6666666667, ans=0.0
2023-11-22 14:51:01,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1985593.3333333333, ans=0.125
2023-11-22 14:51:10,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297850
2023-11-22 14:51:17,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1985660.0, ans=0.0
2023-11-22 14:51:28,686 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.43 vs. limit=15.0
2023-11-22 14:51:37,704 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9300, loss[loss=0.09128, simple_loss=0.1193, pruned_loss=0.02109, audio_tagging_loss=0.01053, over 14573.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09515, pruned_loss=0.01498, audio_tagging_loss=0.009108, over 3056358.98 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:51:44,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1985793.3333333333, ans=0.125
2023-11-22 14:52:07,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1985926.6666666667, ans=0.125
2023-11-22 14:52:09,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1985926.6666666667, ans=0.0
2023-11-22 14:52:14,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297900
2023-11-22 14:52:23,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1985993.3333333333, ans=0.125
2023-11-22 14:52:24,528 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.158e+01 8.859e+01 9.535e+01 1.280e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 14:52:41,519 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9350, loss[loss=0.08211, simple_loss=0.1132, pruned_loss=0.01652, audio_tagging_loss=0.008984, over 15782.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09522, pruned_loss=0.01503, audio_tagging_loss=0.009195, over 3054859.58 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:52:53,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1986193.3333333333, ans=0.125
2023-11-22 14:53:02,303 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:53:12,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1986260.0, ans=0.0
2023-11-22 14:53:20,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 297950
2023-11-22 14:53:27,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.44 vs. limit=15.0
2023-11-22 14:53:33,301 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:53:39,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.30 vs. limit=15.0
2023-11-22 14:53:48,022 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9400, loss[loss=0.08339, simple_loss=0.1158, pruned_loss=0.019, audio_tagging_loss=0.006473, over 15257.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.0952, pruned_loss=0.01489, audio_tagging_loss=0.009246, over 3053070.52 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:53:49,879 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.07 vs. limit=15.0
2023-11-22 14:54:24,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298000
2023-11-22 14:54:35,748 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.116e+01 8.691e+01 9.663e+01 1.322e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 14:54:43,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1986726.6666666667, ans=10.0
2023-11-22 14:54:52,095 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:54:53,325 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9450, loss[loss=0.05948, simple_loss=0.06844, pruned_loss=0.01483, audio_tagging_loss=0.01044, over 14625.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09401, pruned_loss=0.01484, audio_tagging_loss=0.009399, over 3051419.02 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:54:56,308 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.93 vs. limit=12.0
2023-11-22 14:55:14,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1986860.0, ans=0.2
2023-11-22 14:55:20,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1986926.6666666667, ans=0.125
2023-11-22 14:55:21,448 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.44 vs. limit=22.5
2023-11-22 14:55:24,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1986926.6666666667, ans=0.125
2023-11-22 14:55:30,960 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298050
2023-11-22 14:55:57,817 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9500, loss[loss=0.0777, simple_loss=0.1018, pruned_loss=0.01819, audio_tagging_loss=0.0086, over 15310.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09434, pruned_loss=0.01501, audio_tagging_loss=0.009401, over 3058242.08 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:56:05,299 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:56:05,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1987126.6666666667, ans=0.125
2023-11-22 14:56:15,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1987193.3333333333, ans=0.0
2023-11-22 14:56:35,392 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298100
2023-11-22 14:56:38,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1987326.6666666667, ans=0.0
2023-11-22 14:56:45,216 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.353e+01 9.126e+01 1.009e+02 1.546e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-22 14:56:45,875 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.63 vs. limit=10.0
2023-11-22 14:56:47,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1987326.6666666667, ans=0.125
2023-11-22 14:56:52,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1987393.3333333333, ans=0.0
2023-11-22 14:56:55,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1987393.3333333333, ans=0.125
2023-11-22 14:57:02,711 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9550, loss[loss=0.06102, simple_loss=0.0818, pruned_loss=0.01068, audio_tagging_loss=0.009446, over 15776.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.0948, pruned_loss=0.01512, audio_tagging_loss=0.009429, over 3062569.28 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:57:04,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1987460.0, ans=0.1
2023-11-22 14:57:18,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1987526.6666666667, ans=0.07
2023-11-22 14:57:27,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1987593.3333333333, ans=0.2
2023-11-22 14:57:41,037 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298150
2023-11-22 14:58:09,001 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9600, loss[loss=0.05998, simple_loss=0.08217, pruned_loss=0.01139, audio_tagging_loss=0.007504, over 16670.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09502, pruned_loss=0.01497, audio_tagging_loss=0.009459, over 3069331.73 frames. ], batch size: 64, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 14:58:18,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.85 vs. limit=15.0
2023-11-22 14:58:44,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.74 vs. limit=15.0
2023-11-22 14:58:45,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1987926.6666666667, ans=0.125
2023-11-22 14:58:46,270 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298200
2023-11-22 14:58:57,429 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.183e+01 8.755e+01 9.694e+01 2.318e+02, threshold=1.751e+02, percent-clipped=1.0
2023-11-22 14:59:01,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1988060.0, ans=0.0
2023-11-22 14:59:11,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.67 vs. limit=10.0
2023-11-22 14:59:13,474 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9650, loss[loss=0.08032, simple_loss=0.1055, pruned_loss=0.01915, audio_tagging_loss=0.008432, over 15408.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.0941, pruned_loss=0.01467, audio_tagging_loss=0.009432, over 3065721.27 frames. ], batch size: 59, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 14:59:34,866 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.12 vs. limit=10.0
2023-11-22 14:59:51,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298250
2023-11-22 14:59:58,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1988326.6666666667, ans=0.125
2023-11-22 15:00:05,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1988393.3333333333, ans=0.95
2023-11-22 15:00:17,257 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9700, loss[loss=0.07924, simple_loss=0.1017, pruned_loss=0.02146, audio_tagging_loss=0.006937, over 15454.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09387, pruned_loss=0.01475, audio_tagging_loss=0.009342, over 3063303.12 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:00:25,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1988460.0, ans=0.125
2023-11-22 15:00:44,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1988593.3333333333, ans=0.95
2023-11-22 15:00:55,305 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298300
2023-11-22 15:00:59,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1988660.0, ans=0.125
2023-11-22 15:01:04,933 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.285e+01 8.809e+01 9.678e+01 1.163e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 15:01:06,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1988660.0, ans=0.09899494936611666
2023-11-22 15:01:11,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1988726.6666666667, ans=0.125
2023-11-22 15:01:13,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1988726.6666666667, ans=0.1
2023-11-22 15:01:18,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1988726.6666666667, ans=10.0
2023-11-22 15:01:21,866 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9750, loss[loss=0.09489, simple_loss=0.134, pruned_loss=0.02192, audio_tagging_loss=0.005993, over 15054.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09378, pruned_loss=0.01475, audio_tagging_loss=0.009234, over 3062378.77 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:01:46,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1988926.6666666667, ans=10.0
2023-11-22 15:01:47,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1988926.6666666667, ans=0.2
2023-11-22 15:01:58,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298350
2023-11-22 15:02:01,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1988993.3333333333, ans=0.125
2023-11-22 15:02:13,234 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=15.0
2023-11-22 15:02:20,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1989060.0, ans=0.1
2023-11-22 15:02:25,379 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9800, loss[loss=0.08982, simple_loss=0.1163, pruned_loss=0.02332, audio_tagging_loss=0.008345, over 14732.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09342, pruned_loss=0.01473, audio_tagging_loss=0.009298, over 3049921.32 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:02:29,711 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.35 vs. limit=15.0
2023-11-22 15:02:32,496 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.83 vs. limit=10.0
2023-11-22 15:02:59,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1989260.0, ans=0.125
2023-11-22 15:03:00,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1989260.0, ans=0.125
2023-11-22 15:03:02,461 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298400
2023-11-22 15:03:14,256 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.717e+01 7.942e+01 8.704e+01 9.561e+01 1.255e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 15:03:23,525 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:03:29,595 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9850, loss[loss=0.08458, simple_loss=0.1094, pruned_loss=0.02194, audio_tagging_loss=0.007922, over 15003.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09515, pruned_loss=0.0151, audio_tagging_loss=0.009159, over 3051443.18 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:03:29,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1989460.0, ans=0.1
2023-11-22 15:03:35,769 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.79 vs. limit=10.0
2023-11-22 15:03:38,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1989460.0, ans=0.2
2023-11-22 15:03:49,576 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.24 vs. limit=22.5
2023-11-22 15:03:50,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1989526.6666666667, ans=0.125
2023-11-22 15:03:52,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1989526.6666666667, ans=0.125
2023-11-22 15:04:02,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1989593.3333333333, ans=0.125
2023-11-22 15:04:06,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298450
2023-11-22 15:04:06,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1989660.0, ans=0.0
2023-11-22 15:04:09,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1989660.0, ans=0.1
2023-11-22 15:04:19,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1989726.6666666667, ans=0.2
2023-11-22 15:04:33,836 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9900, loss[loss=0.08396, simple_loss=0.1082, pruned_loss=0.02047, audio_tagging_loss=0.009382, over 15557.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09494, pruned_loss=0.01514, audio_tagging_loss=0.009189, over 3055437.52 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:04:49,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1989860.0, ans=0.0
2023-11-22 15:04:57,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1989860.0, ans=0.125
2023-11-22 15:05:06,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1989926.6666666667, ans=0.125
2023-11-22 15:05:10,766 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298500
2023-11-22 15:05:23,015 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.052e+01 8.333e+01 9.011e+01 9.582e+01 1.423e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-22 15:05:37,736 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 9950, loss[loss=0.08186, simple_loss=0.1155, pruned_loss=0.01736, audio_tagging_loss=0.006748, over 15991.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09421, pruned_loss=0.01491, audio_tagging_loss=0.009209, over 3057061.86 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:06:15,358 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298550
2023-11-22 15:06:34,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1990393.3333333333, ans=0.125
2023-11-22 15:06:34,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1990393.3333333333, ans=0.0
2023-11-22 15:06:42,059 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10000, loss[loss=0.08225, simple_loss=0.1144, pruned_loss=0.01645, audio_tagging_loss=0.008595, over 15670.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09402, pruned_loss=0.01494, audio_tagging_loss=0.009237, over 3050720.88 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:06:45,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1990460.0, ans=0.0
2023-11-22 15:06:46,490 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.01 vs. limit=15.0
2023-11-22 15:06:56,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1990526.6666666667, ans=0.09899494936611666
2023-11-22 15:07:03,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1990526.6666666667, ans=0.2
2023-11-22 15:07:04,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.56 vs. limit=22.5
2023-11-22 15:07:06,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1990593.3333333333, ans=0.0
2023-11-22 15:07:19,523 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298600
2023-11-22 15:07:22,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1990660.0, ans=0.2
2023-11-22 15:07:31,287 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.143e+01 8.830e+01 9.648e+01 3.146e+02, threshold=1.766e+02, percent-clipped=1.0
2023-11-22 15:07:47,381 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10050, loss[loss=0.05692, simple_loss=0.07068, pruned_loss=0.0112, audio_tagging_loss=0.01038, over 14235.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09362, pruned_loss=0.01483, audio_tagging_loss=0.009252, over 3051942.17 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:08:17,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1990926.6666666667, ans=0.0
2023-11-22 15:08:24,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298650
2023-11-22 15:08:28,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1990993.3333333333, ans=0.0
2023-11-22 15:08:33,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1990993.3333333333, ans=0.125
2023-11-22 15:08:37,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1991060.0, ans=0.125
2023-11-22 15:08:51,068 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10100, loss[loss=0.06731, simple_loss=0.08198, pruned_loss=0.01344, audio_tagging_loss=0.01287, over 14513.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09346, pruned_loss=0.01464, audio_tagging_loss=0.009325, over 3050671.11 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:09:03,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0
2023-11-22 15:09:25,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.08 vs. limit=15.0
2023-11-22 15:09:26,270 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.83 vs. limit=15.0
2023-11-22 15:09:28,245 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298700
2023-11-22 15:09:39,082 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.326e+01 8.430e+01 8.908e+01 9.951e+01 1.277e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 15:09:41,541 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:09:54,829 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10150, loss[loss=0.06376, simple_loss=0.08516, pruned_loss=0.01073, audio_tagging_loss=0.01045, over 15542.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09268, pruned_loss=0.01454, audio_tagging_loss=0.009371, over 3046244.18 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:09:55,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1991460.0, ans=0.1
2023-11-22 15:09:59,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1991460.0, ans=0.125
2023-11-22 15:09:59,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1991460.0, ans=0.125
2023-11-22 15:10:02,712 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:10:11,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1991526.6666666667, ans=0.125
2023-11-22 15:10:24,413 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:10:30,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.49 vs. limit=15.0
2023-11-22 15:10:31,750 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298750
2023-11-22 15:10:39,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1991660.0, ans=0.125
2023-11-22 15:10:51,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1991726.6666666667, ans=0.0
2023-11-22 15:10:53,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1991726.6666666667, ans=0.2
2023-11-22 15:10:58,813 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10200, loss[loss=0.05237, simple_loss=0.07221, pruned_loss=0.007885, audio_tagging_loss=0.008377, over 15385.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09313, pruned_loss=0.0146, audio_tagging_loss=0.009412, over 3043110.95 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:10:58,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1991793.3333333333, ans=0.0
2023-11-22 15:11:19,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1991860.0, ans=0.0
2023-11-22 15:11:21,505 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:11:21,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1991860.0, ans=0.125
2023-11-22 15:11:28,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1991926.6666666667, ans=0.95
2023-11-22 15:11:35,987 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298800
2023-11-22 15:11:48,668 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.315e+01 8.823e+01 9.660e+01 1.198e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 15:12:02,803 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10250, loss[loss=0.08397, simple_loss=0.1107, pruned_loss=0.0199, audio_tagging_loss=0.008732, over 14952.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09294, pruned_loss=0.0147, audio_tagging_loss=0.009614, over 3038769.91 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:12:30,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1992260.0, ans=0.125
2023-11-22 15:12:32,795 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.08 vs. limit=6.0
2023-11-22 15:12:40,219 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298850
2023-11-22 15:12:53,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1992393.3333333333, ans=0.125
2023-11-22 15:13:06,229 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10300, loss[loss=0.07468, simple_loss=0.1018, pruned_loss=0.01497, audio_tagging_loss=0.008829, over 15029.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09332, pruned_loss=0.01484, audio_tagging_loss=0.00957, over 3033678.30 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:13:22,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1992526.6666666667, ans=0.125
2023-11-22 15:13:33,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1992593.3333333333, ans=0.125
2023-11-22 15:13:43,228 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298900
2023-11-22 15:13:52,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.28 vs. limit=6.0
2023-11-22 15:13:54,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1992660.0, ans=0.125
2023-11-22 15:13:55,927 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.205e+01 8.813e+01 9.475e+01 1.270e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 15:14:10,581 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10350, loss[loss=0.08119, simple_loss=0.1008, pruned_loss=0.02219, audio_tagging_loss=0.008615, over 14836.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09434, pruned_loss=0.01502, audio_tagging_loss=0.009536, over 3041422.21 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:14:22,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1992860.0, ans=0.125
2023-11-22 15:14:25,063 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0
2023-11-22 15:14:33,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1992860.0, ans=0.0
2023-11-22 15:14:47,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 298950
2023-11-22 15:14:57,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.79 vs. limit=6.0
2023-11-22 15:15:04,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1993060.0, ans=0.125
2023-11-22 15:15:07,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1993060.0, ans=0.125
2023-11-22 15:15:08,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1993060.0, ans=0.125
2023-11-22 15:15:14,427 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10400, loss[loss=0.07094, simple_loss=0.09517, pruned_loss=0.01383, audio_tagging_loss=0.009527, over 16301.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09481, pruned_loss=0.01504, audio_tagging_loss=0.009626, over 3042649.60 frames. ], batch size: 61, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:15:49,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1993260.0, ans=0.0
2023-11-22 15:15:51,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299000
2023-11-22 15:16:03,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1993326.6666666667, ans=0.125
2023-11-22 15:16:05,353 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.066e+01 8.819e+01 9.372e+01 1.193e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 15:16:18,138 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10450, loss[loss=0.07926, simple_loss=0.1012, pruned_loss=0.01886, audio_tagging_loss=0.009783, over 15872.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09457, pruned_loss=0.01491, audio_tagging_loss=0.009539, over 3046322.44 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:16:22,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1993460.0, ans=0.1
2023-11-22 15:16:25,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1993460.0, ans=0.125
2023-11-22 15:16:43,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1993593.3333333333, ans=0.125
2023-11-22 15:16:44,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1993593.3333333333, ans=0.2
2023-11-22 15:16:51,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1993593.3333333333, ans=0.05
2023-11-22 15:16:54,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1993593.3333333333, ans=0.0
2023-11-22 15:16:55,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299050
2023-11-22 15:16:59,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.37 vs. limit=10.0
2023-11-22 15:17:21,641 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10500, loss[loss=0.08214, simple_loss=0.1181, pruned_loss=0.01422, audio_tagging_loss=0.008845, over 15770.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09471, pruned_loss=0.01505, audio_tagging_loss=0.00944, over 3039416.24 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:17:28,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.37 vs. limit=15.0
2023-11-22 15:17:38,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1993860.0, ans=0.0
2023-11-22 15:17:51,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1993926.6666666667, ans=0.125
2023-11-22 15:17:51,336 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.40 vs. limit=15.0
2023-11-22 15:17:58,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299100
2023-11-22 15:18:13,202 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.803e+01 8.251e+01 8.969e+01 9.494e+01 1.165e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 15:18:25,934 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10550, loss[loss=0.08116, simple_loss=0.1131, pruned_loss=0.01623, audio_tagging_loss=0.008367, over 16044.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09509, pruned_loss=0.01517, audio_tagging_loss=0.009339, over 3044342.10 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:18:27,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1994126.6666666667, ans=0.125
2023-11-22 15:18:59,619 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-22 15:19:03,316 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299150
2023-11-22 15:19:10,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1994326.6666666667, ans=0.0
2023-11-22 15:19:13,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1994326.6666666667, ans=0.07
2023-11-22 15:19:19,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1994393.3333333333, ans=0.0
2023-11-22 15:19:28,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1994460.0, ans=0.1
2023-11-22 15:19:29,002 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10600, loss[loss=0.07968, simple_loss=0.107, pruned_loss=0.01565, audio_tagging_loss=0.01052, over 16507.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09557, pruned_loss=0.01532, audio_tagging_loss=0.009315, over 3048124.96 frames. ], batch size: 65, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:19:29,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1994460.0, ans=0.125
2023-11-22 15:19:31,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1994460.0, ans=0.2
2023-11-22 15:19:32,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1994460.0, ans=0.0
2023-11-22 15:19:49,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1994526.6666666667, ans=0.2
2023-11-22 15:20:00,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1994593.3333333333, ans=0.125
2023-11-22 15:20:01,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1994593.3333333333, ans=0.0
2023-11-22 15:20:06,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299200
2023-11-22 15:20:18,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.86 vs. limit=12.0
2023-11-22 15:20:20,925 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.207e+01 8.203e+01 8.764e+01 9.272e+01 1.300e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 15:20:32,954 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10650, loss[loss=0.05542, simple_loss=0.06498, pruned_loss=0.01358, audio_tagging_loss=0.00935, over 13585.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.0951, pruned_loss=0.01523, audio_tagging_loss=0.009284, over 3040504.00 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:20:41,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1994793.3333333333, ans=0.2
2023-11-22 15:20:48,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1994860.0, ans=0.0
2023-11-22 15:20:50,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1994860.0, ans=0.125
2023-11-22 15:20:52,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1994860.0, ans=0.125
2023-11-22 15:20:57,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1994926.6666666667, ans=0.125
2023-11-22 15:21:06,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=1994926.6666666667, ans=0.02
2023-11-22 15:21:10,968 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299250
2023-11-22 15:21:13,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1994993.3333333333, ans=0.125
2023-11-22 15:21:19,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1994993.3333333333, ans=0.0
2023-11-22 15:21:27,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1995060.0, ans=0.125
2023-11-22 15:21:28,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1995060.0, ans=0.2
2023-11-22 15:21:28,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1995060.0, ans=0.125
2023-11-22 15:21:37,633 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10700, loss[loss=0.05658, simple_loss=0.05992, pruned_loss=0.01077, audio_tagging_loss=0.01585, over 13573.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09337, pruned_loss=0.01486, audio_tagging_loss=0.009357, over 3035351.86 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:21:42,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1995126.6666666667, ans=0.1
2023-11-22 15:21:52,766 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.63 vs. limit=15.0
2023-11-22 15:22:10,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1995260.0, ans=0.0
2023-11-22 15:22:14,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299300
2023-11-22 15:22:28,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.250e+01 8.791e+01 9.452e+01 1.180e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 15:22:28,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1995393.3333333333, ans=0.125
2023-11-22 15:22:30,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1995393.3333333333, ans=0.0
2023-11-22 15:22:38,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1995393.3333333333, ans=0.125
2023-11-22 15:22:40,684 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10750, loss[loss=0.05788, simple_loss=0.07711, pruned_loss=0.01227, audio_tagging_loss=0.007057, over 16121.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09399, pruned_loss=0.01496, audio_tagging_loss=0.009242, over 3040378.83 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:22:57,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1995526.6666666667, ans=0.07
2023-11-22 15:23:18,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299350
2023-11-22 15:23:43,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1995793.3333333333, ans=0.125
2023-11-22 15:23:44,103 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10800, loss[loss=0.06054, simple_loss=0.07082, pruned_loss=0.01374, audio_tagging_loss=0.01139, over 16855.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09361, pruned_loss=0.01487, audio_tagging_loss=0.009271, over 3039118.21 frames. ], batch size: 65, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:23:44,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1995793.3333333333, ans=0.125
2023-11-22 15:23:51,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1995793.3333333333, ans=0.125
2023-11-22 15:23:52,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1995793.3333333333, ans=0.125
2023-11-22 15:23:57,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1995860.0, ans=0.125
2023-11-22 15:24:01,834 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.38 vs. limit=6.0
2023-11-22 15:24:04,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1995860.0, ans=0.0
2023-11-22 15:24:14,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.59 vs. limit=22.5
2023-11-22 15:24:16,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1995926.6666666667, ans=0.125
2023-11-22 15:24:21,640 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299400
2023-11-22 15:24:23,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1995993.3333333333, ans=0.2
2023-11-22 15:24:36,974 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.241e+01 8.735e+01 9.368e+01 1.325e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 15:24:49,267 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10850, loss[loss=0.04401, simple_loss=0.05771, pruned_loss=0.006957, audio_tagging_loss=0.008198, over 14447.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09293, pruned_loss=0.01477, audio_tagging_loss=0.009239, over 3038352.75 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:25:07,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1996193.3333333333, ans=0.125
2023-11-22 15:25:15,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1996260.0, ans=0.1
2023-11-22 15:25:25,934 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299450
2023-11-22 15:25:45,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1996393.3333333333, ans=0.1
2023-11-22 15:25:48,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1996393.3333333333, ans=0.2
2023-11-22 15:25:49,788 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:25:53,360 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10900, loss[loss=0.09032, simple_loss=0.1239, pruned_loss=0.01914, audio_tagging_loss=0.009221, over 15701.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09312, pruned_loss=0.01474, audio_tagging_loss=0.009241, over 3036863.88 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:26:16,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1996526.6666666667, ans=0.1
2023-11-22 15:26:30,792 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299500
2023-11-22 15:26:33,696 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.90 vs. limit=15.0
2023-11-22 15:26:45,938 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.150e+01 8.424e+01 9.041e+01 9.724e+01 1.556e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-22 15:26:57,225 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 10950, loss[loss=0.05605, simple_loss=0.07186, pruned_loss=0.009761, audio_tagging_loss=0.01036, over 13791.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09354, pruned_loss=0.01474, audio_tagging_loss=0.009234, over 3038882.92 frames. ], batch size: 55, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:27:34,163 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299550
2023-11-22 15:27:45,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1996993.3333333333, ans=0.125
2023-11-22 15:27:48,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1997060.0, ans=0.125
2023-11-22 15:27:59,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1997126.6666666667, ans=0.2
2023-11-22 15:28:01,238 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11000, loss[loss=0.07118, simple_loss=0.08971, pruned_loss=0.01734, audio_tagging_loss=0.008981, over 15315.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09391, pruned_loss=0.01498, audio_tagging_loss=0.009268, over 3034046.84 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:28:03,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1997126.6666666667, ans=0.1
2023-11-22 15:28:05,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.44 vs. limit=15.0
2023-11-22 15:28:11,050 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:28:17,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1997193.3333333333, ans=0.125
2023-11-22 15:28:22,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1997193.3333333333, ans=0.0
2023-11-22 15:28:27,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1997260.0, ans=0.2
2023-11-22 15:28:31,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1997260.0, ans=0.1
2023-11-22 15:28:36,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1997260.0, ans=0.0
2023-11-22 15:28:37,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299600
2023-11-22 15:28:43,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1997326.6666666667, ans=0.125
2023-11-22 15:28:43,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1997326.6666666667, ans=0.125
2023-11-22 15:28:53,982 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.275e+01 8.821e+01 9.536e+01 1.341e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 15:29:03,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1997460.0, ans=0.2
2023-11-22 15:29:05,399 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11050, loss[loss=0.0595, simple_loss=0.06804, pruned_loss=0.01433, audio_tagging_loss=0.01116, over 15191.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09379, pruned_loss=0.01514, audio_tagging_loss=0.009383, over 3040180.19 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:29:08,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1997460.0, ans=0.125
2023-11-22 15:29:17,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1997526.6666666667, ans=0.125
2023-11-22 15:29:42,137 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299650
2023-11-22 15:29:45,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1997660.0, ans=0.0
2023-11-22 15:30:02,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1997726.6666666667, ans=0.125
2023-11-22 15:30:08,989 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11100, loss[loss=0.05407, simple_loss=0.06191, pruned_loss=0.01098, audio_tagging_loss=0.01213, over 13046.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09342, pruned_loss=0.01512, audio_tagging_loss=0.009553, over 3041624.87 frames. ], batch size: 53, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:30:20,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1997860.0, ans=0.025
2023-11-22 15:30:25,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1997860.0, ans=0.025
2023-11-22 15:30:25,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1997860.0, ans=0.1
2023-11-22 15:30:30,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1997860.0, ans=0.0
2023-11-22 15:30:45,986 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299700
2023-11-22 15:31:01,208 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.427e+01 8.934e+01 9.666e+01 1.581e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 15:31:12,914 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11150, loss[loss=0.07101, simple_loss=0.08918, pruned_loss=0.0139, audio_tagging_loss=0.01252, over 15329.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09268, pruned_loss=0.01501, audio_tagging_loss=0.009717, over 3040364.26 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:31:24,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1998193.3333333333, ans=0.1
2023-11-22 15:31:49,991 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299750
2023-11-22 15:32:00,800 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=15.0
2023-11-22 15:32:13,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1998393.3333333333, ans=0.125
2023-11-22 15:32:16,629 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11200, loss[loss=0.08546, simple_loss=0.1133, pruned_loss=0.01703, audio_tagging_loss=0.01176, over 13651.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.0931, pruned_loss=0.01498, audio_tagging_loss=0.009734, over 3037030.31 frames. ], batch size: 50, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:32:36,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1998526.6666666667, ans=0.125
2023-11-22 15:32:40,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1998526.6666666667, ans=0.0
2023-11-22 15:32:44,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1998593.3333333333, ans=0.0
2023-11-22 15:32:54,717 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299800
2023-11-22 15:33:09,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.174e+01 8.782e+01 9.719e+01 1.179e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 15:33:12,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1998726.6666666667, ans=0.125
2023-11-22 15:33:22,081 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11250, loss[loss=0.07545, simple_loss=0.09973, pruned_loss=0.01534, audio_tagging_loss=0.01024, over 14806.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09214, pruned_loss=0.01463, audio_tagging_loss=0.009729, over 3042199.73 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:33:23,990 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.76 vs. limit=15.0
2023-11-22 15:33:24,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1998793.3333333333, ans=0.1
2023-11-22 15:33:48,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1998926.6666666667, ans=0.125
2023-11-22 15:33:58,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299850
2023-11-22 15:34:02,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1998993.3333333333, ans=0.0
2023-11-22 15:34:17,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1999060.0, ans=0.125
2023-11-22 15:34:19,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1999060.0, ans=0.2
2023-11-22 15:34:19,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1999060.0, ans=0.04949747468305833
2023-11-22 15:34:26,190 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11300, loss[loss=0.06079, simple_loss=0.0753, pruned_loss=0.01217, audio_tagging_loss=0.01098, over 15638.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09239, pruned_loss=0.01465, audio_tagging_loss=0.009514, over 3047137.09 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:34:28,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1999126.6666666667, ans=0.125
2023-11-22 15:34:30,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1999126.6666666667, ans=0.125
2023-11-22 15:34:34,612 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=15.0
2023-11-22 15:34:35,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1999126.6666666667, ans=0.1
2023-11-22 15:34:43,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1999193.3333333333, ans=0.125
2023-11-22 15:34:47,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.09 vs. limit=12.0
2023-11-22 15:34:58,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1999260.0, ans=0.125
2023-11-22 15:35:03,153 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299900
2023-11-22 15:35:08,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1999326.6666666667, ans=0.125
2023-11-22 15:35:18,542 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.156e+01 8.491e+01 9.004e+01 9.715e+01 1.251e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 15:35:29,973 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11350, loss[loss=0.07414, simple_loss=0.09848, pruned_loss=0.01662, audio_tagging_loss=0.008279, over 14807.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09328, pruned_loss=0.01502, audio_tagging_loss=0.009345, over 3042173.74 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:36:02,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1999593.3333333333, ans=0.04949747468305833
2023-11-22 15:36:07,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 299950
2023-11-22 15:36:22,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1999726.6666666667, ans=0.125
2023-11-22 15:36:33,727 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11400, loss[loss=0.0839, simple_loss=0.1115, pruned_loss=0.02037, audio_tagging_loss=0.00776, over 14380.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09354, pruned_loss=0.01503, audio_tagging_loss=0.009337, over 3040742.00 frames. ], batch size: 52, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:36:42,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1999793.3333333333, ans=0.125
2023-11-22 15:36:55,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1999860.0, ans=0.0
2023-11-22 15:37:10,931 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300000
2023-11-22 15:37:18,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1999993.3333333333, ans=0.125
2023-11-22 15:37:24,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1999993.3333333333, ans=0.125
2023-11-22 15:37:31,200 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.071e+01 8.858e+01 9.602e+01 1.299e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 15:37:41,011 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11450, loss[loss=0.05887, simple_loss=0.07546, pruned_loss=0.01096, audio_tagging_loss=0.01018, over 14311.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09392, pruned_loss=0.01506, audio_tagging_loss=0.00929, over 3044193.68 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:38:18,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300050
2023-11-22 15:38:22,554 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.55 vs. limit=22.5
2023-11-22 15:38:28,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2000326.6666666667, ans=0.0
2023-11-22 15:38:33,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2000393.3333333333, ans=0.0
2023-11-22 15:38:44,820 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11500, loss[loss=0.07547, simple_loss=0.09904, pruned_loss=0.01771, audio_tagging_loss=0.008246, over 15331.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09301, pruned_loss=0.01485, audio_tagging_loss=0.009409, over 3040560.57 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:38:51,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2000460.0, ans=0.0
2023-11-22 15:38:57,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2000526.6666666667, ans=0.0
2023-11-22 15:38:58,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2000526.6666666667, ans=0.125
2023-11-22 15:39:08,647 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.81 vs. limit=15.0
2023-11-22 15:39:21,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300100
2023-11-22 15:39:30,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2000660.0, ans=0.0
2023-11-22 15:39:35,877 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:39:38,051 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.223e+01 8.870e+01 9.437e+01 1.249e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 15:39:47,790 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11550, loss[loss=0.0993, simple_loss=0.1352, pruned_loss=0.02446, audio_tagging_loss=0.007222, over 14611.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09308, pruned_loss=0.01492, audio_tagging_loss=0.009344, over 3046136.17 frames. ], batch size: 52, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:40:01,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2000860.0, ans=0.0
2023-11-22 15:40:25,314 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300150
2023-11-22 15:40:27,688 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:40:29,645 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.04 vs. limit=6.0
2023-11-22 15:40:38,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2001060.0, ans=0.125
2023-11-22 15:40:52,023 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11600, loss[loss=0.0705, simple_loss=0.09602, pruned_loss=0.01511, audio_tagging_loss=0.007387, over 15617.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09348, pruned_loss=0.01481, audio_tagging_loss=0.009337, over 3051151.93 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:41:19,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2001260.0, ans=0.125
2023-11-22 15:41:25,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2001260.0, ans=0.0
2023-11-22 15:41:29,216 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300200
2023-11-22 15:41:39,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.91 vs. limit=22.5
2023-11-22 15:41:45,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2001393.3333333333, ans=0.125
2023-11-22 15:41:46,180 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.408e+01 8.800e+01 9.381e+01 1.504e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 15:41:56,488 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11650, loss[loss=0.07133, simple_loss=0.09571, pruned_loss=0.01659, audio_tagging_loss=0.006889, over 14990.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09384, pruned_loss=0.01487, audio_tagging_loss=0.009249, over 3049290.45 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:41:56,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2001460.0, ans=0.1
2023-11-22 15:42:04,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0
2023-11-22 15:42:12,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2001526.6666666667, ans=0.125
2023-11-22 15:42:16,663 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.44 vs. limit=15.0
2023-11-22 15:42:33,488 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300250
2023-11-22 15:42:38,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2001660.0, ans=0.125
2023-11-22 15:42:40,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2001660.0, ans=0.2
2023-11-22 15:42:47,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2001726.6666666667, ans=0.125
2023-11-22 15:42:50,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2001726.6666666667, ans=0.125
2023-11-22 15:42:59,628 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11700, loss[loss=0.09479, simple_loss=0.1291, pruned_loss=0.02321, audio_tagging_loss=0.007006, over 15910.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09452, pruned_loss=0.01497, audio_tagging_loss=0.009162, over 3054901.18 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:43:04,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2001793.3333333333, ans=0.125
2023-11-22 15:43:05,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.34 vs. limit=15.0
2023-11-22 15:43:17,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2001860.0, ans=0.125
2023-11-22 15:43:27,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2001926.6666666667, ans=0.125
2023-11-22 15:43:31,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2001926.6666666667, ans=0.125
2023-11-22 15:43:37,173 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300300
2023-11-22 15:43:43,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2001993.3333333333, ans=0.125
2023-11-22 15:43:50,382 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.22 vs. limit=15.0
2023-11-22 15:43:53,162 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.806e+01 8.322e+01 8.823e+01 9.489e+01 1.251e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 15:43:56,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2002060.0, ans=0.125
2023-11-22 15:43:58,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2002060.0, ans=0.1
2023-11-22 15:44:03,413 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11750, loss[loss=0.05912, simple_loss=0.07574, pruned_loss=0.01026, audio_tagging_loss=0.01099, over 14115.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09373, pruned_loss=0.01498, audio_tagging_loss=0.009334, over 3046845.97 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:44:06,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2002126.6666666667, ans=0.125
2023-11-22 15:44:07,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2002126.6666666667, ans=0.2
2023-11-22 15:44:12,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2002126.6666666667, ans=0.125
2023-11-22 15:44:20,912 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.04 vs. limit=6.0
2023-11-22 15:44:22,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2002193.3333333333, ans=0.0
2023-11-22 15:44:34,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2002260.0, ans=0.2
2023-11-22 15:44:37,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2002260.0, ans=0.125
2023-11-22 15:44:39,729 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300350
2023-11-22 15:44:41,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2002326.6666666667, ans=0.1
2023-11-22 15:44:45,724 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.21 vs. limit=22.5
2023-11-22 15:44:54,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2002393.3333333333, ans=0.125
2023-11-22 15:45:01,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2002393.3333333333, ans=0.125
2023-11-22 15:45:07,027 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11800, loss[loss=0.07729, simple_loss=0.09562, pruned_loss=0.01761, audio_tagging_loss=0.01187, over 14159.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09344, pruned_loss=0.01499, audio_tagging_loss=0.009365, over 3043074.13 frames. ], batch size: 53, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:45:09,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2002460.0, ans=0.2
2023-11-22 15:45:14,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2002460.0, ans=0.125
2023-11-22 15:45:43,041 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300400
2023-11-22 15:46:01,596 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.204e+01 8.722e+01 9.366e+01 1.185e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 15:46:10,184 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11850, loss[loss=0.05191, simple_loss=0.06254, pruned_loss=0.006739, audio_tagging_loss=0.0139, over 14387.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09384, pruned_loss=0.01488, audio_tagging_loss=0.009455, over 3054419.87 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:46:24,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.25 vs. limit=15.0
2023-11-22 15:46:45,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2002926.6666666667, ans=0.125
2023-11-22 15:46:45,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2002926.6666666667, ans=0.125
2023-11-22 15:46:47,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300450
2023-11-22 15:47:06,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0
2023-11-22 15:47:11,861 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.22 vs. limit=15.0
2023-11-22 15:47:13,753 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11900, loss[loss=0.06176, simple_loss=0.08428, pruned_loss=0.01065, audio_tagging_loss=0.008968, over 15986.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09413, pruned_loss=0.01489, audio_tagging_loss=0.009478, over 3052503.03 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:47:16,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2003126.6666666667, ans=0.125
2023-11-22 15:47:30,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2003193.3333333333, ans=0.0
2023-11-22 15:47:50,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300500
2023-11-22 15:47:50,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2003326.6666666667, ans=0.125
2023-11-22 15:47:51,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2003326.6666666667, ans=0.0
2023-11-22 15:47:53,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2003326.6666666667, ans=0.1
2023-11-22 15:48:03,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2003393.3333333333, ans=0.0
2023-11-22 15:48:05,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2003393.3333333333, ans=0.125
2023-11-22 15:48:07,932 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.243e+01 8.822e+01 9.597e+01 1.131e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 15:48:17,681 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 11950, loss[loss=0.07219, simple_loss=0.1003, pruned_loss=0.0157, audio_tagging_loss=0.006335, over 15068.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09436, pruned_loss=0.01477, audio_tagging_loss=0.00951, over 3050544.59 frames. ], batch size: 55, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:48:28,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2003460.0, ans=0.05
2023-11-22 15:48:33,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.34 vs. limit=15.0
2023-11-22 15:48:48,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2003593.3333333333, ans=0.0
2023-11-22 15:48:54,163 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300550
2023-11-22 15:48:55,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2003660.0, ans=0.125
2023-11-22 15:48:57,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2003660.0, ans=0.125
2023-11-22 15:49:15,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2003726.6666666667, ans=0.0
2023-11-22 15:49:18,852 INFO [train_asr.py:1221] (2/4) Epoch 25, batch 12000, loss[loss=0.07644, simple_loss=0.105, pruned_loss=0.01714, audio_tagging_loss=0.006806, over 15339.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09496, pruned_loss=0.01494, audio_tagging_loss=0.00946, over 3047720.73 frames. ], batch size: 54, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:49:18,853 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 15:49:50,747 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.4780, 3.8271, 4.3717, 3.3608], device='cuda:2')
2023-11-22 15:49:59,248 INFO [train_asr.py:1253] (2/4) Epoch 25, validation: loss=0.05961, simple_loss=0.05152, pruned_loss=0.005134, audio_tagging_loss=0.02872, over 4681554.00 frames. 
2023-11-22 15:49:59,249 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 15:49:59,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.94 vs. limit=12.0
2023-11-22 15:50:15,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2003860.0, ans=0.125
2023-11-22 15:50:21,437 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.48 vs. limit=12.0
2023-11-22 15:51:02,499 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 0, loss[loss=0.0754, simple_loss=0.077, pruned_loss=0.01205, audio_tagging_loss=0.02485, over 15396.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.077, pruned_loss=0.01205, audio_tagging_loss=0.02485, over 15396.00 frames. ], batch size: 57, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:51:02,499 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 15:51:37,687 INFO [train_asr.py:1253] (2/4) Epoch 26, validation: loss=0.05869, simple_loss=0.05153, pruned_loss=0.005094, audio_tagging_loss=0.02783, over 4681554.00 frames. 
2023-11-22 15:51:37,688 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 15:51:38,381 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0
2023-11-22 15:51:39,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2003960.0, ans=0.0
2023-11-22 15:51:43,330 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300600
2023-11-22 15:51:43,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2003960.0, ans=0.125
2023-11-22 15:51:47,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2003960.0, ans=0.1
2023-11-22 15:52:01,849 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.448e+01 9.337e+01 1.025e+02 1.392e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-22 15:52:37,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2004226.6666666667, ans=0.5
2023-11-22 15:52:43,332 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 50, loss[loss=0.08764, simple_loss=0.1058, pruned_loss=0.0166, audio_tagging_loss=0.01812, over 15066.00 frames. ], tot_loss[loss=0.08091, simple_loss=0.09567, pruned_loss=0.01531, audio_tagging_loss=0.01776, over 687113.57 frames. ], batch size: 55, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:52:43,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2004293.3333333333, ans=0.0
2023-11-22 15:52:48,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300650
2023-11-22 15:53:06,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2004360.0, ans=0.125
2023-11-22 15:53:27,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2004493.3333333333, ans=0.125
2023-11-22 15:53:48,101 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 100, loss[loss=0.07345, simple_loss=0.08573, pruned_loss=0.01319, audio_tagging_loss=0.01739, over 15323.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09188, pruned_loss=0.01472, audio_tagging_loss=0.01746, over 1212797.63 frames. ], batch size: 58, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:53:53,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300700
2023-11-22 15:54:11,741 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.950e+01 8.718e+01 9.300e+01 1.020e+02 1.184e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-22 15:54:44,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2004893.3333333333, ans=0.125
2023-11-22 15:54:53,099 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 150, loss[loss=0.08616, simple_loss=0.1129, pruned_loss=0.01898, audio_tagging_loss=0.01075, over 16273.00 frames. ], tot_loss[loss=0.0767, simple_loss=0.09273, pruned_loss=0.01462, audio_tagging_loss=0.01571, over 1623118.85 frames. ], batch size: 59, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:54:58,123 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300750
2023-11-22 15:55:03,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2004960.0, ans=0.125
2023-11-22 15:55:11,188 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:55:11,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2005026.6666666667, ans=0.1
2023-11-22 15:55:16,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2005026.6666666667, ans=0.1
2023-11-22 15:55:20,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2005093.3333333333, ans=0.1
2023-11-22 15:55:26,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2005093.3333333333, ans=0.2
2023-11-22 15:55:57,039 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 200, loss[loss=0.07128, simple_loss=0.09844, pruned_loss=0.01174, audio_tagging_loss=0.01032, over 15324.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09363, pruned_loss=0.01461, audio_tagging_loss=0.01391, over 1944641.99 frames. ], batch size: 58, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:56:02,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300800
2023-11-22 15:56:16,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2005360.0, ans=0.2
2023-11-22 15:56:20,091 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.352e+01 8.945e+01 1.004e+02 1.668e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-22 15:56:39,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2005493.3333333333, ans=0.1
2023-11-22 15:56:40,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2005493.3333333333, ans=0.2
2023-11-22 15:56:42,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2005493.3333333333, ans=0.0
2023-11-22 15:56:43,468 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=12.0
2023-11-22 15:57:01,855 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 250, loss[loss=0.05138, simple_loss=0.06847, pruned_loss=0.0103, audio_tagging_loss=0.006841, over 14647.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09342, pruned_loss=0.01478, audio_tagging_loss=0.01246, over 2195872.60 frames. ], batch size: 55, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:57:06,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300850
2023-11-22 15:57:20,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2005693.3333333333, ans=10.0
2023-11-22 15:57:42,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2005826.6666666667, ans=0.0
2023-11-22 15:57:44,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2005826.6666666667, ans=0.125
2023-11-22 15:57:58,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=15.0
2023-11-22 15:58:07,217 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 300, loss[loss=0.07182, simple_loss=0.1007, pruned_loss=0.01522, audio_tagging_loss=0.006272, over 15455.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09477, pruned_loss=0.015, audio_tagging_loss=0.01148, over 2383416.53 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 15:58:12,253 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300900
2023-11-22 15:58:30,066 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.299e+01 8.395e+01 9.138e+01 9.781e+01 1.359e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-22 15:58:36,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2006093.3333333333, ans=0.125
2023-11-22 15:58:47,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2006160.0, ans=0.125
2023-11-22 15:58:49,998 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:58:57,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2006160.0, ans=0.1
2023-11-22 15:58:59,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0
2023-11-22 15:59:00,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.36 vs. limit=15.0
2023-11-22 15:59:06,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2006226.6666666667, ans=0.125
2023-11-22 15:59:09,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-22 15:59:12,892 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 350, loss[loss=0.07, simple_loss=0.09352, pruned_loss=0.01468, audio_tagging_loss=0.008558, over 15764.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09458, pruned_loss=0.01505, audio_tagging_loss=0.01094, over 2535394.46 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 15:59:13,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2006293.3333333333, ans=0.125
2023-11-22 15:59:18,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 300950
2023-11-22 15:59:19,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2006293.3333333333, ans=0.5
2023-11-22 15:59:23,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2006293.3333333333, ans=0.125
2023-11-22 15:59:55,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2006493.3333333333, ans=0.2
2023-11-22 15:59:59,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2006493.3333333333, ans=0.125
2023-11-22 16:00:15,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2006560.0, ans=0.125
2023-11-22 16:00:17,754 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 400, loss[loss=0.06818, simple_loss=0.09089, pruned_loss=0.01257, audio_tagging_loss=0.01017, over 14850.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09411, pruned_loss=0.01493, audio_tagging_loss=0.01049, over 2645101.01 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:00:23,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301000
2023-11-22 16:00:33,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2006693.3333333333, ans=0.125
2023-11-22 16:00:36,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2006693.3333333333, ans=0.0
2023-11-22 16:00:42,278 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.126e+01 8.693e+01 9.257e+01 1.166e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 16:00:43,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2006760.0, ans=0.1
2023-11-22 16:00:50,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2006760.0, ans=0.2
2023-11-22 16:00:57,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2006826.6666666667, ans=0.025
2023-11-22 16:00:59,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2006826.6666666667, ans=0.125
2023-11-22 16:01:23,355 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 450, loss[loss=0.07789, simple_loss=0.09621, pruned_loss=0.02072, audio_tagging_loss=0.009057, over 13910.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09414, pruned_loss=0.01493, audio_tagging_loss=0.01025, over 2727840.25 frames. ], batch size: 53, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:01:29,077 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301050
2023-11-22 16:01:51,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2007093.3333333333, ans=0.125
2023-11-22 16:01:55,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2007093.3333333333, ans=0.2
2023-11-22 16:02:20,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2007226.6666666667, ans=0.0
2023-11-22 16:02:22,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.88 vs. limit=10.0
2023-11-22 16:02:27,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2007293.3333333333, ans=0.0
2023-11-22 16:02:28,746 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 500, loss[loss=0.09316, simple_loss=0.1282, pruned_loss=0.02103, audio_tagging_loss=0.008019, over 15919.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09513, pruned_loss=0.01519, audio_tagging_loss=0.009877, over 2799960.43 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:02:33,734 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301100
2023-11-22 16:02:38,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.79 vs. limit=15.0
2023-11-22 16:02:39,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2007360.0, ans=0.2
2023-11-22 16:02:50,564 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.306e+01 8.908e+01 9.851e+01 1.431e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 16:03:32,321 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 550, loss[loss=0.06749, simple_loss=0.08989, pruned_loss=0.01426, audio_tagging_loss=0.008295, over 16703.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09445, pruned_loss=0.015, audio_tagging_loss=0.009718, over 2854803.10 frames. ], batch size: 62, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:03:37,302 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301150
2023-11-22 16:03:41,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2007626.6666666667, ans=0.125
2023-11-22 16:04:10,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2007826.6666666667, ans=0.125
2023-11-22 16:04:36,966 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 600, loss[loss=0.08848, simple_loss=0.1183, pruned_loss=0.01787, audio_tagging_loss=0.01147, over 15144.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09454, pruned_loss=0.01511, audio_tagging_loss=0.009636, over 2894886.92 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:04:42,587 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301200
2023-11-22 16:04:42,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2007960.0, ans=0.025
2023-11-22 16:05:01,749 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.281e+01 8.745e+01 9.297e+01 1.116e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 16:05:04,813 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.99 vs. limit=10.0
2023-11-22 16:05:07,425 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.36 vs. limit=22.5
2023-11-22 16:05:19,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2008160.0, ans=0.125
2023-11-22 16:05:22,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2008160.0, ans=0.125
2023-11-22 16:05:35,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2008226.6666666667, ans=0.125
2023-11-22 16:05:42,002 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 650, loss[loss=0.05797, simple_loss=0.07012, pruned_loss=0.01215, audio_tagging_loss=0.01076, over 15994.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09437, pruned_loss=0.01513, audio_tagging_loss=0.009579, over 2933417.40 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:05:46,976 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301250
2023-11-22 16:06:00,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2008360.0, ans=0.125
2023-11-22 16:06:03,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.88 vs. limit=22.5
2023-11-22 16:06:06,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=12.0
2023-11-22 16:06:10,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2008426.6666666667, ans=0.04949747468305833
2023-11-22 16:06:16,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2008426.6666666667, ans=0.1
2023-11-22 16:06:21,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2008493.3333333333, ans=0.125
2023-11-22 16:06:22,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2008493.3333333333, ans=0.09899494936611666
2023-11-22 16:06:45,895 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 700, loss[loss=0.0612, simple_loss=0.07583, pruned_loss=0.0108, audio_tagging_loss=0.01248, over 15922.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.0926, pruned_loss=0.01463, audio_tagging_loss=0.009568, over 2954288.10 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:06:50,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301300
2023-11-22 16:07:11,109 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 8.183e+01 8.760e+01 9.542e+01 1.269e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 16:07:12,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2008760.0, ans=0.2
2023-11-22 16:07:35,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2008826.6666666667, ans=0.0
2023-11-22 16:07:36,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2008893.3333333333, ans=0.125
2023-11-22 16:07:49,628 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 750, loss[loss=0.09369, simple_loss=0.1171, pruned_loss=0.02553, audio_tagging_loss=0.009612, over 14944.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09234, pruned_loss=0.01467, audio_tagging_loss=0.009561, over 2977734.11 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:07:50,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2008960.0, ans=0.0
2023-11-22 16:07:54,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2008960.0, ans=0.025
2023-11-22 16:07:55,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301350
2023-11-22 16:08:04,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2009026.6666666667, ans=0.125
2023-11-22 16:08:04,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2009026.6666666667, ans=0.0
2023-11-22 16:08:07,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2009026.6666666667, ans=0.2
2023-11-22 16:08:42,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.80 vs. limit=6.0
2023-11-22 16:08:45,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.03 vs. limit=15.0
2023-11-22 16:08:46,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2009226.6666666667, ans=0.125
2023-11-22 16:08:48,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2009226.6666666667, ans=0.05
2023-11-22 16:08:55,392 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 800, loss[loss=0.06085, simple_loss=0.07636, pruned_loss=0.01159, audio_tagging_loss=0.01109, over 14798.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09331, pruned_loss=0.01491, audio_tagging_loss=0.0095, over 2994777.63 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:09:00,857 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301400
2023-11-22 16:09:19,557 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.335e+01 8.906e+01 9.640e+01 1.225e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 16:10:00,005 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 850, loss[loss=0.07127, simple_loss=0.09012, pruned_loss=0.01789, audio_tagging_loss=0.008325, over 14832.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09408, pruned_loss=0.01506, audio_tagging_loss=0.009481, over 3009408.29 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:10:04,932 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301450
2023-11-22 16:10:20,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2009693.3333333333, ans=0.0
2023-11-22 16:10:42,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2009826.6666666667, ans=0.2
2023-11-22 16:11:03,338 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 900, loss[loss=0.0748, simple_loss=0.09866, pruned_loss=0.01547, audio_tagging_loss=0.009996, over 14884.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09383, pruned_loss=0.01494, audio_tagging_loss=0.009634, over 3016109.61 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:11:08,964 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301500
2023-11-22 16:11:25,622 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.10 vs. limit=15.0
2023-11-22 16:11:28,581 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.648e+01 9.209e+01 1.007e+02 1.462e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-22 16:11:34,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2010093.3333333333, ans=0.0
2023-11-22 16:11:45,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2010160.0, ans=0.125
2023-11-22 16:12:07,505 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 950, loss[loss=0.07086, simple_loss=0.09152, pruned_loss=0.01556, audio_tagging_loss=0.009534, over 16281.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09372, pruned_loss=0.01495, audio_tagging_loss=0.009478, over 3024226.46 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:12:13,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301550
2023-11-22 16:12:40,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2010426.6666666667, ans=0.0
2023-11-22 16:12:44,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2010493.3333333333, ans=0.1
2023-11-22 16:13:05,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2010560.0, ans=0.05
2023-11-22 16:13:11,455 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1000, loss[loss=0.07058, simple_loss=0.09285, pruned_loss=0.01599, audio_tagging_loss=0.008164, over 14565.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09382, pruned_loss=0.01494, audio_tagging_loss=0.009308, over 3029364.65 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:13:11,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2010626.6666666667, ans=0.125
2023-11-22 16:13:15,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2010626.6666666667, ans=0.1
2023-11-22 16:13:16,277 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301600
2023-11-22 16:13:18,110 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0
2023-11-22 16:13:25,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2010693.3333333333, ans=0.125
2023-11-22 16:13:35,433 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.207e+01 8.822e+01 9.755e+01 1.375e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 16:13:39,671 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:13:51,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2010826.6666666667, ans=0.125
2023-11-22 16:13:51,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2010826.6666666667, ans=0.0
2023-11-22 16:14:06,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2010893.3333333333, ans=0.0
2023-11-22 16:14:08,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.90 vs. limit=15.0
2023-11-22 16:14:09,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.52 vs. limit=15.0
2023-11-22 16:14:11,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2010893.3333333333, ans=0.125
2023-11-22 16:14:15,387 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1050, loss[loss=0.08738, simple_loss=0.1064, pruned_loss=0.02476, audio_tagging_loss=0.009421, over 15802.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09416, pruned_loss=0.01501, audio_tagging_loss=0.009154, over 3041301.40 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:14:19,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.25 vs. limit=22.5
2023-11-22 16:14:20,334 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301650
2023-11-22 16:14:27,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2011026.6666666667, ans=0.2
2023-11-22 16:14:34,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.95 vs. limit=15.0
2023-11-22 16:15:02,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0
2023-11-22 16:15:08,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2011226.6666666667, ans=0.0
2023-11-22 16:15:20,083 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1100, loss[loss=0.07275, simple_loss=0.0982, pruned_loss=0.01603, audio_tagging_loss=0.007619, over 14906.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09356, pruned_loss=0.01484, audio_tagging_loss=0.009169, over 3046031.01 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:15:23,756 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:15:25,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301700
2023-11-22 16:15:38,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2011360.0, ans=0.125
2023-11-22 16:15:42,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2011360.0, ans=0.0
2023-11-22 16:15:44,417 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.766e+01 8.159e+01 8.793e+01 9.276e+01 1.252e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 16:15:46,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2011426.6666666667, ans=0.0
2023-11-22 16:15:46,269 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-22 16:15:50,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2011426.6666666667, ans=0.0
2023-11-22 16:16:16,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.08 vs. limit=10.0
2023-11-22 16:16:24,777 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1150, loss[loss=0.07476, simple_loss=0.1044, pruned_loss=0.0163, audio_tagging_loss=0.006243, over 14817.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09323, pruned_loss=0.01464, audio_tagging_loss=0.009119, over 3039363.39 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:16:25,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2011626.6666666667, ans=0.1
2023-11-22 16:16:27,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2011626.6666666667, ans=0.0
2023-11-22 16:16:29,748 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301750
2023-11-22 16:16:38,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2011693.3333333333, ans=0.1
2023-11-22 16:16:40,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2011693.3333333333, ans=0.2
2023-11-22 16:16:46,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.10 vs. limit=15.0
2023-11-22 16:16:54,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2011760.0, ans=0.04949747468305833
2023-11-22 16:17:19,967 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.23 vs. limit=15.0
2023-11-22 16:17:28,014 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1200, loss[loss=0.07038, simple_loss=0.08195, pruned_loss=0.01778, audio_tagging_loss=0.01162, over 16403.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09358, pruned_loss=0.01477, audio_tagging_loss=0.009094, over 3047653.61 frames. ], batch size: 62, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:17:29,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=15.0
2023-11-22 16:17:33,021 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301800
2023-11-22 16:17:33,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2011960.0, ans=0.125
2023-11-22 16:17:54,133 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 8.163e+01 8.768e+01 9.430e+01 1.301e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 16:18:09,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2012160.0, ans=0.035
2023-11-22 16:18:26,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2012226.6666666667, ans=0.125
2023-11-22 16:18:32,628 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1250, loss[loss=0.06923, simple_loss=0.09526, pruned_loss=0.01368, audio_tagging_loss=0.007915, over 15347.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09423, pruned_loss=0.01476, audio_tagging_loss=0.009063, over 3046021.34 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:18:35,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0
2023-11-22 16:18:37,642 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301850
2023-11-22 16:18:41,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2012293.3333333333, ans=0.125
2023-11-22 16:18:43,780 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.45 vs. limit=6.0
2023-11-22 16:19:32,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.66 vs. limit=15.0
2023-11-22 16:19:37,029 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1300, loss[loss=0.06683, simple_loss=0.09617, pruned_loss=0.01219, audio_tagging_loss=0.006554, over 16018.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09372, pruned_loss=0.01455, audio_tagging_loss=0.009122, over 3041511.69 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:19:37,800 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.58 vs. limit=15.0
2023-11-22 16:19:42,725 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301900
2023-11-22 16:19:51,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2012693.3333333333, ans=0.0
2023-11-22 16:19:51,992 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.21 vs. limit=15.0
2023-11-22 16:20:02,337 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.148e+01 8.961e+01 9.729e+01 1.373e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 16:20:08,966 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.04 vs. limit=15.0
2023-11-22 16:20:13,026 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.90 vs. limit=15.0
2023-11-22 16:20:18,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2012826.6666666667, ans=0.0
2023-11-22 16:20:31,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.73 vs. limit=22.5
2023-11-22 16:20:34,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2012893.3333333333, ans=0.125
2023-11-22 16:20:38,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2012893.3333333333, ans=0.125
2023-11-22 16:20:41,743 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1350, loss[loss=0.06458, simple_loss=0.08421, pruned_loss=0.0116, audio_tagging_loss=0.01088, over 15913.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09416, pruned_loss=0.01465, audio_tagging_loss=0.009061, over 3035167.80 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:20:46,771 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 301950
2023-11-22 16:20:48,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2012960.0, ans=0.2
2023-11-22 16:20:53,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2013026.6666666667, ans=0.1
2023-11-22 16:21:00,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2013026.6666666667, ans=0.0
2023-11-22 16:21:29,500 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:21:46,589 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1400, loss[loss=0.07123, simple_loss=0.08735, pruned_loss=0.01903, audio_tagging_loss=0.008525, over 14673.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09416, pruned_loss=0.01464, audio_tagging_loss=0.009102, over 3039843.17 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:21:46,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2013293.3333333333, ans=0.2
2023-11-22 16:21:51,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2013293.3333333333, ans=15.0
2023-11-22 16:21:51,678 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302000
2023-11-22 16:21:55,977 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.99 vs. limit=15.0
2023-11-22 16:21:59,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2013360.0, ans=0.1
2023-11-22 16:22:13,009 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.088e+01 8.833e+01 9.465e+01 1.089e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 16:22:24,099 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.52 vs. limit=10.0
2023-11-22 16:22:25,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.70 vs. limit=15.0
2023-11-22 16:22:43,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.34 vs. limit=12.0
2023-11-22 16:22:50,766 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1450, loss[loss=0.08628, simple_loss=0.1089, pruned_loss=0.02011, audio_tagging_loss=0.0117, over 16049.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09515, pruned_loss=0.01489, audio_tagging_loss=0.009096, over 3042962.02 frames. ], batch size: 60, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:22:53,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2013626.6666666667, ans=0.125
2023-11-22 16:22:55,909 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302050
2023-11-22 16:23:05,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2013693.3333333333, ans=0.125
2023-11-22 16:23:16,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2013760.0, ans=0.125
2023-11-22 16:23:24,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2013760.0, ans=0.0
2023-11-22 16:23:37,197 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.77 vs. limit=15.0
2023-11-22 16:23:53,703 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1500, loss[loss=0.06686, simple_loss=0.07999, pruned_loss=0.01629, audio_tagging_loss=0.01058, over 15552.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09436, pruned_loss=0.01486, audio_tagging_loss=0.009196, over 3043469.77 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:23:59,426 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302100
2023-11-22 16:24:01,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2013960.0, ans=0.125
2023-11-22 16:24:05,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2014026.6666666667, ans=0.125
2023-11-22 16:24:13,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2014026.6666666667, ans=0.0
2023-11-22 16:24:15,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.89 vs. limit=15.0
2023-11-22 16:24:20,918 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.201e+01 8.781e+01 9.408e+01 1.231e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 16:24:26,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2014093.3333333333, ans=0.2
2023-11-22 16:24:48,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2014226.6666666667, ans=0.125
2023-11-22 16:24:58,108 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1550, loss[loss=0.09134, simple_loss=0.1277, pruned_loss=0.01994, audio_tagging_loss=0.007528, over 15494.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09433, pruned_loss=0.01488, audio_tagging_loss=0.0094, over 3042115.73 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:25:04,259 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302150
2023-11-22 16:25:22,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2014360.0, ans=0.0
2023-11-22 16:25:29,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2014426.6666666667, ans=0.2
2023-11-22 16:25:37,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2014493.3333333333, ans=0.125
2023-11-22 16:25:39,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2014493.3333333333, ans=0.125
2023-11-22 16:25:43,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2014493.3333333333, ans=0.125
2023-11-22 16:26:03,117 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1600, loss[loss=0.06359, simple_loss=0.0817, pruned_loss=0.01125, audio_tagging_loss=0.01149, over 15282.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09413, pruned_loss=0.015, audio_tagging_loss=0.009526, over 3055504.75 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:26:07,939 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302200
2023-11-22 16:26:24,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2014693.3333333333, ans=0.1
2023-11-22 16:26:29,463 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.323e+01 8.320e+01 8.949e+01 9.605e+01 1.369e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 16:26:46,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2014826.6666666667, ans=0.1
2023-11-22 16:27:06,579 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1650, loss[loss=0.06611, simple_loss=0.08661, pruned_loss=0.01454, audio_tagging_loss=0.008266, over 13648.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09523, pruned_loss=0.01506, audio_tagging_loss=0.009442, over 3054812.11 frames. ], batch size: 52, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:27:11,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302250
2023-11-22 16:27:11,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2014960.0, ans=0.0
2023-11-22 16:27:56,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2015226.6666666667, ans=0.5
2023-11-22 16:27:56,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2015226.6666666667, ans=0.0
2023-11-22 16:28:03,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.25 vs. limit=15.0
2023-11-22 16:28:10,473 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1700, loss[loss=0.07145, simple_loss=0.1102, pruned_loss=0.009584, audio_tagging_loss=0.006768, over 15646.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09497, pruned_loss=0.01485, audio_tagging_loss=0.009437, over 3052609.84 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:28:11,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.50 vs. limit=6.0
2023-11-22 16:28:15,398 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302300
2023-11-22 16:28:31,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2015360.0, ans=0.0
2023-11-22 16:28:34,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2015426.6666666667, ans=0.125
2023-11-22 16:28:36,855 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.605e+01 8.196e+01 8.694e+01 9.442e+01 1.208e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 16:28:40,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.87 vs. limit=15.0
2023-11-22 16:28:42,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.48 vs. limit=15.0
2023-11-22 16:28:48,309 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.35 vs. limit=22.5
2023-11-22 16:28:59,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2015560.0, ans=0.125
2023-11-22 16:29:13,280 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1750, loss[loss=0.07612, simple_loss=0.1066, pruned_loss=0.01498, audio_tagging_loss=0.007818, over 15322.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09487, pruned_loss=0.01478, audio_tagging_loss=0.00934, over 3050185.39 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:29:18,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302350
2023-11-22 16:29:25,523 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.02 vs. limit=15.0
2023-11-22 16:29:28,208 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.67 vs. limit=10.0
2023-11-22 16:29:29,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.17 vs. limit=10.0
2023-11-22 16:29:29,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2015693.3333333333, ans=0.1
2023-11-22 16:29:31,500 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.19 vs. limit=22.5
2023-11-22 16:30:16,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2015960.0, ans=0.125
2023-11-22 16:30:17,048 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1800, loss[loss=0.05929, simple_loss=0.0802, pruned_loss=0.01331, audio_tagging_loss=0.005873, over 14982.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09446, pruned_loss=0.01467, audio_tagging_loss=0.009271, over 3051135.19 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:30:22,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302400
2023-11-22 16:30:44,308 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.100e+01 8.696e+01 9.181e+01 1.134e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 16:31:16,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2016226.6666666667, ans=0.125
2023-11-22 16:31:20,407 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1850, loss[loss=0.05889, simple_loss=0.0797, pruned_loss=0.01149, audio_tagging_loss=0.007557, over 15207.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09465, pruned_loss=0.01469, audio_tagging_loss=0.009221, over 3054896.84 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:31:25,336 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302450
2023-11-22 16:31:35,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2016360.0, ans=0.125
2023-11-22 16:31:38,473 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.16 vs. limit=22.5
2023-11-22 16:31:39,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2016360.0, ans=0.1
2023-11-22 16:32:03,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2016493.3333333333, ans=0.0
2023-11-22 16:32:04,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2016493.3333333333, ans=0.1
2023-11-22 16:32:26,256 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1900, loss[loss=0.07078, simple_loss=0.0904, pruned_loss=0.0142, audio_tagging_loss=0.01138, over 14938.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09294, pruned_loss=0.0144, audio_tagging_loss=0.009296, over 3064782.52 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:32:31,986 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302500
2023-11-22 16:32:33,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2016626.6666666667, ans=0.125
2023-11-22 16:32:52,804 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.955e+01 8.093e+01 8.828e+01 9.639e+01 1.190e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 16:32:56,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2016760.0, ans=0.125
2023-11-22 16:33:09,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.69 vs. limit=15.0
2023-11-22 16:33:13,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2016826.6666666667, ans=0.0
2023-11-22 16:33:15,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2016826.6666666667, ans=0.125
2023-11-22 16:33:22,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.63 vs. limit=15.0
2023-11-22 16:33:29,825 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 1950, loss[loss=0.06965, simple_loss=0.09402, pruned_loss=0.01103, audio_tagging_loss=0.01161, over 14388.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09294, pruned_loss=0.01434, audio_tagging_loss=0.009261, over 3065343.52 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:33:34,846 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302550
2023-11-22 16:33:47,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2017026.6666666667, ans=0.0
2023-11-22 16:33:53,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2017093.3333333333, ans=0.125
2023-11-22 16:34:13,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2017160.0, ans=0.125
2023-11-22 16:34:14,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2017160.0, ans=0.125
2023-11-22 16:34:20,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2017226.6666666667, ans=0.1
2023-11-22 16:34:23,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2017226.6666666667, ans=0.1
2023-11-22 16:34:32,528 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2000, loss[loss=0.0672, simple_loss=0.08305, pruned_loss=0.01429, audio_tagging_loss=0.01138, over 15516.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09315, pruned_loss=0.01451, audio_tagging_loss=0.009205, over 3058592.22 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:34:37,530 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302600
2023-11-22 16:34:43,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2017293.3333333333, ans=0.125
2023-11-22 16:34:50,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2017360.0, ans=0.1
2023-11-22 16:34:53,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2017360.0, ans=0.0
2023-11-22 16:35:00,829 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.236e+01 8.987e+01 9.619e+01 1.204e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 16:35:13,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2017493.3333333333, ans=0.125
2023-11-22 16:35:21,882 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.83 vs. limit=15.0
2023-11-22 16:35:36,078 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:35:37,071 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2050, loss[loss=0.07573, simple_loss=0.1036, pruned_loss=0.01648, audio_tagging_loss=0.007429, over 14605.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09335, pruned_loss=0.01465, audio_tagging_loss=0.009165, over 3054257.46 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:35:37,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2017626.6666666667, ans=0.125
2023-11-22 16:35:42,617 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302650
2023-11-22 16:35:45,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2017626.6666666667, ans=0.2
2023-11-22 16:35:59,470 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.71 vs. limit=22.5
2023-11-22 16:36:09,638 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.10 vs. limit=6.0
2023-11-22 16:36:40,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2017960.0, ans=0.0
2023-11-22 16:36:41,102 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2100, loss[loss=0.08467, simple_loss=0.1188, pruned_loss=0.01663, audio_tagging_loss=0.008656, over 15586.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09387, pruned_loss=0.01489, audio_tagging_loss=0.009192, over 3051997.75 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:36:46,159 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302700
2023-11-22 16:37:06,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.20 vs. limit=15.0
2023-11-22 16:37:09,222 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.575e+01 8.367e+01 8.996e+01 9.804e+01 1.259e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 16:37:22,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2018160.0, ans=0.125
2023-11-22 16:37:28,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-22 16:37:39,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2018226.6666666667, ans=0.1
2023-11-22 16:37:42,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2018226.6666666667, ans=0.0
2023-11-22 16:37:44,289 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2150, loss[loss=0.07981, simple_loss=0.1061, pruned_loss=0.01619, audio_tagging_loss=0.01058, over 14900.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09456, pruned_loss=0.01507, audio_tagging_loss=0.00917, over 3046528.86 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:37:46,096 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.40 vs. limit=12.0
2023-11-22 16:37:48,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2018293.3333333333, ans=0.1
2023-11-22 16:37:49,334 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302750
2023-11-22 16:38:07,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2018360.0, ans=0.125
2023-11-22 16:38:23,072 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:38:41,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2018560.0, ans=0.1
2023-11-22 16:38:47,927 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2200, loss[loss=0.06545, simple_loss=0.08043, pruned_loss=0.01468, audio_tagging_loss=0.01056, over 14421.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09408, pruned_loss=0.01494, audio_tagging_loss=0.009282, over 3044666.11 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:38:52,923 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302800
2023-11-22 16:38:57,774 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.86 vs. limit=12.0
2023-11-22 16:39:15,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2018760.0, ans=0.125
2023-11-22 16:39:16,788 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 8.391e+01 8.919e+01 9.600e+01 1.144e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 16:39:39,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2018893.3333333333, ans=0.2
2023-11-22 16:39:50,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=22.5
2023-11-22 16:39:51,943 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2250, loss[loss=0.06571, simple_loss=0.08693, pruned_loss=0.01554, audio_tagging_loss=0.006697, over 15569.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09339, pruned_loss=0.01481, audio_tagging_loss=0.009344, over 3038455.61 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:39:56,878 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302850
2023-11-22 16:40:12,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2019026.6666666667, ans=0.0
2023-11-22 16:40:21,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2019093.3333333333, ans=0.1
2023-11-22 16:40:45,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2019226.6666666667, ans=0.0
2023-11-22 16:40:51,865 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.06 vs. limit=15.0
2023-11-22 16:40:54,897 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2300, loss[loss=0.1046, simple_loss=0.1496, pruned_loss=0.02288, audio_tagging_loss=0.006943, over 15347.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09293, pruned_loss=0.01465, audio_tagging_loss=0.009327, over 3040404.78 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:40:55,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2019293.3333333333, ans=0.125
2023-11-22 16:40:59,867 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302900
2023-11-22 16:41:00,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0
2023-11-22 16:41:24,526 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.135e+01 8.659e+01 9.382e+01 1.572e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-22 16:41:50,895 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:41:58,430 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2350, loss[loss=0.07893, simple_loss=0.1009, pruned_loss=0.01702, audio_tagging_loss=0.01144, over 14463.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09324, pruned_loss=0.01471, audio_tagging_loss=0.00936, over 3044415.26 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:42:03,932 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 302950
2023-11-22 16:42:12,557 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.10 vs. limit=15.0
2023-11-22 16:42:14,835 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.94 vs. limit=22.5
2023-11-22 16:42:17,829 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.05 vs. limit=22.5
2023-11-22 16:42:30,112 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-22 16:42:46,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.06 vs. limit=15.0
2023-11-22 16:43:01,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2019960.0, ans=0.125
2023-11-22 16:43:02,276 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2400, loss[loss=0.1, simple_loss=0.1268, pruned_loss=0.02629, audio_tagging_loss=0.01034, over 14683.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09356, pruned_loss=0.01476, audio_tagging_loss=0.009363, over 3042210.18 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:43:07,865 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303000
2023-11-22 16:43:15,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2020026.6666666667, ans=0.125
2023-11-22 16:43:24,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2020026.6666666667, ans=0.2
2023-11-22 16:43:27,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2020093.3333333333, ans=0.07
2023-11-22 16:43:31,206 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.331e+01 8.278e+01 8.806e+01 9.570e+01 1.233e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 16:43:56,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.21 vs. limit=15.0
2023-11-22 16:44:04,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2020293.3333333333, ans=0.0
2023-11-22 16:44:05,438 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2450, loss[loss=0.07151, simple_loss=0.08634, pruned_loss=0.02028, audio_tagging_loss=0.008062, over 15450.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09332, pruned_loss=0.01488, audio_tagging_loss=0.009491, over 3041534.91 frames. ], batch size: 63, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:44:10,417 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303050
2023-11-22 16:44:29,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2020426.6666666667, ans=0.0
2023-11-22 16:44:37,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2020426.6666666667, ans=0.0
2023-11-22 16:44:38,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2020426.6666666667, ans=0.0
2023-11-22 16:44:59,725 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:45:08,173 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2500, loss[loss=0.09093, simple_loss=0.1236, pruned_loss=0.01989, audio_tagging_loss=0.009246, over 15866.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09253, pruned_loss=0.01472, audio_tagging_loss=0.009633, over 3038997.69 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:45:08,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2020626.6666666667, ans=0.1
2023-11-22 16:45:13,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303100
2023-11-22 16:45:32,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2020760.0, ans=0.0
2023-11-22 16:45:37,911 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.127e+01 8.699e+01 9.305e+01 1.342e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 16:45:39,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2020760.0, ans=0.125
2023-11-22 16:45:45,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2020826.6666666667, ans=0.0
2023-11-22 16:45:48,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2020826.6666666667, ans=0.5
2023-11-22 16:46:12,050 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2550, loss[loss=0.07244, simple_loss=0.105, pruned_loss=0.01379, audio_tagging_loss=0.006142, over 15469.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09371, pruned_loss=0.01486, audio_tagging_loss=0.009431, over 3038032.35 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:46:16,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303150
2023-11-22 16:46:23,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2021026.6666666667, ans=0.2
2023-11-22 16:46:23,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.74 vs. limit=22.5
2023-11-22 16:46:26,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2021026.6666666667, ans=0.1
2023-11-22 16:46:33,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2021026.6666666667, ans=0.2
2023-11-22 16:46:47,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2021093.3333333333, ans=0.125
2023-11-22 16:46:51,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2021160.0, ans=0.125
2023-11-22 16:47:15,969 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2600, loss[loss=0.07606, simple_loss=0.1006, pruned_loss=0.01732, audio_tagging_loss=0.008453, over 14416.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09365, pruned_loss=0.01483, audio_tagging_loss=0.009324, over 3039831.60 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:47:20,929 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303200
2023-11-22 16:47:21,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.70 vs. limit=12.0
2023-11-22 16:47:44,775 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 8.118e+01 8.704e+01 9.634e+01 1.548e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 16:47:50,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2021426.6666666667, ans=0.1
2023-11-22 16:47:54,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2021493.3333333333, ans=0.125
2023-11-22 16:48:13,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2021560.0, ans=0.0
2023-11-22 16:48:18,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2021626.6666666667, ans=0.125
2023-11-22 16:48:19,344 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2650, loss[loss=0.06698, simple_loss=0.08576, pruned_loss=0.01379, audio_tagging_loss=0.01032, over 14555.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09258, pruned_loss=0.01464, audio_tagging_loss=0.009274, over 3036109.89 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:48:23,872 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.60 vs. limit=22.5
2023-11-22 16:48:24,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303250
2023-11-22 16:48:29,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2021626.6666666667, ans=0.07
2023-11-22 16:48:30,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.51 vs. limit=15.0
2023-11-22 16:48:31,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2021693.3333333333, ans=0.95
2023-11-22 16:48:39,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2021693.3333333333, ans=0.125
2023-11-22 16:48:43,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2021693.3333333333, ans=0.125
2023-11-22 16:49:05,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=2021826.6666666667, ans=0.2
2023-11-22 16:49:23,153 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2700, loss[loss=0.06426, simple_loss=0.08441, pruned_loss=0.01223, audio_tagging_loss=0.009828, over 15587.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09295, pruned_loss=0.01474, audio_tagging_loss=0.009143, over 3041561.75 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:49:25,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.08 vs. limit=15.0
2023-11-22 16:49:26,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2021960.0, ans=0.2
2023-11-22 16:49:28,822 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303300
2023-11-22 16:49:44,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2022026.6666666667, ans=0.0
2023-11-22 16:49:47,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2022093.3333333333, ans=0.0
2023-11-22 16:49:51,932 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.09 vs. limit=22.5
2023-11-22 16:49:53,747 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.226e+01 8.902e+01 9.755e+01 1.486e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 16:50:14,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2022226.6666666667, ans=0.125
2023-11-22 16:50:20,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2022226.6666666667, ans=0.125
2023-11-22 16:50:26,594 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2750, loss[loss=0.07479, simple_loss=0.09335, pruned_loss=0.01614, audio_tagging_loss=0.01198, over 14800.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09354, pruned_loss=0.0147, audio_tagging_loss=0.009051, over 3040483.11 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 8.0
2023-11-22 16:50:31,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303350
2023-11-22 16:50:49,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2022360.0, ans=0.0
2023-11-22 16:50:57,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2022426.6666666667, ans=0.125
2023-11-22 16:51:05,881 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.16 vs. limit=10.0
2023-11-22 16:51:10,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2022493.3333333333, ans=0.0
2023-11-22 16:51:19,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2022560.0, ans=0.0
2023-11-22 16:51:21,016 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:51:30,232 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2800, loss[loss=0.06168, simple_loss=0.08359, pruned_loss=0.01127, audio_tagging_loss=0.008611, over 14430.00 frames. ], tot_loss[loss=0.07, simple_loss=0.0929, pruned_loss=0.01448, audio_tagging_loss=0.009064, over 3038134.49 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:51:35,410 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303400
2023-11-22 16:51:38,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2022626.6666666667, ans=0.0
2023-11-22 16:51:43,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2022693.3333333333, ans=0.0
2023-11-22 16:51:54,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2022693.3333333333, ans=0.2
2023-11-22 16:52:00,957 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.48 vs. limit=15.0
2023-11-22 16:52:01,430 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.550e+01 8.038e+01 8.757e+01 9.417e+01 1.241e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 16:52:34,469 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2850, loss[loss=0.05531, simple_loss=0.07574, pruned_loss=0.01066, audio_tagging_loss=0.006787, over 14871.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09264, pruned_loss=0.01425, audio_tagging_loss=0.009119, over 3038310.78 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:52:36,588 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0
2023-11-22 16:52:38,945 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.80 vs. limit=15.0
2023-11-22 16:52:40,036 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303450
2023-11-22 16:52:45,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2022960.0, ans=0.0
2023-11-22 16:52:47,986 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.65 vs. limit=15.0
2023-11-22 16:52:52,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.58 vs. limit=22.5
2023-11-22 16:53:02,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2023093.3333333333, ans=0.1
2023-11-22 16:53:07,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2023093.3333333333, ans=0.0
2023-11-22 16:53:24,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2023226.6666666667, ans=0.0
2023-11-22 16:53:25,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.22 vs. limit=15.0
2023-11-22 16:53:37,519 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2900, loss[loss=0.05829, simple_loss=0.07228, pruned_loss=0.01284, audio_tagging_loss=0.009317, over 15594.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09326, pruned_loss=0.01465, audio_tagging_loss=0.009119, over 3040171.39 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:53:40,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2023293.3333333333, ans=0.0
2023-11-22 16:53:42,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303500
2023-11-22 16:54:07,705 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.261e+01 8.993e+01 9.849e+01 1.244e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 16:54:28,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2023560.0, ans=0.0
2023-11-22 16:54:39,786 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 2950, loss[loss=0.09273, simple_loss=0.1247, pruned_loss=0.02326, audio_tagging_loss=0.007141, over 14546.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09357, pruned_loss=0.01482, audio_tagging_loss=0.009176, over 3047865.33 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:54:45,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303550
2023-11-22 16:54:59,904 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.84 vs. limit=6.0
2023-11-22 16:55:00,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2023693.3333333333, ans=0.0
2023-11-22 16:55:23,628 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-22 16:55:28,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2023826.6666666667, ans=0.125
2023-11-22 16:55:43,915 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3000, loss[loss=0.06541, simple_loss=0.08431, pruned_loss=0.01337, audio_tagging_loss=0.009885, over 15281.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09324, pruned_loss=0.01481, audio_tagging_loss=0.009196, over 3049289.59 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:55:43,915 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 16:56:11,674 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.9908, 5.8631, 5.6633, 5.5927], device='cuda:2')
2023-11-22 16:56:24,257 INFO [train_asr.py:1253] (2/4) Epoch 26, validation: loss=0.05863, simple_loss=0.05148, pruned_loss=0.005087, audio_tagging_loss=0.0278, over 4681554.00 frames. 
2023-11-22 16:56:24,258 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 16:56:26,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.09 vs. limit=10.0
2023-11-22 16:56:29,254 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303600
2023-11-22 16:56:39,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2024026.6666666667, ans=0.125
2023-11-22 16:56:55,187 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.309e+01 8.958e+01 9.598e+01 2.915e+02, threshold=1.792e+02, percent-clipped=1.0
2023-11-22 16:56:57,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0
2023-11-22 16:57:08,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2024160.0, ans=0.125
2023-11-22 16:57:09,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2024160.0, ans=0.125
2023-11-22 16:57:23,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2024226.6666666667, ans=0.0
2023-11-22 16:57:27,852 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3050, loss[loss=0.08425, simple_loss=0.1136, pruned_loss=0.02028, audio_tagging_loss=0.007165, over 14568.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09417, pruned_loss=0.01495, audio_tagging_loss=0.009206, over 3047538.78 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:57:33,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303650
2023-11-22 16:57:45,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2024360.0, ans=0.09899494936611666
2023-11-22 16:57:50,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2024360.0, ans=0.125
2023-11-22 16:57:52,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2024360.0, ans=0.125
2023-11-22 16:57:55,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2024426.6666666667, ans=0.125
2023-11-22 16:58:06,781 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:58:18,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2024560.0, ans=0.0
2023-11-22 16:58:19,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2024560.0, ans=0.125
2023-11-22 16:58:26,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2024560.0, ans=0.0
2023-11-22 16:58:33,076 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3100, loss[loss=0.07295, simple_loss=0.09243, pruned_loss=0.01699, audio_tagging_loss=0.00974, over 15325.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09517, pruned_loss=0.01527, audio_tagging_loss=0.009234, over 3042387.00 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:58:38,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303700
2023-11-22 16:58:51,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2024693.3333333333, ans=0.125
2023-11-22 16:58:54,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2024693.3333333333, ans=0.125
2023-11-22 16:58:56,498 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.73 vs. limit=15.0
2023-11-22 16:58:57,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.33 vs. limit=22.5
2023-11-22 16:59:03,005 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.458e+01 8.373e+01 8.955e+01 9.319e+01 1.385e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 16:59:27,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2024893.3333333333, ans=0.0
2023-11-22 16:59:36,710 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3150, loss[loss=0.051, simple_loss=0.06506, pruned_loss=0.006749, audio_tagging_loss=0.01172, over 16237.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09505, pruned_loss=0.01516, audio_tagging_loss=0.009293, over 3039249.08 frames. ], batch size: 63, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:59:41,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303750
2023-11-22 16:59:49,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2025026.6666666667, ans=0.1
2023-11-22 17:00:05,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2025093.3333333333, ans=0.1
2023-11-22 17:00:16,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2025160.0, ans=0.1
2023-11-22 17:00:18,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.31 vs. limit=15.0
2023-11-22 17:00:39,343 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3200, loss[loss=0.06536, simple_loss=0.082, pruned_loss=0.01361, audio_tagging_loss=0.01074, over 15328.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09468, pruned_loss=0.01507, audio_tagging_loss=0.009346, over 3036637.91 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:00:44,343 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303800
2023-11-22 17:00:55,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2025360.0, ans=0.09899494936611666
2023-11-22 17:01:00,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2025360.0, ans=0.1
2023-11-22 17:01:10,782 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.434e+01 8.209e+01 8.808e+01 9.580e+01 1.231e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 17:01:14,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2025426.6666666667, ans=0.125
2023-11-22 17:01:20,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2025493.3333333333, ans=0.2
2023-11-22 17:01:25,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2025493.3333333333, ans=0.0
2023-11-22 17:01:34,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2025560.0, ans=0.0
2023-11-22 17:01:34,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=12.0
2023-11-22 17:01:42,691 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3250, loss[loss=0.06467, simple_loss=0.08627, pruned_loss=0.01052, audio_tagging_loss=0.01102, over 14807.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09316, pruned_loss=0.01472, audio_tagging_loss=0.009498, over 3031075.33 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:01:47,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2025626.6666666667, ans=0.0
2023-11-22 17:01:48,402 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303850
2023-11-22 17:01:55,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2025693.3333333333, ans=0.1
2023-11-22 17:02:21,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2025826.6666666667, ans=0.125
2023-11-22 17:02:27,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2025826.6666666667, ans=0.1
2023-11-22 17:02:46,299 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3300, loss[loss=0.0704, simple_loss=0.08497, pruned_loss=0.01665, audio_tagging_loss=0.01127, over 15346.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09251, pruned_loss=0.01457, audio_tagging_loss=0.009605, over 3036560.22 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:02:49,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2025960.0, ans=0.125
2023-11-22 17:02:51,229 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303900
2023-11-22 17:02:56,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2025960.0, ans=0.1
2023-11-22 17:03:12,138 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:03:12,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2026093.3333333333, ans=0.1
2023-11-22 17:03:16,784 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.797e+01 8.259e+01 8.793e+01 9.669e+01 1.578e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 17:03:26,158 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=15.0
2023-11-22 17:03:28,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2026160.0, ans=0.125
2023-11-22 17:03:30,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.46 vs. limit=15.0
2023-11-22 17:03:32,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2026160.0, ans=0.0
2023-11-22 17:03:32,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2026160.0, ans=0.125
2023-11-22 17:03:35,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2026226.6666666667, ans=0.125
2023-11-22 17:03:47,565 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.09 vs. limit=22.5
2023-11-22 17:03:49,232 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3350, loss[loss=0.08155, simple_loss=0.1103, pruned_loss=0.01806, audio_tagging_loss=0.008343, over 15720.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09271, pruned_loss=0.01472, audio_tagging_loss=0.009518, over 3043512.17 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:03:54,296 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 303950
2023-11-22 17:04:21,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2026426.6666666667, ans=0.125
2023-11-22 17:04:42,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2026560.0, ans=0.1
2023-11-22 17:04:51,528 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3400, loss[loss=0.06, simple_loss=0.07167, pruned_loss=0.015, audio_tagging_loss=0.009158, over 14666.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09356, pruned_loss=0.01479, audio_tagging_loss=0.009222, over 3044497.18 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:04:55,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.05 vs. limit=15.0
2023-11-22 17:04:56,957 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304000
2023-11-22 17:05:02,937 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.06 vs. limit=12.0
2023-11-22 17:05:08,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2026693.3333333333, ans=0.125
2023-11-22 17:05:10,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2026693.3333333333, ans=0.0
2023-11-22 17:05:10,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2026693.3333333333, ans=0.125
2023-11-22 17:05:26,254 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.152e+01 8.823e+01 9.410e+01 1.182e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 17:05:39,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2026826.6666666667, ans=0.125
2023-11-22 17:05:49,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.35 vs. limit=22.5
2023-11-22 17:05:58,798 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3450, loss[loss=0.06019, simple_loss=0.08462, pruned_loss=0.01069, audio_tagging_loss=0.007195, over 14603.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09337, pruned_loss=0.01479, audio_tagging_loss=0.009171, over 3045784.52 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:06:01,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2026960.0, ans=0.1
2023-11-22 17:06:03,736 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304050
2023-11-22 17:06:13,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2027026.6666666667, ans=0.125
2023-11-22 17:06:18,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2027026.6666666667, ans=0.0
2023-11-22 17:06:42,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2027160.0, ans=0.0
2023-11-22 17:07:01,446 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3500, loss[loss=0.07253, simple_loss=0.08913, pruned_loss=0.01827, audio_tagging_loss=0.009691, over 15563.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.0937, pruned_loss=0.01484, audio_tagging_loss=0.009031, over 3045893.12 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:07:06,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304100
2023-11-22 17:07:10,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2027293.3333333333, ans=0.125
2023-11-22 17:07:33,997 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.704e+01 8.467e+01 8.969e+01 9.686e+01 1.239e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 17:07:36,545 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:07:58,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2027560.0, ans=0.125
2023-11-22 17:08:04,509 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3550, loss[loss=0.07762, simple_loss=0.1098, pruned_loss=0.01638, audio_tagging_loss=0.006329, over 14464.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09304, pruned_loss=0.01461, audio_tagging_loss=0.009027, over 3033888.17 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:08:04,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2027626.6666666667, ans=0.0
2023-11-22 17:08:07,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2027626.6666666667, ans=0.125
2023-11-22 17:08:10,121 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304150
2023-11-22 17:08:10,711 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.61 vs. limit=12.0
2023-11-22 17:08:12,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2027626.6666666667, ans=0.125
2023-11-22 17:08:56,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2027893.3333333333, ans=0.125
2023-11-22 17:08:58,637 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:09:08,653 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3600, loss[loss=0.06901, simple_loss=0.09469, pruned_loss=0.01405, audio_tagging_loss=0.007615, over 16122.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09311, pruned_loss=0.01466, audio_tagging_loss=0.009051, over 3035050.42 frames. ], batch size: 62, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:09:14,325 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304200
2023-11-22 17:09:41,227 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.567e+01 8.177e+01 8.783e+01 9.582e+01 1.117e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 17:10:06,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2028226.6666666667, ans=0.1
2023-11-22 17:10:13,381 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3650, loss[loss=0.0552, simple_loss=0.07735, pruned_loss=0.0104, audio_tagging_loss=0.006123, over 15013.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09314, pruned_loss=0.01465, audio_tagging_loss=0.009005, over 3040253.77 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:10:18,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304250
2023-11-22 17:10:26,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2028360.0, ans=0.1
2023-11-22 17:10:29,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2028360.0, ans=0.125
2023-11-22 17:10:30,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.16 vs. limit=22.5
2023-11-22 17:10:32,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2028360.0, ans=0.125
2023-11-22 17:10:41,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2028426.6666666667, ans=0.125
2023-11-22 17:10:59,074 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.17 vs. limit=15.0
2023-11-22 17:11:03,912 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-22 17:11:13,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2028560.0, ans=0.125
2023-11-22 17:11:16,820 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3700, loss[loss=0.0589, simple_loss=0.07271, pruned_loss=0.01294, audio_tagging_loss=0.009612, over 15954.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09339, pruned_loss=0.0147, audio_tagging_loss=0.009022, over 3046826.04 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:11:21,762 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304300
2023-11-22 17:11:50,444 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.264e+01 8.888e+01 9.594e+01 1.594e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 17:12:06,282 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.17 vs. limit=15.0
2023-11-22 17:12:21,546 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3750, loss[loss=0.08903, simple_loss=0.1124, pruned_loss=0.02395, audio_tagging_loss=0.008868, over 15276.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09282, pruned_loss=0.0146, audio_tagging_loss=0.009171, over 3046880.23 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:12:26,511 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304350
2023-11-22 17:12:42,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2029026.6666666667, ans=0.125
2023-11-22 17:12:57,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2029160.0, ans=0.04949747468305833
2023-11-22 17:13:05,481 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:13:05,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2029160.0, ans=0.125
2023-11-22 17:13:19,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.75 vs. limit=15.0
2023-11-22 17:13:25,466 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3800, loss[loss=0.05905, simple_loss=0.07744, pruned_loss=0.01066, audio_tagging_loss=0.009667, over 15175.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.0924, pruned_loss=0.01459, audio_tagging_loss=0.009327, over 3051136.64 frames. ], batch size: 61, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:13:25,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2029293.3333333333, ans=0.07
2023-11-22 17:13:30,572 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304400
2023-11-22 17:13:50,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2029426.6666666667, ans=0.125
2023-11-22 17:14:00,217 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.081e+01 8.257e+01 8.903e+01 9.664e+01 1.355e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 17:14:09,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2029493.3333333333, ans=0.1
2023-11-22 17:14:24,178 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.96 vs. limit=6.0
2023-11-22 17:14:30,793 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3850, loss[loss=0.09514, simple_loss=0.1271, pruned_loss=0.02373, audio_tagging_loss=0.007867, over 13731.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09256, pruned_loss=0.0146, audio_tagging_loss=0.009431, over 3048212.28 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:14:31,326 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.70 vs. limit=12.0
2023-11-22 17:14:33,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2029626.6666666667, ans=0.2
2023-11-22 17:14:35,758 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304450
2023-11-22 17:14:59,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2029760.0, ans=0.0
2023-11-22 17:15:02,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2029760.0, ans=10.0
2023-11-22 17:15:09,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.17 vs. limit=15.0
2023-11-22 17:15:15,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2029826.6666666667, ans=0.0
2023-11-22 17:15:15,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0
2023-11-22 17:15:33,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2029893.3333333333, ans=0.125
2023-11-22 17:15:35,613 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3900, loss[loss=0.06447, simple_loss=0.08976, pruned_loss=0.01135, audio_tagging_loss=0.008238, over 15211.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09179, pruned_loss=0.01438, audio_tagging_loss=0.009471, over 3042054.44 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:15:41,107 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304500
2023-11-22 17:15:52,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2030026.6666666667, ans=0.0
2023-11-22 17:15:59,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2030026.6666666667, ans=0.2
2023-11-22 17:16:07,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2030093.3333333333, ans=0.125
2023-11-22 17:16:08,779 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.197e+01 8.799e+01 9.518e+01 1.700e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 17:16:18,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2030160.0, ans=0.2
2023-11-22 17:16:31,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2030226.6666666667, ans=0.2
2023-11-22 17:16:35,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2030226.6666666667, ans=0.0
2023-11-22 17:16:39,732 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 3950, loss[loss=0.07127, simple_loss=0.09546, pruned_loss=0.01268, audio_tagging_loss=0.01086, over 14565.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09234, pruned_loss=0.01438, audio_tagging_loss=0.009428, over 3035607.45 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:16:42,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2030293.3333333333, ans=0.125
2023-11-22 17:16:44,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304550
2023-11-22 17:16:46,578 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-22 17:16:52,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2030360.0, ans=0.035
2023-11-22 17:17:05,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2030426.6666666667, ans=0.125
2023-11-22 17:17:11,485 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=15.0
2023-11-22 17:17:16,875 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.66 vs. limit=22.5
2023-11-22 17:17:18,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2030493.3333333333, ans=0.2
2023-11-22 17:17:19,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2030493.3333333333, ans=0.0
2023-11-22 17:17:43,616 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4000, loss[loss=0.06291, simple_loss=0.08208, pruned_loss=0.01097, audio_tagging_loss=0.0109, over 15630.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09264, pruned_loss=0.0144, audio_tagging_loss=0.009493, over 3038410.36 frames. ], batch size: 61, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:17:43,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2030626.6666666667, ans=0.125
2023-11-22 17:17:48,565 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304600
2023-11-22 17:17:48,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=12.0
2023-11-22 17:18:06,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2030693.3333333333, ans=0.125
2023-11-22 17:18:17,460 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.393e+01 9.096e+01 9.757e+01 1.242e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-22 17:18:29,806 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.43 vs. limit=15.0
2023-11-22 17:18:48,580 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4050, loss[loss=0.0679, simple_loss=0.0901, pruned_loss=0.01527, audio_tagging_loss=0.007589, over 14723.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09309, pruned_loss=0.0146, audio_tagging_loss=0.009449, over 3040008.89 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:18:52,376 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:18:54,237 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304650
2023-11-22 17:18:59,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2030960.0, ans=0.2
2023-11-22 17:19:14,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2031093.3333333333, ans=0.0
2023-11-22 17:19:29,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2031160.0, ans=0.025
2023-11-22 17:19:47,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2031226.6666666667, ans=0.125
2023-11-22 17:19:50,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2031226.6666666667, ans=0.5
2023-11-22 17:19:52,423 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4100, loss[loss=0.08076, simple_loss=0.1099, pruned_loss=0.01753, audio_tagging_loss=0.008275, over 16358.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09333, pruned_loss=0.01462, audio_tagging_loss=0.009458, over 3037186.97 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:19:57,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304700
2023-11-22 17:20:12,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2031360.0, ans=0.125
2023-11-22 17:20:12,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2031360.0, ans=0.125
2023-11-22 17:20:25,420 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.231e+01 8.289e+01 8.885e+01 9.485e+01 1.195e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 17:20:34,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2031493.3333333333, ans=0.125
2023-11-22 17:20:56,261 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4150, loss[loss=0.06761, simple_loss=0.08416, pruned_loss=0.01289, audio_tagging_loss=0.01264, over 14612.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09366, pruned_loss=0.01468, audio_tagging_loss=0.009392, over 3034761.13 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:20:57,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2031626.6666666667, ans=0.125
2023-11-22 17:21:01,811 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304750
2023-11-22 17:21:06,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2031626.6666666667, ans=0.0
2023-11-22 17:21:10,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2031693.3333333333, ans=0.125
2023-11-22 17:21:23,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2031760.0, ans=0.0
2023-11-22 17:21:38,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2031826.6666666667, ans=0.0
2023-11-22 17:21:43,416 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:22:00,329 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4200, loss[loss=0.06807, simple_loss=0.09224, pruned_loss=0.0126, audio_tagging_loss=0.009352, over 15626.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09348, pruned_loss=0.01467, audio_tagging_loss=0.009234, over 3042492.24 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:22:05,829 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304800
2023-11-22 17:22:07,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2031960.0, ans=0.125
2023-11-22 17:22:09,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2031960.0, ans=0.2
2023-11-22 17:22:13,489 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.86 vs. limit=8.0
2023-11-22 17:22:17,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2032026.6666666667, ans=0.125
2023-11-22 17:22:17,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2032026.6666666667, ans=0.125
2023-11-22 17:22:34,567 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.472e+01 8.973e+01 9.606e+01 1.148e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-22 17:22:35,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.01 vs. limit=15.0
2023-11-22 17:22:52,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2032226.6666666667, ans=0.125
2023-11-22 17:23:01,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2032226.6666666667, ans=0.0
2023-11-22 17:23:04,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.17 vs. limit=15.0
2023-11-22 17:23:04,800 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4250, loss[loss=0.06503, simple_loss=0.09029, pruned_loss=0.01178, audio_tagging_loss=0.008111, over 15248.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09361, pruned_loss=0.01461, audio_tagging_loss=0.009224, over 3044228.01 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:23:09,732 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304850
2023-11-22 17:23:15,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2032360.0, ans=0.125
2023-11-22 17:23:15,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2032360.0, ans=0.125
2023-11-22 17:23:25,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2032360.0, ans=0.125
2023-11-22 17:23:41,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=2032426.6666666667, ans=22.5
2023-11-22 17:23:56,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2032560.0, ans=0.0
2023-11-22 17:24:04,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2032560.0, ans=0.0
2023-11-22 17:24:07,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2032626.6666666667, ans=0.125
2023-11-22 17:24:08,358 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4300, loss[loss=0.0609, simple_loss=0.08057, pruned_loss=0.01048, audio_tagging_loss=0.01014, over 14356.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09396, pruned_loss=0.01468, audio_tagging_loss=0.009099, over 3047343.03 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:24:09,202 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.36 vs. limit=15.0
2023-11-22 17:24:13,363 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304900
2023-11-22 17:24:31,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2032693.3333333333, ans=0.125
2023-11-22 17:24:44,066 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.878e+01 8.402e+01 8.884e+01 9.738e+01 1.155e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 17:24:55,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2032826.6666666667, ans=0.1
2023-11-22 17:25:07,331 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.44 vs. limit=12.0
2023-11-22 17:25:09,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2032893.3333333333, ans=0.0
2023-11-22 17:25:12,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2032960.0, ans=0.125
2023-11-22 17:25:13,334 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4350, loss[loss=0.06831, simple_loss=0.08455, pruned_loss=0.01439, audio_tagging_loss=0.01165, over 14746.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09456, pruned_loss=0.01492, audio_tagging_loss=0.009108, over 3050526.21 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:25:19,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 304950
2023-11-22 17:25:48,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.72 vs. limit=15.0
2023-11-22 17:25:51,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2033160.0, ans=0.0
2023-11-22 17:25:56,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2033160.0, ans=0.0
2023-11-22 17:26:08,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2033226.6666666667, ans=0.0
2023-11-22 17:26:18,665 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4400, loss[loss=0.05069, simple_loss=0.06229, pruned_loss=0.01104, audio_tagging_loss=0.008501, over 15521.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09505, pruned_loss=0.01505, audio_tagging_loss=0.008978, over 3046720.03 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:26:19,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2033293.3333333333, ans=0.125
2023-11-22 17:26:23,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305000
2023-11-22 17:26:27,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2033293.3333333333, ans=0.0
2023-11-22 17:26:36,530 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.06 vs. limit=15.0
2023-11-22 17:26:52,035 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.342e+01 8.115e+01 8.926e+01 9.630e+01 1.276e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 17:27:05,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2033493.3333333333, ans=0.1
2023-11-22 17:27:06,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2033493.3333333333, ans=0.125
2023-11-22 17:27:22,547 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4450, loss[loss=0.06571, simple_loss=0.08924, pruned_loss=0.01497, audio_tagging_loss=0.006114, over 15387.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.095, pruned_loss=0.01505, audio_tagging_loss=0.008976, over 3051067.16 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:27:27,564 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305050
2023-11-22 17:27:54,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2033760.0, ans=0.125
2023-11-22 17:28:24,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2033960.0, ans=0.125
2023-11-22 17:28:25,535 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4500, loss[loss=0.08102, simple_loss=0.1106, pruned_loss=0.01637, audio_tagging_loss=0.00937, over 15322.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09401, pruned_loss=0.01488, audio_tagging_loss=0.008985, over 3046834.54 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:28:31,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305100
2023-11-22 17:28:38,021 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.80 vs. limit=15.0
2023-11-22 17:28:41,725 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:29:01,029 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.181e+01 8.834e+01 9.479e+01 1.227e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 17:29:12,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2034160.0, ans=0.07
2023-11-22 17:29:20,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2034226.6666666667, ans=0.2
2023-11-22 17:29:31,642 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4550, loss[loss=0.07515, simple_loss=0.1054, pruned_loss=0.01478, audio_tagging_loss=0.007656, over 15511.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09347, pruned_loss=0.01483, audio_tagging_loss=0.009057, over 3048393.67 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:29:34,826 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.53 vs. limit=15.0
2023-11-22 17:29:36,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305150
2023-11-22 17:29:37,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.35 vs. limit=22.5
2023-11-22 17:29:38,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2034293.3333333333, ans=0.125
2023-11-22 17:29:44,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2034360.0, ans=0.2
2023-11-22 17:29:45,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2034360.0, ans=0.0
2023-11-22 17:29:58,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2034426.6666666667, ans=10.0
2023-11-22 17:30:07,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2034493.3333333333, ans=0.0
2023-11-22 17:30:20,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2034493.3333333333, ans=0.125
2023-11-22 17:30:21,210 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:30:34,626 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4600, loss[loss=0.0699, simple_loss=0.09128, pruned_loss=0.01528, audio_tagging_loss=0.008985, over 15108.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09302, pruned_loss=0.015, audio_tagging_loss=0.009151, over 3047170.92 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:30:36,631 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.95 vs. limit=12.0
2023-11-22 17:30:39,716 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305200
2023-11-22 17:30:43,147 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.57 vs. limit=6.0
2023-11-22 17:30:54,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2034693.3333333333, ans=0.0
2023-11-22 17:31:04,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2034760.0, ans=0.07
2023-11-22 17:31:08,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2034760.0, ans=0.2
2023-11-22 17:31:11,474 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.052e+01 8.212e+01 8.747e+01 9.317e+01 1.226e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 17:31:26,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2034893.3333333333, ans=15.0
2023-11-22 17:31:37,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2034960.0, ans=0.07
2023-11-22 17:31:38,365 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4650, loss[loss=0.07511, simple_loss=0.09251, pruned_loss=0.01837, audio_tagging_loss=0.01048, over 13722.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09311, pruned_loss=0.01511, audio_tagging_loss=0.009278, over 3043962.20 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:31:43,268 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305250
2023-11-22 17:31:43,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2034960.0, ans=0.125
2023-11-22 17:32:10,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2035093.3333333333, ans=0.2
2023-11-22 17:32:22,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2035160.0, ans=0.2
2023-11-22 17:32:39,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2035226.6666666667, ans=0.0
2023-11-22 17:32:43,677 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4700, loss[loss=0.07045, simple_loss=0.0898, pruned_loss=0.01496, audio_tagging_loss=0.01059, over 15676.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09276, pruned_loss=0.01502, audio_tagging_loss=0.009337, over 3047017.60 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:32:45,089 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:32:49,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305300
2023-11-22 17:33:01,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2035360.0, ans=0.1
2023-11-22 17:33:05,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2035360.0, ans=0.125
2023-11-22 17:33:06,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=5.99 vs. limit=12.0
2023-11-22 17:33:18,409 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.537e+01 8.112e+01 8.766e+01 9.442e+01 1.103e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 17:33:33,083 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.88 vs. limit=15.0
2023-11-22 17:33:34,075 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:33:36,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2035560.0, ans=0.125
2023-11-22 17:33:46,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2035626.6666666667, ans=0.125
2023-11-22 17:33:47,889 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4750, loss[loss=0.07095, simple_loss=0.094, pruned_loss=0.01721, audio_tagging_loss=0.006737, over 15449.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09355, pruned_loss=0.01502, audio_tagging_loss=0.0094, over 3043955.58 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:33:52,952 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305350
2023-11-22 17:33:55,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2035626.6666666667, ans=0.125
2023-11-22 17:34:05,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2035693.3333333333, ans=0.0
2023-11-22 17:34:20,384 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.36 vs. limit=12.0
2023-11-22 17:34:21,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2035760.0, ans=0.0
2023-11-22 17:34:35,274 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.18 vs. limit=15.0
2023-11-22 17:34:51,753 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4800, loss[loss=0.05585, simple_loss=0.07244, pruned_loss=0.008635, audio_tagging_loss=0.01099, over 15639.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09316, pruned_loss=0.01477, audio_tagging_loss=0.009501, over 3041952.03 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:34:56,890 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305400
2023-11-22 17:35:02,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2035960.0, ans=0.125
2023-11-22 17:35:26,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2036093.3333333333, ans=0.125
2023-11-22 17:35:28,867 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.281e+01 8.887e+01 9.672e+01 1.150e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 17:35:57,061 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4850, loss[loss=0.09246, simple_loss=0.1297, pruned_loss=0.01882, audio_tagging_loss=0.008778, over 15261.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09339, pruned_loss=0.0148, audio_tagging_loss=0.009539, over 3040752.60 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:36:02,717 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305450
2023-11-22 17:36:33,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2036493.3333333333, ans=0.2
2023-11-22 17:36:47,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2036560.0, ans=0.2
2023-11-22 17:36:48,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2036560.0, ans=0.1
2023-11-22 17:36:57,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2036560.0, ans=0.125
2023-11-22 17:36:58,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2036560.0, ans=0.125
2023-11-22 17:36:58,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2036560.0, ans=0.04949747468305833
2023-11-22 17:37:01,306 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4900, loss[loss=0.08848, simple_loss=0.1199, pruned_loss=0.02165, audio_tagging_loss=0.006881, over 14750.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09328, pruned_loss=0.0148, audio_tagging_loss=0.009491, over 3039214.87 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:37:06,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305500
2023-11-22 17:37:38,243 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.464e+01 8.092e+01 8.921e+01 9.687e+01 1.263e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 17:37:49,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2036826.6666666667, ans=0.125
2023-11-22 17:38:01,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.43 vs. limit=15.0
2023-11-22 17:38:04,892 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 4950, loss[loss=0.07172, simple_loss=0.09046, pruned_loss=0.01452, audio_tagging_loss=0.01198, over 14690.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09394, pruned_loss=0.01488, audio_tagging_loss=0.009332, over 3043153.39 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:38:09,228 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.20 vs. limit=15.0
2023-11-22 17:38:09,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305550
2023-11-22 17:38:10,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2036960.0, ans=0.1
2023-11-22 17:38:12,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2036960.0, ans=0.0
2023-11-22 17:38:53,372 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:38:55,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2037226.6666666667, ans=0.0
2023-11-22 17:38:57,630 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.31 vs. limit=22.5
2023-11-22 17:38:59,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2037226.6666666667, ans=0.125
2023-11-22 17:39:10,267 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5000, loss[loss=0.07618, simple_loss=0.09777, pruned_loss=0.01682, audio_tagging_loss=0.01048, over 15838.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09416, pruned_loss=0.01502, audio_tagging_loss=0.009195, over 3048781.23 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:39:15,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305600
2023-11-22 17:39:32,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2037360.0, ans=0.125
2023-11-22 17:39:42,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2037426.6666666667, ans=0.0
2023-11-22 17:39:42,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2037426.6666666667, ans=0.1
2023-11-22 17:39:47,348 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.198e+01 8.797e+01 9.538e+01 1.250e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 17:39:52,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2037493.3333333333, ans=0.125
2023-11-22 17:39:55,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2037493.3333333333, ans=0.125
2023-11-22 17:40:15,643 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5050, loss[loss=0.06713, simple_loss=0.09187, pruned_loss=0.0109, audio_tagging_loss=0.01029, over 16117.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.0942, pruned_loss=0.01499, audio_tagging_loss=0.00918, over 3045699.55 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:40:21,488 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305650
2023-11-22 17:40:31,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2037693.3333333333, ans=0.125
2023-11-22 17:40:36,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2037693.3333333333, ans=0.125
2023-11-22 17:40:55,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.66 vs. limit=22.5
2023-11-22 17:41:40,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2037893.3333333333, ans=0.125
2023-11-22 17:41:46,527 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5100, loss[loss=0.06485, simple_loss=0.085, pruned_loss=0.01148, audio_tagging_loss=0.01087, over 15894.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09386, pruned_loss=0.01499, audio_tagging_loss=0.009107, over 3048150.94 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 8.0
2023-11-22 17:41:47,045 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.28 vs. limit=15.0
2023-11-22 17:41:54,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305700
2023-11-22 17:42:21,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2038093.3333333333, ans=0.2
2023-11-22 17:42:41,744 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.755e+01 8.235e+01 8.784e+01 9.418e+01 1.512e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 17:42:53,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.50 vs. limit=15.0
2023-11-22 17:42:54,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2038160.0, ans=0.0
2023-11-22 17:43:14,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2038226.6666666667, ans=0.125
2023-11-22 17:43:19,583 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5150, loss[loss=0.07078, simple_loss=0.101, pruned_loss=0.01325, audio_tagging_loss=0.007049, over 14645.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09334, pruned_loss=0.01481, audio_tagging_loss=0.009116, over 3046310.02 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 8.0
2023-11-22 17:43:27,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305750
2023-11-22 17:43:27,421 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:43:36,772 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0
2023-11-22 17:43:49,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2038360.0, ans=0.07
2023-11-22 17:44:10,612 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.23 vs. limit=15.0
2023-11-22 17:44:21,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2038493.3333333333, ans=0.0
2023-11-22 17:44:44,825 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:44:52,035 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5200, loss[loss=0.0822, simple_loss=0.09785, pruned_loss=0.0215, audio_tagging_loss=0.01178, over 14100.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09359, pruned_loss=0.01478, audio_tagging_loss=0.009058, over 3037820.27 frames. ], batch size: 53, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:44:59,513 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305800
2023-11-22 17:45:06,023 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.95 vs. limit=12.0
2023-11-22 17:45:11,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.09 vs. limit=15.0
2023-11-22 17:45:13,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2038693.3333333333, ans=0.125
2023-11-22 17:45:48,134 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.015e+01 8.193e+01 8.836e+01 9.389e+01 1.241e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 17:46:01,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2038826.6666666667, ans=0.1
2023-11-22 17:46:24,762 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5250, loss[loss=0.08432, simple_loss=0.1284, pruned_loss=0.01457, audio_tagging_loss=0.005545, over 15553.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09437, pruned_loss=0.01494, audio_tagging_loss=0.009011, over 3038332.98 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:46:32,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305850
2023-11-22 17:46:32,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2038960.0, ans=0.125
2023-11-22 17:46:36,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2038960.0, ans=0.125
2023-11-22 17:47:44,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2039226.6666666667, ans=0.1
2023-11-22 17:47:50,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2039226.6666666667, ans=0.1
2023-11-22 17:47:57,885 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5300, loss[loss=0.05342, simple_loss=0.06571, pruned_loss=0.00979, audio_tagging_loss=0.01078, over 14227.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09412, pruned_loss=0.01493, audio_tagging_loss=0.008973, over 3045951.68 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:48:05,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305900
2023-11-22 17:48:14,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2039360.0, ans=0.1
2023-11-22 17:48:29,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2039360.0, ans=0.125
2023-11-22 17:48:53,024 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.119e+01 8.483e+01 8.855e+01 9.483e+01 1.164e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 17:48:55,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2039493.3333333333, ans=0.1
2023-11-22 17:49:06,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2039493.3333333333, ans=0.0
2023-11-22 17:49:20,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.16 vs. limit=12.0
2023-11-22 17:49:30,731 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5350, loss[loss=0.07353, simple_loss=0.09548, pruned_loss=0.01565, audio_tagging_loss=0.01014, over 15655.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09461, pruned_loss=0.01503, audio_tagging_loss=0.00899, over 3045647.70 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:49:38,129 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 305950
2023-11-22 17:49:45,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2039626.6666666667, ans=0.0
2023-11-22 17:49:56,919 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.22 vs. limit=15.0
2023-11-22 17:50:04,448 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.54 vs. limit=15.0
2023-11-22 17:50:36,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.60 vs. limit=22.5
2023-11-22 17:50:51,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2039893.3333333333, ans=0.1
2023-11-22 17:51:03,649 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5400, loss[loss=0.08425, simple_loss=0.1106, pruned_loss=0.02117, audio_tagging_loss=0.007796, over 14928.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09294, pruned_loss=0.01478, audio_tagging_loss=0.009148, over 3052322.22 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:51:11,134 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306000
2023-11-22 17:51:23,081 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.73 vs. limit=15.0
2023-11-22 17:51:24,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2040026.6666666667, ans=0.05
2023-11-22 17:51:35,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2040026.6666666667, ans=0.07
2023-11-22 17:51:43,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2040093.3333333333, ans=0.0
2023-11-22 17:51:59,431 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.288e+01 8.843e+01 9.633e+01 1.187e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 17:52:10,823 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:52:36,544 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5450, loss[loss=0.08514, simple_loss=0.114, pruned_loss=0.02175, audio_tagging_loss=0.006392, over 15883.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09332, pruned_loss=0.0149, audio_tagging_loss=0.009229, over 3052088.13 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:52:44,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306050
2023-11-22 17:52:59,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2040360.0, ans=10.0
2023-11-22 17:53:07,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2040360.0, ans=0.05
2023-11-22 17:53:19,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2040426.6666666667, ans=0.125
2023-11-22 17:53:51,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2040560.0, ans=0.125
2023-11-22 17:53:55,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2040560.0, ans=0.125
2023-11-22 17:53:57,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2040560.0, ans=0.125
2023-11-22 17:54:10,066 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5500, loss[loss=0.06195, simple_loss=0.08149, pruned_loss=0.01222, audio_tagging_loss=0.008979, over 14132.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09364, pruned_loss=0.01515, audio_tagging_loss=0.009364, over 3046765.71 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:54:10,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2040626.6666666667, ans=0.0
2023-11-22 17:54:17,293 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306100
2023-11-22 17:54:22,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2040626.6666666667, ans=0.125
2023-11-22 17:54:35,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2040693.3333333333, ans=0.125
2023-11-22 17:54:51,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2040760.0, ans=0.2
2023-11-22 17:55:05,354 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.963e+01 8.451e+01 8.930e+01 9.689e+01 1.282e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 17:55:13,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2040826.6666666667, ans=0.0
2023-11-22 17:55:41,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2040960.0, ans=0.0
2023-11-22 17:55:41,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2040960.0, ans=0.0
2023-11-22 17:55:42,888 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5550, loss[loss=0.05129, simple_loss=0.0589, pruned_loss=0.01106, audio_tagging_loss=0.01078, over 14169.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09403, pruned_loss=0.01507, audio_tagging_loss=0.009407, over 3045913.69 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:55:50,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306150
2023-11-22 17:56:16,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2041026.6666666667, ans=10.0
2023-11-22 17:57:14,858 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5600, loss[loss=0.07133, simple_loss=0.09815, pruned_loss=0.01014, audio_tagging_loss=0.01211, over 16510.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.094, pruned_loss=0.01482, audio_tagging_loss=0.009492, over 3044959.45 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 17:57:17,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2041293.3333333333, ans=0.2
2023-11-22 17:57:22,301 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306200
2023-11-22 17:57:40,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2041360.0, ans=22.5
2023-11-22 17:58:11,312 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.138e+01 8.696e+01 9.555e+01 1.151e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 17:58:20,017 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:58:38,241 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5650, loss[loss=0.06251, simple_loss=0.07483, pruned_loss=0.01457, audio_tagging_loss=0.01052, over 15389.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09318, pruned_loss=0.01463, audio_tagging_loss=0.009576, over 3049699.62 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 17:58:43,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306250
2023-11-22 17:58:53,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2041693.3333333333, ans=0.0
2023-11-22 17:59:01,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2041693.3333333333, ans=0.125
2023-11-22 17:59:08,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2041760.0, ans=0.0
2023-11-22 17:59:15,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2041826.6666666667, ans=0.1
2023-11-22 17:59:17,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2041826.6666666667, ans=0.1
2023-11-22 17:59:18,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2041826.6666666667, ans=0.125
2023-11-22 17:59:29,603 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.98 vs. limit=10.0
2023-11-22 17:59:42,498 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5700, loss[loss=0.06462, simple_loss=0.08275, pruned_loss=0.01376, audio_tagging_loss=0.009486, over 15256.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09209, pruned_loss=0.01441, audio_tagging_loss=0.009623, over 3051776.51 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:59:47,618 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306300
2023-11-22 17:59:58,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2042026.6666666667, ans=0.125
2023-11-22 18:00:22,595 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.203e+01 8.769e+01 9.418e+01 1.173e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 18:00:46,244 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5750, loss[loss=0.06197, simple_loss=0.07997, pruned_loss=0.01285, audio_tagging_loss=0.00913, over 14974.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09216, pruned_loss=0.01448, audio_tagging_loss=0.009424, over 3043522.47 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:00:51,416 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306350
2023-11-22 18:00:55,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2042293.3333333333, ans=0.2
2023-11-22 18:00:58,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2042360.0, ans=0.0
2023-11-22 18:00:59,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2042360.0, ans=0.0
2023-11-22 18:01:00,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.64 vs. limit=22.5
2023-11-22 18:01:08,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2042360.0, ans=0.1
2023-11-22 18:01:37,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2042560.0, ans=0.125
2023-11-22 18:01:42,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2042560.0, ans=0.0
2023-11-22 18:01:51,688 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5800, loss[loss=0.07555, simple_loss=0.1001, pruned_loss=0.01652, audio_tagging_loss=0.008958, over 16004.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09245, pruned_loss=0.01462, audio_tagging_loss=0.009363, over 3038164.48 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:01:57,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306400
2023-11-22 18:02:30,901 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.67 vs. limit=15.0
2023-11-22 18:02:31,134 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.101e+01 8.686e+01 9.623e+01 1.169e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 18:02:31,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2042826.6666666667, ans=0.125
2023-11-22 18:02:44,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.38 vs. limit=15.0
2023-11-22 18:02:56,978 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5850, loss[loss=0.07984, simple_loss=0.1051, pruned_loss=0.01735, audio_tagging_loss=0.009961, over 16094.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09254, pruned_loss=0.0145, audio_tagging_loss=0.009287, over 3037127.54 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:03:02,014 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306450
2023-11-22 18:03:07,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2042960.0, ans=0.125
2023-11-22 18:03:09,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2043026.6666666667, ans=0.125
2023-11-22 18:03:20,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-22 18:03:22,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0
2023-11-22 18:04:06,858 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5900, loss[loss=0.0985, simple_loss=0.1407, pruned_loss=0.01893, audio_tagging_loss=0.009225, over 15223.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09335, pruned_loss=0.01463, audio_tagging_loss=0.009148, over 3037228.67 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:04:13,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306500
2023-11-22 18:04:42,817 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.31 vs. limit=22.5
2023-11-22 18:04:50,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2043426.6666666667, ans=0.125
2023-11-22 18:05:00,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2043493.3333333333, ans=0.125
2023-11-22 18:05:02,417 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.183e+01 8.699e+01 9.524e+01 1.128e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 18:05:22,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2043560.0, ans=0.2
2023-11-22 18:05:25,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2043560.0, ans=0.2
2023-11-22 18:05:35,780 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 5950, loss[loss=0.06176, simple_loss=0.08614, pruned_loss=0.01045, audio_tagging_loss=0.008241, over 16047.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09275, pruned_loss=0.01459, audio_tagging_loss=0.009235, over 3048124.19 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:05:43,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306550
2023-11-22 18:05:52,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=2043693.3333333333, ans=15.0
2023-11-22 18:05:56,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2043693.3333333333, ans=0.125
2023-11-22 18:07:05,351 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6000, loss[loss=0.0695, simple_loss=0.08784, pruned_loss=0.01706, audio_tagging_loss=0.008516, over 15254.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09355, pruned_loss=0.01486, audio_tagging_loss=0.009134, over 3049224.39 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:07:05,352 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 18:07:34,649 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.0051, 5.8811, 5.6599, 5.5967], device='cuda:2')
2023-11-22 18:07:56,216 INFO [train_asr.py:1253] (2/4) Epoch 26, validation: loss=0.05819, simple_loss=0.05149, pruned_loss=0.005105, audio_tagging_loss=0.02734, over 4681554.00 frames. 
2023-11-22 18:07:56,218 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 18:07:56,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2043960.0, ans=0.0
2023-11-22 18:08:01,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306600
2023-11-22 18:08:28,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2044093.3333333333, ans=0.125
2023-11-22 18:08:36,922 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.220e+01 8.809e+01 9.457e+01 1.359e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 18:08:43,044 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:08:46,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2044226.6666666667, ans=0.0
2023-11-22 18:08:58,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2044226.6666666667, ans=0.0
2023-11-22 18:09:01,884 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6050, loss[loss=0.07126, simple_loss=0.09692, pruned_loss=0.01464, audio_tagging_loss=0.008156, over 15405.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09349, pruned_loss=0.01487, audio_tagging_loss=0.009148, over 3043715.71 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:09:06,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306650
2023-11-22 18:09:38,132 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.01 vs. limit=15.0
2023-11-22 18:10:00,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2044560.0, ans=0.0
2023-11-22 18:10:01,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2044560.0, ans=0.2
2023-11-22 18:10:04,705 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6100, loss[loss=0.08191, simple_loss=0.1078, pruned_loss=0.01672, audio_tagging_loss=0.0113, over 15763.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09275, pruned_loss=0.01463, audio_tagging_loss=0.009279, over 3046353.78 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:10:06,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2044626.6666666667, ans=0.1
2023-11-22 18:10:10,225 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306700
2023-11-22 18:10:22,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff3.min_abs, batch_count=2044693.3333333333, ans=0.2
2023-11-22 18:10:46,189 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.417e+01 8.381e+01 8.850e+01 9.348e+01 1.200e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 18:10:55,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2044893.3333333333, ans=0.125
2023-11-22 18:11:03,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2044893.3333333333, ans=0.125
2023-11-22 18:11:08,984 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6150, loss[loss=0.05322, simple_loss=0.06631, pruned_loss=0.009624, audio_tagging_loss=0.01045, over 16152.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.0929, pruned_loss=0.01475, audio_tagging_loss=0.009305, over 3052353.54 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:11:13,855 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306750
2023-11-22 18:11:50,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2045160.0, ans=0.0
2023-11-22 18:12:04,237 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.26 vs. limit=15.0
2023-11-22 18:12:13,091 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6200, loss[loss=0.05484, simple_loss=0.07587, pruned_loss=0.008713, audio_tagging_loss=0.008188, over 14124.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.0931, pruned_loss=0.01474, audio_tagging_loss=0.009366, over 3051507.68 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:12:15,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2045293.3333333333, ans=0.2
2023-11-22 18:12:18,695 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306800
2023-11-22 18:12:33,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2045360.0, ans=0.0
2023-11-22 18:12:53,571 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.167e+01 8.049e+01 8.588e+01 9.515e+01 1.238e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 18:12:58,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2045493.3333333333, ans=0.125
2023-11-22 18:13:17,029 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6250, loss[loss=0.06315, simple_loss=0.0762, pruned_loss=0.01154, audio_tagging_loss=0.01352, over 13993.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09248, pruned_loss=0.0146, audio_tagging_loss=0.009382, over 3048609.13 frames. ], batch size: 53, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:13:17,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.80 vs. limit=15.0
2023-11-22 18:13:22,082 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306850
2023-11-22 18:13:25,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2045626.6666666667, ans=0.125
2023-11-22 18:13:51,559 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:14:04,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2045826.6666666667, ans=0.09899494936611666
2023-11-22 18:14:07,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2045893.3333333333, ans=0.0
2023-11-22 18:14:16,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2045893.3333333333, ans=0.1
2023-11-22 18:14:19,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2045893.3333333333, ans=0.0
2023-11-22 18:14:21,611 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6300, loss[loss=0.06885, simple_loss=0.08665, pruned_loss=0.01618, audio_tagging_loss=0.009342, over 15807.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09248, pruned_loss=0.01457, audio_tagging_loss=0.009405, over 3048446.77 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:14:26,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306900
2023-11-22 18:14:31,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.80 vs. limit=6.0
2023-11-22 18:14:31,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2045960.0, ans=0.1
2023-11-22 18:14:42,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2046026.6666666667, ans=0.0
2023-11-22 18:14:43,577 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.89 vs. limit=22.5
2023-11-22 18:14:50,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2046093.3333333333, ans=0.0
2023-11-22 18:14:55,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.97 vs. limit=15.0
2023-11-22 18:14:58,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2046160.0, ans=0.1
2023-11-22 18:15:02,393 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.359e+01 8.414e+01 9.060e+01 9.921e+01 1.444e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-22 18:15:03,121 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.12 vs. limit=15.0
2023-11-22 18:15:05,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2046160.0, ans=0.0
2023-11-22 18:15:11,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2046226.6666666667, ans=0.05
2023-11-22 18:15:17,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-22 18:15:26,356 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6350, loss[loss=0.07272, simple_loss=0.09808, pruned_loss=0.01373, audio_tagging_loss=0.009956, over 15032.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09322, pruned_loss=0.01464, audio_tagging_loss=0.009446, over 3050391.68 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:15:26,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2046293.3333333333, ans=0.125
2023-11-22 18:15:31,305 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 306950
2023-11-22 18:15:38,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2046360.0, ans=0.0
2023-11-22 18:15:47,731 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.18 vs. limit=6.0
2023-11-22 18:15:48,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2046360.0, ans=0.2
2023-11-22 18:16:06,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2046493.3333333333, ans=0.5
2023-11-22 18:16:17,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.55 vs. limit=12.0
2023-11-22 18:16:29,688 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6400, loss[loss=0.05986, simple_loss=0.08115, pruned_loss=0.009218, audio_tagging_loss=0.01006, over 15882.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09384, pruned_loss=0.01478, audio_tagging_loss=0.009457, over 3051283.20 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:16:31,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2046626.6666666667, ans=0.0
2023-11-22 18:16:34,572 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307000
2023-11-22 18:16:35,097 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.24 vs. limit=6.0
2023-11-22 18:16:43,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2046693.3333333333, ans=0.125
2023-11-22 18:16:55,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.32 vs. limit=10.0
2023-11-22 18:17:06,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.83 vs. limit=15.0
2023-11-22 18:17:10,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2046826.6666666667, ans=0.0
2023-11-22 18:17:10,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2046826.6666666667, ans=0.125
2023-11-22 18:17:10,984 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.820e+01 8.418e+01 9.159e+01 1.026e+02 1.241e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-22 18:17:32,163 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:17:33,077 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6450, loss[loss=0.07567, simple_loss=0.09702, pruned_loss=0.01552, audio_tagging_loss=0.01164, over 14837.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09365, pruned_loss=0.01471, audio_tagging_loss=0.009537, over 3047718.62 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:17:36,513 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.09 vs. limit=15.0
2023-11-22 18:17:38,680 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307050
2023-11-22 18:17:47,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0
2023-11-22 18:18:03,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2047093.3333333333, ans=0.125
2023-11-22 18:18:36,796 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6500, loss[loss=0.07805, simple_loss=0.09649, pruned_loss=0.01831, audio_tagging_loss=0.01149, over 14399.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.093, pruned_loss=0.01461, audio_tagging_loss=0.009547, over 3043931.72 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:18:42,960 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307100
2023-11-22 18:18:53,363 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.62 vs. limit=6.0
2023-11-22 18:19:10,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2047426.6666666667, ans=0.1
2023-11-22 18:19:12,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2047426.6666666667, ans=0.0
2023-11-22 18:19:16,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2047493.3333333333, ans=0.125
2023-11-22 18:19:19,258 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.186e+01 8.231e+01 8.746e+01 9.661e+01 1.657e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 18:19:24,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2047493.3333333333, ans=0.125
2023-11-22 18:19:37,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2047560.0, ans=15.0
2023-11-22 18:19:39,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2047560.0, ans=0.2
2023-11-22 18:19:41,391 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6550, loss[loss=0.06327, simple_loss=0.0841, pruned_loss=0.01361, audio_tagging_loss=0.007605, over 14997.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09346, pruned_loss=0.01473, audio_tagging_loss=0.009415, over 3040201.01 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:19:46,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307150
2023-11-22 18:19:55,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2047693.3333333333, ans=0.125
2023-11-22 18:20:22,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2047826.6666666667, ans=0.125
2023-11-22 18:20:25,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2047826.6666666667, ans=0.125
2023-11-22 18:20:44,972 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6600, loss[loss=0.07017, simple_loss=0.09846, pruned_loss=0.01298, audio_tagging_loss=0.007959, over 15580.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.0923, pruned_loss=0.01453, audio_tagging_loss=0.009342, over 3040170.63 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:20:50,121 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307200
2023-11-22 18:20:59,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2048026.6666666667, ans=0.1
2023-11-22 18:21:18,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2048093.3333333333, ans=0.05
2023-11-22 18:21:21,284 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.61 vs. limit=15.0
2023-11-22 18:21:24,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2048160.0, ans=0.125
2023-11-22 18:21:27,582 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.316e+01 8.739e+01 9.394e+01 1.270e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 18:21:49,013 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6650, loss[loss=0.04777, simple_loss=0.05894, pruned_loss=0.007455, audio_tagging_loss=0.01084, over 14262.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09216, pruned_loss=0.01444, audio_tagging_loss=0.009268, over 3045111.47 frames. ], batch size: 53, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:21:54,575 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307250
2023-11-22 18:21:58,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.74 vs. limit=12.0
2023-11-22 18:21:59,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2048293.3333333333, ans=0.125
2023-11-22 18:22:09,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2048360.0, ans=0.0
2023-11-22 18:22:12,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2048360.0, ans=0.1
2023-11-22 18:22:23,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2048426.6666666667, ans=0.0
2023-11-22 18:22:26,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2048493.3333333333, ans=0.125
2023-11-22 18:22:53,624 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6700, loss[loss=0.06676, simple_loss=0.08217, pruned_loss=0.01283, audio_tagging_loss=0.01284, over 14181.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09228, pruned_loss=0.01438, audio_tagging_loss=0.009196, over 3038676.99 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:22:53,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2048626.6666666667, ans=0.2
2023-11-22 18:22:58,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307300
2023-11-22 18:23:18,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2048760.0, ans=0.0
2023-11-22 18:23:21,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.16 vs. limit=22.5
2023-11-22 18:23:25,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2048760.0, ans=0.125
2023-11-22 18:23:35,471 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.221e+01 9.103e+01 9.902e+01 1.404e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-22 18:23:36,063 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.99 vs. limit=22.5
2023-11-22 18:23:49,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2048893.3333333333, ans=0.125
2023-11-22 18:23:51,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.82 vs. limit=15.0
2023-11-22 18:23:54,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2048893.3333333333, ans=0.0
2023-11-22 18:23:56,350 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6750, loss[loss=0.06523, simple_loss=0.08781, pruned_loss=0.01173, audio_tagging_loss=0.009596, over 16509.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09128, pruned_loss=0.01418, audio_tagging_loss=0.009292, over 3035089.27 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:24:01,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307350
2023-11-22 18:24:10,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2049026.6666666667, ans=0.0
2023-11-22 18:24:11,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2049026.6666666667, ans=0.125
2023-11-22 18:24:19,325 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:24:26,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2049093.3333333333, ans=0.2
2023-11-22 18:24:31,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2049093.3333333333, ans=0.125
2023-11-22 18:24:49,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2049226.6666666667, ans=0.035
2023-11-22 18:24:59,206 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6800, loss[loss=0.06733, simple_loss=0.09382, pruned_loss=0.01262, audio_tagging_loss=0.007795, over 15395.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09135, pruned_loss=0.01418, audio_tagging_loss=0.009245, over 3036124.42 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:25:04,155 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307400
2023-11-22 18:25:40,524 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.331e+01 8.948e+01 9.524e+01 1.247e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 18:25:50,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2049560.0, ans=0.2
2023-11-22 18:26:03,615 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6850, loss[loss=0.08297, simple_loss=0.1137, pruned_loss=0.0189, audio_tagging_loss=0.007246, over 15068.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09221, pruned_loss=0.0143, audio_tagging_loss=0.00916, over 3045023.03 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:26:08,457 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307450
2023-11-22 18:26:20,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2049693.3333333333, ans=0.04949747468305833
2023-11-22 18:26:53,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2049893.3333333333, ans=0.0
2023-11-22 18:27:06,591 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6900, loss[loss=0.0547, simple_loss=0.07102, pruned_loss=0.01218, audio_tagging_loss=0.007005, over 13573.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09245, pruned_loss=0.0144, audio_tagging_loss=0.009145, over 3042230.68 frames. ], batch size: 52, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:27:11,737 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307500
2023-11-22 18:27:33,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2050093.3333333333, ans=0.1
2023-11-22 18:27:40,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2050093.3333333333, ans=0.2
2023-11-22 18:27:49,682 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.175e+01 8.211e+01 8.819e+01 9.704e+01 1.350e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 18:27:56,104 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:28:11,191 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 6950, loss[loss=0.0856, simple_loss=0.1162, pruned_loss=0.02092, audio_tagging_loss=0.006559, over 14931.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.0931, pruned_loss=0.0145, audio_tagging_loss=0.009143, over 3040491.51 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:28:16,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307550
2023-11-22 18:28:42,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2050426.6666666667, ans=0.1
2023-11-22 18:28:45,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2050426.6666666667, ans=0.0
2023-11-22 18:28:47,984 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-22 18:29:04,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2050560.0, ans=0.0
2023-11-22 18:29:17,592 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7000, loss[loss=0.07607, simple_loss=0.1013, pruned_loss=0.01818, audio_tagging_loss=0.007223, over 15123.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09348, pruned_loss=0.01451, audio_tagging_loss=0.009111, over 3047322.32 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:29:18,179 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.50 vs. limit=12.0
2023-11-22 18:29:23,221 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307600
2023-11-22 18:29:53,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=12.0
2023-11-22 18:29:58,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.27 vs. limit=15.0
2023-11-22 18:29:59,742 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 8.302e+01 8.887e+01 9.651e+01 1.249e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 18:30:04,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2050826.6666666667, ans=0.125
2023-11-22 18:30:09,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2050893.3333333333, ans=0.125
2023-11-22 18:30:22,296 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7050, loss[loss=0.07713, simple_loss=0.1042, pruned_loss=0.01791, audio_tagging_loss=0.007108, over 15079.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09291, pruned_loss=0.01456, audio_tagging_loss=0.009227, over 3047671.31 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:30:27,201 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307650
2023-11-22 18:30:28,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2050960.0, ans=0.125
2023-11-22 18:30:41,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2051026.6666666667, ans=0.025
2023-11-22 18:30:55,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2051093.3333333333, ans=0.125
2023-11-22 18:31:07,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2051160.0, ans=0.125
2023-11-22 18:31:24,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2051226.6666666667, ans=0.125
2023-11-22 18:31:27,264 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7100, loss[loss=0.06532, simple_loss=0.09141, pruned_loss=0.009879, audio_tagging_loss=0.009735, over 16106.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09376, pruned_loss=0.01479, audio_tagging_loss=0.009316, over 3053729.80 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:31:32,407 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307700
2023-11-22 18:31:39,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2051360.0, ans=0.125
2023-11-22 18:31:40,471 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.25 vs. limit=10.0
2023-11-22 18:31:51,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2051360.0, ans=0.0
2023-11-22 18:32:11,641 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.336e+01 9.020e+01 9.845e+01 2.040e+02, threshold=1.804e+02, percent-clipped=1.0
2023-11-22 18:32:19,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2051560.0, ans=0.04949747468305833
2023-11-22 18:32:32,724 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7150, loss[loss=0.07477, simple_loss=0.09802, pruned_loss=0.01524, audio_tagging_loss=0.01051, over 15174.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09367, pruned_loss=0.01483, audio_tagging_loss=0.009319, over 3050056.81 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:32:33,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.25 vs. limit=15.0
2023-11-22 18:32:37,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2051626.6666666667, ans=0.125
2023-11-22 18:32:38,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307750
2023-11-22 18:32:44,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2051693.3333333333, ans=0.2
2023-11-22 18:32:45,000 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.67 vs. limit=6.0
2023-11-22 18:33:01,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2051760.0, ans=10.0
2023-11-22 18:33:09,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.47 vs. limit=22.5
2023-11-22 18:33:21,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2051826.6666666667, ans=0.035
2023-11-22 18:33:27,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2051893.3333333333, ans=0.1
2023-11-22 18:33:29,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2051893.3333333333, ans=0.05
2023-11-22 18:33:33,544 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:33:36,859 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7200, loss[loss=0.07168, simple_loss=0.0846, pruned_loss=0.01866, audio_tagging_loss=0.01072, over 15499.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09345, pruned_loss=0.01486, audio_tagging_loss=0.009483, over 3051768.71 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:33:40,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2051960.0, ans=0.125
2023-11-22 18:33:42,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307800
2023-11-22 18:33:48,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.59 vs. limit=15.0
2023-11-22 18:33:56,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2052026.6666666667, ans=0.125
2023-11-22 18:34:20,786 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.282e+01 8.868e+01 9.678e+01 1.341e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 18:34:40,979 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7250, loss[loss=0.06564, simple_loss=0.09035, pruned_loss=0.01231, audio_tagging_loss=0.008153, over 15538.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09316, pruned_loss=0.01478, audio_tagging_loss=0.009647, over 3046983.18 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:34:46,170 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307850
2023-11-22 18:34:52,816 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.28 vs. limit=6.0
2023-11-22 18:35:03,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2052360.0, ans=0.125
2023-11-22 18:35:30,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2052493.3333333333, ans=10.0
2023-11-22 18:35:45,796 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7300, loss[loss=0.07338, simple_loss=0.1077, pruned_loss=0.01101, audio_tagging_loss=0.008524, over 15727.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09292, pruned_loss=0.01449, audio_tagging_loss=0.009568, over 3045101.47 frames. ], batch size: 62, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:35:51,260 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307900
2023-11-22 18:36:07,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2052693.3333333333, ans=0.125
2023-11-22 18:36:19,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0
2023-11-22 18:36:28,169 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.001e+01 8.613e+01 9.353e+01 1.295e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-22 18:36:49,236 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7350, loss[loss=0.07275, simple_loss=0.09278, pruned_loss=0.01735, audio_tagging_loss=0.009004, over 15071.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09407, pruned_loss=0.01465, audio_tagging_loss=0.009369, over 3043067.05 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:36:49,954 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0
2023-11-22 18:36:54,149 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 307950
2023-11-22 18:37:00,976 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:37:04,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2053026.6666666667, ans=0.125
2023-11-22 18:37:35,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2053160.0, ans=0.125
2023-11-22 18:37:53,617 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7400, loss[loss=0.07802, simple_loss=0.1027, pruned_loss=0.01751, audio_tagging_loss=0.009178, over 15407.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09357, pruned_loss=0.01462, audio_tagging_loss=0.009292, over 3046646.01 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:37:53,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2053293.3333333333, ans=0.125
2023-11-22 18:37:58,539 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308000
2023-11-22 18:38:06,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2053293.3333333333, ans=0.0
2023-11-22 18:38:29,397 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.89 vs. limit=8.0
2023-11-22 18:38:32,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2053426.6666666667, ans=0.1
2023-11-22 18:38:34,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2053493.3333333333, ans=0.125
2023-11-22 18:38:40,999 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.189e+01 8.818e+01 9.680e+01 1.265e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 18:38:57,162 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.99 vs. limit=12.0
2023-11-22 18:39:02,136 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7450, loss[loss=0.08429, simple_loss=0.1156, pruned_loss=0.01703, audio_tagging_loss=0.009456, over 15082.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09333, pruned_loss=0.0146, audio_tagging_loss=0.00919, over 3045413.68 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:39:05,818 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.61 vs. limit=12.0
2023-11-22 18:39:08,388 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308050
2023-11-22 18:39:13,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2053626.6666666667, ans=0.125
2023-11-22 18:39:22,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2053693.3333333333, ans=0.0
2023-11-22 18:39:52,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2053893.3333333333, ans=0.0
2023-11-22 18:40:02,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2053893.3333333333, ans=0.125
2023-11-22 18:40:07,064 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7500, loss[loss=0.06298, simple_loss=0.08097, pruned_loss=0.01299, audio_tagging_loss=0.009502, over 15585.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09278, pruned_loss=0.01446, audio_tagging_loss=0.009164, over 3039212.85 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:40:12,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308100
2023-11-22 18:40:31,348 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:40:52,204 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.656e+01 8.267e+01 8.775e+01 9.244e+01 1.175e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 18:41:10,629 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7550, loss[loss=0.09381, simple_loss=0.1319, pruned_loss=0.01979, audio_tagging_loss=0.00808, over 15605.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09273, pruned_loss=0.01451, audio_tagging_loss=0.009281, over 3043403.87 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:41:13,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2054293.3333333333, ans=0.125
2023-11-22 18:41:16,127 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308150
2023-11-22 18:41:17,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2054293.3333333333, ans=0.125
2023-11-22 18:41:58,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2054493.3333333333, ans=0.5
2023-11-22 18:42:00,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2054560.0, ans=0.04949747468305833
2023-11-22 18:42:13,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2054626.6666666667, ans=0.125
2023-11-22 18:42:14,489 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7600, loss[loss=0.07749, simple_loss=0.1014, pruned_loss=0.01857, audio_tagging_loss=0.008206, over 15183.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09303, pruned_loss=0.01452, audio_tagging_loss=0.009255, over 3043515.25 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:42:14,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2054626.6666666667, ans=0.125
2023-11-22 18:42:20,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308200
2023-11-22 18:42:40,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=15.0
2023-11-22 18:42:48,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.86 vs. limit=22.5
2023-11-22 18:42:49,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2054760.0, ans=0.125
2023-11-22 18:42:59,729 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 7.973e+01 8.478e+01 9.439e+01 1.214e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-22 18:43:20,453 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7650, loss[loss=0.07291, simple_loss=0.09495, pruned_loss=0.01443, audio_tagging_loss=0.01101, over 15512.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09301, pruned_loss=0.0146, audio_tagging_loss=0.009173, over 3046265.83 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:43:25,674 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308250
2023-11-22 18:43:48,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2055093.3333333333, ans=0.125
2023-11-22 18:43:51,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2055093.3333333333, ans=0.1
2023-11-22 18:43:54,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2055093.3333333333, ans=0.04949747468305833
2023-11-22 18:43:58,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff3.min_abs, batch_count=2055160.0, ans=0.2
2023-11-22 18:43:59,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2055160.0, ans=0.125
2023-11-22 18:44:02,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=2055160.0, ans=0.1
2023-11-22 18:44:15,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.47 vs. limit=22.5
2023-11-22 18:44:24,546 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7700, loss[loss=0.06521, simple_loss=0.08085, pruned_loss=0.01542, audio_tagging_loss=0.009361, over 15955.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09166, pruned_loss=0.01431, audio_tagging_loss=0.009186, over 3041814.34 frames. ], batch size: 64, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:44:29,501 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308300
2023-11-22 18:44:32,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2055293.3333333333, ans=0.07
2023-11-22 18:44:44,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.33 vs. limit=15.0
2023-11-22 18:45:10,226 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.353e+01 9.009e+01 9.552e+01 1.417e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-22 18:45:29,286 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7750, loss[loss=0.08893, simple_loss=0.1159, pruned_loss=0.02165, audio_tagging_loss=0.009304, over 15699.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09239, pruned_loss=0.01456, audio_tagging_loss=0.009165, over 3047117.19 frames. ], batch size: 60, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:45:34,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308350
2023-11-22 18:45:34,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2055626.6666666667, ans=0.0
2023-11-22 18:45:50,122 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.69 vs. limit=15.0
2023-11-22 18:45:59,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2055760.0, ans=0.125
2023-11-22 18:46:00,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2055760.0, ans=0.1
2023-11-22 18:46:17,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2055826.6666666667, ans=0.125
2023-11-22 18:46:23,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2055893.3333333333, ans=0.125
2023-11-22 18:46:34,443 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7800, loss[loss=0.06329, simple_loss=0.08935, pruned_loss=0.01204, audio_tagging_loss=0.006572, over 14706.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09327, pruned_loss=0.01463, audio_tagging_loss=0.00913, over 3040209.52 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:46:39,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308400
2023-11-22 18:46:49,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2056026.6666666667, ans=0.125
2023-11-22 18:47:05,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2056093.3333333333, ans=0.2
2023-11-22 18:47:06,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2056093.3333333333, ans=0.0
2023-11-22 18:47:20,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2056160.0, ans=0.0
2023-11-22 18:47:21,059 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.003e+01 8.337e+01 8.829e+01 9.514e+01 1.389e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 18:47:38,094 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7850, loss[loss=0.05943, simple_loss=0.07548, pruned_loss=0.01194, audio_tagging_loss=0.009751, over 14383.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09285, pruned_loss=0.0145, audio_tagging_loss=0.00924, over 3044672.52 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:47:39,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.77 vs. limit=15.0
2023-11-22 18:47:43,155 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308450
2023-11-22 18:47:47,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2056293.3333333333, ans=0.0
2023-11-22 18:47:52,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2056360.0, ans=0.2
2023-11-22 18:47:59,933 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.60 vs. limit=10.0
2023-11-22 18:48:01,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2056360.0, ans=0.125
2023-11-22 18:48:15,198 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0
2023-11-22 18:48:21,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.40 vs. limit=15.0
2023-11-22 18:48:25,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2056493.3333333333, ans=0.0
2023-11-22 18:48:37,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2056560.0, ans=0.1
2023-11-22 18:48:41,449 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7900, loss[loss=0.06331, simple_loss=0.08298, pruned_loss=0.01294, audio_tagging_loss=0.008882, over 15755.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09319, pruned_loss=0.0146, audio_tagging_loss=0.009251, over 3045846.71 frames. ], batch size: 61, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:48:46,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308500
2023-11-22 18:49:27,224 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.747e+01 8.185e+01 8.852e+01 9.450e+01 1.416e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 18:49:46,611 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 7950, loss[loss=0.08776, simple_loss=0.1213, pruned_loss=0.0194, audio_tagging_loss=0.007694, over 15187.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09357, pruned_loss=0.01466, audio_tagging_loss=0.009358, over 3047871.77 frames. ], batch size: 53, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:49:50,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2056960.0, ans=0.2
2023-11-22 18:49:51,651 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308550
2023-11-22 18:50:00,017 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:50:05,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2057026.6666666667, ans=0.125
2023-11-22 18:50:28,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.71 vs. limit=12.0
2023-11-22 18:50:49,760 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8000, loss[loss=0.06369, simple_loss=0.08995, pruned_loss=0.01112, audio_tagging_loss=0.007601, over 14357.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09301, pruned_loss=0.01464, audio_tagging_loss=0.009481, over 3048426.27 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:50:54,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308600
2023-11-22 18:50:56,908 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.77 vs. limit=15.0
2023-11-22 18:51:09,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2057360.0, ans=0.0
2023-11-22 18:51:17,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.whiten.whitening_limit, batch_count=2057426.6666666667, ans=15.0
2023-11-22 18:51:19,763 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:51:36,564 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 8.209e+01 8.705e+01 9.523e+01 1.226e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 18:51:53,700 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8050, loss[loss=0.05631, simple_loss=0.0728, pruned_loss=0.009534, audio_tagging_loss=0.01038, over 14770.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.0929, pruned_loss=0.01466, audio_tagging_loss=0.009494, over 3048891.27 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:51:58,663 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308650
2023-11-22 18:51:59,323 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.65 vs. limit=22.5
2023-11-22 18:52:04,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2057626.6666666667, ans=0.0
2023-11-22 18:52:29,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.86 vs. limit=15.0
2023-11-22 18:52:59,052 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8100, loss[loss=0.0614, simple_loss=0.07674, pruned_loss=0.01356, audio_tagging_loss=0.009472, over 16989.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09349, pruned_loss=0.01486, audio_tagging_loss=0.00948, over 3049170.09 frames. ], batch size: 65, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:53:04,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308700
2023-11-22 18:53:18,030 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:53:22,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2058093.3333333333, ans=0.0
2023-11-22 18:53:35,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2058160.0, ans=0.125
2023-11-22 18:53:45,844 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.339e+01 8.206e+01 8.841e+01 9.625e+01 1.237e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 18:53:50,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2058226.6666666667, ans=0.125
2023-11-22 18:54:03,364 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8150, loss[loss=0.08437, simple_loss=0.1197, pruned_loss=0.01693, audio_tagging_loss=0.007594, over 15779.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09459, pruned_loss=0.01505, audio_tagging_loss=0.009332, over 3053766.45 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:54:08,436 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308750
2023-11-22 18:54:09,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2058293.3333333333, ans=0.2
2023-11-22 18:54:28,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2058426.6666666667, ans=0.0
2023-11-22 18:55:07,309 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8200, loss[loss=0.0713, simple_loss=0.09771, pruned_loss=0.01282, audio_tagging_loss=0.009622, over 14051.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09449, pruned_loss=0.01493, audio_tagging_loss=0.009236, over 3048826.78 frames. ], batch size: 53, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:55:07,324 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:55:12,424 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308800
2023-11-22 18:55:20,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.19 vs. limit=22.5
2023-11-22 18:55:32,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=24.42 vs. limit=22.5
2023-11-22 18:55:55,511 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.302e+01 8.890e+01 9.670e+01 1.288e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 18:56:00,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2058893.3333333333, ans=0.125
2023-11-22 18:56:04,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2058893.3333333333, ans=0.0
2023-11-22 18:56:11,847 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8250, loss[loss=0.06982, simple_loss=0.0862, pruned_loss=0.01637, audio_tagging_loss=0.01035, over 15306.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.0942, pruned_loss=0.01472, audio_tagging_loss=0.009163, over 3044803.99 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:56:13,673 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.32 vs. limit=12.0
2023-11-22 18:56:17,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308850
2023-11-22 18:56:19,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2058960.0, ans=0.0
2023-11-22 18:56:37,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2059093.3333333333, ans=0.0
2023-11-22 18:56:42,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2059093.3333333333, ans=0.1
2023-11-22 18:56:42,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2059093.3333333333, ans=0.125
2023-11-22 18:56:44,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2059093.3333333333, ans=0.125
2023-11-22 18:56:48,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2059160.0, ans=0.125
2023-11-22 18:56:52,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2059160.0, ans=0.0
2023-11-22 18:56:52,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2059160.0, ans=0.125
2023-11-22 18:56:53,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2059160.0, ans=0.125
2023-11-22 18:57:16,335 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8300, loss[loss=0.07618, simple_loss=0.1032, pruned_loss=0.01721, audio_tagging_loss=0.007387, over 14792.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09479, pruned_loss=0.01476, audio_tagging_loss=0.009079, over 3042789.91 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:57:17,827 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:57:21,466 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308900
2023-11-22 18:57:22,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2059293.3333333333, ans=0.125
2023-11-22 18:57:24,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2059293.3333333333, ans=0.125
2023-11-22 18:58:00,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2059493.3333333333, ans=0.07
2023-11-22 18:58:03,823 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.260e+01 8.357e+01 8.920e+01 9.669e+01 1.328e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 18:58:09,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2059560.0, ans=0.1
2023-11-22 18:58:10,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2059560.0, ans=0.125
2023-11-22 18:58:20,134 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8350, loss[loss=0.07188, simple_loss=0.08873, pruned_loss=0.01679, audio_tagging_loss=0.01072, over 14551.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09425, pruned_loss=0.01476, audio_tagging_loss=0.00912, over 3043221.94 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:58:25,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 308950
2023-11-22 18:58:32,488 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:58:41,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2059693.3333333333, ans=0.125
2023-11-22 18:59:12,236 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.05 vs. limit=6.0
2023-11-22 18:59:16,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2059893.3333333333, ans=0.07
2023-11-22 18:59:22,774 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8400, loss[loss=0.07228, simple_loss=0.09123, pruned_loss=0.01644, audio_tagging_loss=0.01023, over 16033.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.094, pruned_loss=0.01458, audio_tagging_loss=0.009102, over 3041931.33 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:59:27,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2059960.0, ans=0.0
2023-11-22 18:59:28,987 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309000
2023-11-22 18:59:35,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2060026.6666666667, ans=0.0
2023-11-22 18:59:47,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2060026.6666666667, ans=0.0
2023-11-22 19:00:10,223 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.438e+01 8.273e+01 8.789e+01 9.898e+01 1.441e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 19:00:23,668 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.98 vs. limit=6.0
2023-11-22 19:00:25,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2060226.6666666667, ans=0.125
2023-11-22 19:00:28,068 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8450, loss[loss=0.08906, simple_loss=0.119, pruned_loss=0.02022, audio_tagging_loss=0.009333, over 15111.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09348, pruned_loss=0.01458, audio_tagging_loss=0.009172, over 3032980.48 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:00:33,060 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309050
2023-11-22 19:00:43,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2060360.0, ans=0.2
2023-11-22 19:00:47,620 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0
2023-11-22 19:00:48,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2060360.0, ans=0.0
2023-11-22 19:00:49,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2060360.0, ans=0.0
2023-11-22 19:00:57,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2060426.6666666667, ans=0.125
2023-11-22 19:01:11,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2060493.3333333333, ans=0.05
2023-11-22 19:01:20,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2060560.0, ans=0.05
2023-11-22 19:01:31,972 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8500, loss[loss=0.07128, simple_loss=0.09166, pruned_loss=0.01662, audio_tagging_loss=0.008835, over 14446.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09273, pruned_loss=0.01438, audio_tagging_loss=0.009202, over 3040661.72 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:01:32,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.16 vs. limit=15.0
2023-11-22 19:01:36,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309100
2023-11-22 19:01:59,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2060760.0, ans=0.0
2023-11-22 19:01:59,766 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0
2023-11-22 19:02:14,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2060826.6666666667, ans=0.2
2023-11-22 19:02:19,382 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.798e+01 8.300e+01 8.864e+01 9.381e+01 1.141e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 19:02:27,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2060893.3333333333, ans=0.1
2023-11-22 19:02:35,141 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8550, loss[loss=0.06592, simple_loss=0.08803, pruned_loss=0.0134, audio_tagging_loss=0.008514, over 15253.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.0931, pruned_loss=0.01457, audio_tagging_loss=0.009255, over 3048863.28 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:02:36,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2060960.0, ans=0.0
2023-11-22 19:02:41,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309150
2023-11-22 19:02:55,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2061026.6666666667, ans=0.125
2023-11-22 19:02:57,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2061026.6666666667, ans=0.125
2023-11-22 19:02:59,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2061026.6666666667, ans=0.0
2023-11-22 19:03:10,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2061093.3333333333, ans=0.0
2023-11-22 19:03:21,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2061160.0, ans=0.0
2023-11-22 19:03:25,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2061226.6666666667, ans=0.0
2023-11-22 19:03:30,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2061226.6666666667, ans=0.125
2023-11-22 19:03:34,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2061226.6666666667, ans=0.125
2023-11-22 19:03:40,641 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8600, loss[loss=0.0621, simple_loss=0.07976, pruned_loss=0.01138, audio_tagging_loss=0.01085, over 15056.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09367, pruned_loss=0.01479, audio_tagging_loss=0.009289, over 3042481.74 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:03:40,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2061293.3333333333, ans=0.125
2023-11-22 19:03:45,525 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309200
2023-11-22 19:03:45,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2061293.3333333333, ans=0.0
2023-11-22 19:03:45,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2061293.3333333333, ans=0.1
2023-11-22 19:04:05,246 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.13 vs. limit=15.0
2023-11-22 19:04:20,885 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.11 vs. limit=15.0
2023-11-22 19:04:27,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2061493.3333333333, ans=0.125
2023-11-22 19:04:27,932 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.278e+01 9.087e+01 9.783e+01 1.157e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-22 19:04:43,833 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8650, loss[loss=0.05521, simple_loss=0.06577, pruned_loss=0.01103, audio_tagging_loss=0.01129, over 14906.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.0937, pruned_loss=0.01486, audio_tagging_loss=0.009275, over 3040280.22 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:04:49,426 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309250
2023-11-22 19:04:53,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2061626.6666666667, ans=0.125
2023-11-22 19:05:13,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.94 vs. limit=15.0
2023-11-22 19:05:19,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2061760.0, ans=0.125
2023-11-22 19:05:21,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2061826.6666666667, ans=0.09899494936611666
2023-11-22 19:05:41,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2061893.3333333333, ans=0.125
2023-11-22 19:05:47,842 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8700, loss[loss=0.0636, simple_loss=0.08437, pruned_loss=0.01226, audio_tagging_loss=0.00916, over 15197.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09492, pruned_loss=0.01504, audio_tagging_loss=0.009277, over 3045372.68 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:05:48,686 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.68 vs. limit=15.0
2023-11-22 19:05:53,631 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309300
2023-11-22 19:06:01,754 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.15 vs. limit=15.0
2023-11-22 19:06:19,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2062093.3333333333, ans=0.2
2023-11-22 19:06:19,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.17 vs. limit=22.5
2023-11-22 19:06:21,115 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.01 vs. limit=12.0
2023-11-22 19:06:36,749 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.774e+01 8.375e+01 8.886e+01 9.746e+01 1.312e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 19:06:36,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2062160.0, ans=0.125
2023-11-22 19:06:43,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2062226.6666666667, ans=0.1
2023-11-22 19:06:52,433 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8750, loss[loss=0.06491, simple_loss=0.07791, pruned_loss=0.01352, audio_tagging_loss=0.01243, over 15274.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09465, pruned_loss=0.01503, audio_tagging_loss=0.009346, over 3040293.18 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:06:57,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309350
2023-11-22 19:07:55,112 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8800, loss[loss=0.08761, simple_loss=0.1224, pruned_loss=0.01787, audio_tagging_loss=0.008563, over 15694.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.0947, pruned_loss=0.01501, audio_tagging_loss=0.009463, over 3049882.46 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:08:00,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309400
2023-11-22 19:08:12,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2062693.3333333333, ans=0.2
2023-11-22 19:08:21,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.44 vs. limit=12.0
2023-11-22 19:08:31,067 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.21 vs. limit=15.0
2023-11-22 19:08:42,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2062826.6666666667, ans=0.125
2023-11-22 19:08:43,646 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.328e+01 8.323e+01 8.895e+01 9.421e+01 1.402e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 19:08:50,453 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.05 vs. limit=15.0
2023-11-22 19:08:58,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.37 vs. limit=15.0
2023-11-22 19:08:58,979 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8850, loss[loss=0.0622, simple_loss=0.08116, pruned_loss=0.01172, audio_tagging_loss=0.009897, over 14761.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09443, pruned_loss=0.01488, audio_tagging_loss=0.009485, over 3049831.75 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:09:03,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309450
2023-11-22 19:09:05,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2062960.0, ans=0.1
2023-11-22 19:09:10,416 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:09:13,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2063026.6666666667, ans=0.125
2023-11-22 19:09:25,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2063093.3333333333, ans=0.1
2023-11-22 19:09:56,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2063226.6666666667, ans=0.125
2023-11-22 19:10:02,969 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8900, loss[loss=0.07753, simple_loss=0.09862, pruned_loss=0.01832, audio_tagging_loss=0.009899, over 14055.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.0947, pruned_loss=0.01502, audio_tagging_loss=0.009263, over 3047793.75 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:10:07,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309500
2023-11-22 19:10:23,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2063360.0, ans=0.125
2023-11-22 19:10:25,658 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.66 vs. limit=10.0
2023-11-22 19:10:39,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2063493.3333333333, ans=0.07
2023-11-22 19:10:50,963 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.152e+01 8.729e+01 9.572e+01 1.153e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 19:11:05,622 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 8950, loss[loss=0.07501, simple_loss=0.1005, pruned_loss=0.01624, audio_tagging_loss=0.00851, over 14697.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.0951, pruned_loss=0.01509, audio_tagging_loss=0.009109, over 3046216.93 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:11:10,649 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309550
2023-11-22 19:11:31,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2063760.0, ans=0.125
2023-11-22 19:11:39,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2063760.0, ans=0.125
2023-11-22 19:11:46,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2063826.6666666667, ans=0.125
2023-11-22 19:11:56,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2063893.3333333333, ans=0.0
2023-11-22 19:11:59,197 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.61 vs. limit=5.0
2023-11-22 19:12:07,965 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9000, loss[loss=0.05737, simple_loss=0.06505, pruned_loss=0.01138, audio_tagging_loss=0.01347, over 15635.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09511, pruned_loss=0.01517, audio_tagging_loss=0.009164, over 3045284.99 frames. ], batch size: 61, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:12:07,966 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 19:12:27,247 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.5226, 6.3509, 6.2941, 6.2763], device='cuda:2')
2023-11-22 19:12:45,260 INFO [train_asr.py:1253] (2/4) Epoch 26, validation: loss=0.0595, simple_loss=0.05137, pruned_loss=0.00505, audio_tagging_loss=0.02877, over 4681554.00 frames. 
2023-11-22 19:12:45,261 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 19:12:50,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309600
2023-11-22 19:12:54,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.40 vs. limit=15.0
2023-11-22 19:12:58,375 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.31 vs. limit=15.0
2023-11-22 19:13:11,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.56 vs. limit=10.0
2023-11-22 19:13:24,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2064160.0, ans=0.125
2023-11-22 19:13:34,773 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.292e+01 8.198e+01 8.954e+01 9.788e+01 1.376e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 19:13:35,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2064226.6666666667, ans=0.0
2023-11-22 19:13:44,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2064226.6666666667, ans=0.125
2023-11-22 19:13:48,380 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9050, loss[loss=0.05656, simple_loss=0.06572, pruned_loss=0.008943, audio_tagging_loss=0.01476, over 14884.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09397, pruned_loss=0.01484, audio_tagging_loss=0.009143, over 3048814.51 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:13:51,503 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.63 vs. limit=15.0
2023-11-22 19:13:53,532 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309650
2023-11-22 19:13:53,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2064293.3333333333, ans=0.05
2023-11-22 19:13:53,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2064293.3333333333, ans=0.09899494936611666
2023-11-22 19:13:54,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064293.3333333333, ans=0.1
2023-11-22 19:13:55,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2064293.3333333333, ans=0.0
2023-11-22 19:13:55,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.70 vs. limit=10.0
2023-11-22 19:14:15,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2064426.6666666667, ans=0.2
2023-11-22 19:14:44,071 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.12 vs. limit=22.5
2023-11-22 19:14:50,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2064626.6666666667, ans=15.0
2023-11-22 19:14:50,835 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9100, loss[loss=0.05932, simple_loss=0.07178, pruned_loss=0.01312, audio_tagging_loss=0.01031, over 14170.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09366, pruned_loss=0.01477, audio_tagging_loss=0.009139, over 3051746.93 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:14:55,772 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309700
2023-11-22 19:14:55,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2064626.6666666667, ans=0.125
2023-11-22 19:14:57,783 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.68 vs. limit=10.0
2023-11-22 19:15:05,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2064693.3333333333, ans=0.125
2023-11-22 19:15:10,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2064693.3333333333, ans=0.0
2023-11-22 19:15:10,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2064693.3333333333, ans=0.0
2023-11-22 19:15:30,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064826.6666666667, ans=0.1
2023-11-22 19:15:39,605 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.178e+01 9.169e+01 9.914e+01 1.442e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-22 19:15:41,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2064893.3333333333, ans=0.125
2023-11-22 19:15:41,573 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.87 vs. limit=15.0
2023-11-22 19:15:54,984 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9150, loss[loss=0.07149, simple_loss=0.08367, pruned_loss=0.01608, audio_tagging_loss=0.01357, over 14679.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09368, pruned_loss=0.01477, audio_tagging_loss=0.009154, over 3047343.20 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:15:59,942 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309750
2023-11-22 19:16:28,589 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.58 vs. limit=10.0
2023-11-22 19:16:35,937 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=12.0
2023-11-22 19:16:36,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2065160.0, ans=0.125
2023-11-22 19:16:57,885 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9200, loss[loss=0.06391, simple_loss=0.0935, pruned_loss=0.009864, audio_tagging_loss=0.007295, over 14941.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09292, pruned_loss=0.01466, audio_tagging_loss=0.009125, over 3047327.72 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:16:59,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2065293.3333333333, ans=0.1
2023-11-22 19:17:02,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309800
2023-11-22 19:17:18,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2065360.0, ans=0.1
2023-11-22 19:17:28,705 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.08 vs. limit=15.0
2023-11-22 19:17:35,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2065493.3333333333, ans=10.0
2023-11-22 19:17:39,898 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:17:42,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2065493.3333333333, ans=0.125
2023-11-22 19:17:48,103 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.424e+01 8.996e+01 9.911e+01 1.165e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 19:17:54,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2065560.0, ans=0.0
2023-11-22 19:17:54,763 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.69 vs. limit=22.5
2023-11-22 19:18:00,304 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9250, loss[loss=0.08326, simple_loss=0.1141, pruned_loss=0.02018, audio_tagging_loss=0.006005, over 14686.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09352, pruned_loss=0.01495, audio_tagging_loss=0.009049, over 3050249.66 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:18:03,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2065626.6666666667, ans=0.2
2023-11-22 19:18:05,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309850
2023-11-22 19:18:10,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2065626.6666666667, ans=0.0
2023-11-22 19:18:21,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2065693.3333333333, ans=0.0
2023-11-22 19:18:38,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.09 vs. limit=15.0
2023-11-22 19:19:00,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2065893.3333333333, ans=0.2
2023-11-22 19:19:04,402 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9300, loss[loss=0.08131, simple_loss=0.1049, pruned_loss=0.02032, audio_tagging_loss=0.008532, over 15444.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09316, pruned_loss=0.01485, audio_tagging_loss=0.009057, over 3054882.40 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:19:09,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309900
2023-11-22 19:19:27,173 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.63 vs. limit=22.5
2023-11-22 19:19:27,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2066026.6666666667, ans=0.1
2023-11-22 19:19:29,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2066093.3333333333, ans=0.125
2023-11-22 19:19:31,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2066093.3333333333, ans=0.0
2023-11-22 19:19:32,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2066093.3333333333, ans=0.0
2023-11-22 19:19:41,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2066160.0, ans=10.0
2023-11-22 19:19:56,379 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 7.933e+01 8.549e+01 9.311e+01 1.177e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 19:20:09,192 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9350, loss[loss=0.05571, simple_loss=0.07106, pruned_loss=0.0111, audio_tagging_loss=0.009081, over 15791.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09355, pruned_loss=0.0149, audio_tagging_loss=0.009098, over 3061464.03 frames. ], batch size: 61, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:20:14,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 309950
2023-11-22 19:20:19,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.95 vs. limit=15.0
2023-11-22 19:20:32,780 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.44 vs. limit=15.0
2023-11-22 19:21:03,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2066560.0, ans=0.125
2023-11-22 19:21:03,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2066560.0, ans=0.125
2023-11-22 19:21:07,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2066560.0, ans=0.125
2023-11-22 19:21:09,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2066560.0, ans=0.0
2023-11-22 19:21:11,935 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9400, loss[loss=0.07641, simple_loss=0.1012, pruned_loss=0.01716, audio_tagging_loss=0.00866, over 14623.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09352, pruned_loss=0.01499, audio_tagging_loss=0.00916, over 3060867.91 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:21:16,952 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310000
2023-11-22 19:21:43,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2066760.0, ans=0.125
2023-11-22 19:21:44,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2066760.0, ans=0.0
2023-11-22 19:21:44,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.26 vs. limit=15.0
2023-11-22 19:22:03,896 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.065e+01 8.287e+01 9.048e+01 9.651e+01 1.161e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 19:22:12,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2066893.3333333333, ans=0.04949747468305833
2023-11-22 19:22:13,665 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:22:16,564 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9450, loss[loss=0.08017, simple_loss=0.1124, pruned_loss=0.01673, audio_tagging_loss=0.007254, over 16159.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09373, pruned_loss=0.0149, audio_tagging_loss=0.009255, over 3057585.29 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:22:17,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff2.min_abs, batch_count=2066960.0, ans=0.1
2023-11-22 19:22:18,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2066960.0, ans=0.125
2023-11-22 19:22:22,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310050
2023-11-22 19:22:35,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2067026.6666666667, ans=0.125
2023-11-22 19:22:47,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2067093.3333333333, ans=0.125
2023-11-22 19:23:08,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2067226.6666666667, ans=0.1
2023-11-22 19:23:16,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2067226.6666666667, ans=0.125
2023-11-22 19:23:18,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.27 vs. limit=15.0
2023-11-22 19:23:20,766 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9500, loss[loss=0.05745, simple_loss=0.07597, pruned_loss=0.0109, audio_tagging_loss=0.008562, over 15264.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09336, pruned_loss=0.01478, audio_tagging_loss=0.009316, over 3056494.91 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 8.0
2023-11-22 19:23:26,362 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310100
2023-11-22 19:23:28,123 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0
2023-11-22 19:23:31,835 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.94 vs. limit=15.0
2023-11-22 19:23:51,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2067426.6666666667, ans=0.0
2023-11-22 19:24:04,962 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.13 vs. limit=10.0
2023-11-22 19:24:07,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2067493.3333333333, ans=0.1
2023-11-22 19:24:08,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2067493.3333333333, ans=0.0
2023-11-22 19:24:12,954 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.401e+01 8.999e+01 9.659e+01 1.129e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 19:24:24,403 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9550, loss[loss=0.04729, simple_loss=0.06314, pruned_loss=0.00416, audio_tagging_loss=0.01156, over 14602.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09312, pruned_loss=0.0147, audio_tagging_loss=0.009387, over 3053826.39 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 8.0
2023-11-22 19:24:27,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2067626.6666666667, ans=0.2
2023-11-22 19:24:29,313 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310150
2023-11-22 19:24:32,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2067626.6666666667, ans=0.04949747468305833
2023-11-22 19:24:34,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2067626.6666666667, ans=0.125
2023-11-22 19:24:36,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2067693.3333333333, ans=0.125
2023-11-22 19:25:01,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.84 vs. limit=6.0
2023-11-22 19:25:08,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2067826.6666666667, ans=0.125
2023-11-22 19:25:14,790 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-22 19:25:20,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2067893.3333333333, ans=0.1
2023-11-22 19:25:27,154 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9600, loss[loss=0.06395, simple_loss=0.08401, pruned_loss=0.012, audio_tagging_loss=0.00994, over 16181.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09326, pruned_loss=0.01462, audio_tagging_loss=0.009409, over 3060733.31 frames. ], batch size: 61, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:25:33,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310200
2023-11-22 19:25:52,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2068093.3333333333, ans=0.05
2023-11-22 19:26:01,271 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.41 vs. limit=15.0
2023-11-22 19:26:20,651 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.239e+01 8.821e+01 9.524e+01 1.134e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 19:26:20,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2068226.6666666667, ans=0.0
2023-11-22 19:26:21,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.81 vs. limit=15.0
2023-11-22 19:26:25,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2068226.6666666667, ans=0.1
2023-11-22 19:26:30,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2068226.6666666667, ans=0.2
2023-11-22 19:26:32,853 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9650, loss[loss=0.07529, simple_loss=0.09544, pruned_loss=0.01696, audio_tagging_loss=0.01061, over 14976.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09296, pruned_loss=0.01454, audio_tagging_loss=0.009421, over 3055876.00 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:26:34,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2068293.3333333333, ans=0.125
2023-11-22 19:26:37,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310250
2023-11-22 19:26:49,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2068360.0, ans=0.125
2023-11-22 19:27:19,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2068493.3333333333, ans=0.125
2023-11-22 19:27:24,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2068560.0, ans=0.125
2023-11-22 19:27:29,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2068560.0, ans=0.0
2023-11-22 19:27:31,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2068560.0, ans=0.1
2023-11-22 19:27:36,280 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9700, loss[loss=0.06228, simple_loss=0.07643, pruned_loss=0.01262, audio_tagging_loss=0.01144, over 14382.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09353, pruned_loss=0.01457, audio_tagging_loss=0.009267, over 3058731.05 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:27:36,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2068626.6666666667, ans=0.125
2023-11-22 19:27:41,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310300
2023-11-22 19:27:53,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2068693.3333333333, ans=0.125
2023-11-22 19:27:56,044 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.97 vs. limit=15.0
2023-11-22 19:28:26,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2068893.3333333333, ans=0.125
2023-11-22 19:28:27,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2068893.3333333333, ans=0.125
2023-11-22 19:28:29,028 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.323e+01 8.859e+01 9.612e+01 1.303e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 19:28:34,619 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.56 vs. limit=15.0
2023-11-22 19:28:37,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2068893.3333333333, ans=0.0
2023-11-22 19:28:38,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2068960.0, ans=0.125
2023-11-22 19:28:39,874 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9750, loss[loss=0.09584, simple_loss=0.1248, pruned_loss=0.02465, audio_tagging_loss=0.008793, over 15325.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09381, pruned_loss=0.01459, audio_tagging_loss=0.009163, over 3063105.53 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:28:45,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310350
2023-11-22 19:28:56,678 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.38 vs. limit=15.0
2023-11-22 19:29:09,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2069093.3333333333, ans=0.125
2023-11-22 19:29:09,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2069093.3333333333, ans=0.05
2023-11-22 19:29:11,415 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.93 vs. limit=22.5
2023-11-22 19:29:14,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2069093.3333333333, ans=0.0
2023-11-22 19:29:17,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2069160.0, ans=0.125
2023-11-22 19:29:26,780 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.34 vs. limit=6.0
2023-11-22 19:29:38,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2069226.6666666667, ans=0.125
2023-11-22 19:29:38,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2069226.6666666667, ans=0.2
2023-11-22 19:29:40,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.97 vs. limit=6.0
2023-11-22 19:29:44,816 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9800, loss[loss=0.08897, simple_loss=0.1245, pruned_loss=0.02059, audio_tagging_loss=0.006134, over 15767.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09478, pruned_loss=0.01487, audio_tagging_loss=0.009045, over 3057144.43 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:29:49,761 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310400
2023-11-22 19:29:51,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.93 vs. limit=15.0
2023-11-22 19:30:31,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2069493.3333333333, ans=0.125
2023-11-22 19:30:37,549 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.460e+01 9.204e+01 9.900e+01 1.495e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-22 19:30:37,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2069560.0, ans=0.0
2023-11-22 19:30:41,215 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:30:41,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2069560.0, ans=0.025
2023-11-22 19:30:48,403 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9850, loss[loss=0.06426, simple_loss=0.08772, pruned_loss=0.01069, audio_tagging_loss=0.009722, over 14411.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09548, pruned_loss=0.01507, audio_tagging_loss=0.008946, over 3053981.47 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 8.0
2023-11-22 19:30:53,383 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310450
2023-11-22 19:31:06,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2069693.3333333333, ans=0.0
2023-11-22 19:31:19,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2069760.0, ans=0.0
2023-11-22 19:31:39,885 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.11 vs. limit=15.0
2023-11-22 19:31:45,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2069893.3333333333, ans=0.125
2023-11-22 19:31:51,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2069960.0, ans=0.2
2023-11-22 19:31:51,986 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9900, loss[loss=0.07495, simple_loss=0.1064, pruned_loss=0.01387, audio_tagging_loss=0.00788, over 15025.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09631, pruned_loss=0.01516, audio_tagging_loss=0.008912, over 3049258.23 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 8.0
2023-11-22 19:31:56,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.17 vs. limit=15.0
2023-11-22 19:31:56,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310500
2023-11-22 19:32:00,414 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.76 vs. limit=22.5
2023-11-22 19:32:03,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0
2023-11-22 19:32:09,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.59 vs. limit=6.0
2023-11-22 19:32:16,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2070026.6666666667, ans=0.125
2023-11-22 19:32:20,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2070093.3333333333, ans=0.125
2023-11-22 19:32:45,208 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.326e+01 8.797e+01 9.739e+01 1.395e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 19:32:45,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2070226.6666666667, ans=0.07
2023-11-22 19:32:56,579 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 9950, loss[loss=0.08135, simple_loss=0.1127, pruned_loss=0.01995, audio_tagging_loss=0.00504, over 14642.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09624, pruned_loss=0.01509, audio_tagging_loss=0.008914, over 3040431.46 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 8.0
2023-11-22 19:32:57,309 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.80 vs. limit=15.0
2023-11-22 19:32:59,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2070293.3333333333, ans=0.0
2023-11-22 19:32:59,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.12 vs. limit=15.0
2023-11-22 19:33:01,496 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310550
2023-11-22 19:33:06,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2070293.3333333333, ans=0.0
2023-11-22 19:33:13,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2070360.0, ans=0.2
2023-11-22 19:33:32,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2070493.3333333333, ans=0.0
2023-11-22 19:33:44,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2070493.3333333333, ans=0.1
2023-11-22 19:33:55,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2070560.0, ans=0.125
2023-11-22 19:33:59,657 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10000, loss[loss=0.0429, simple_loss=0.05009, pruned_loss=0.006941, audio_tagging_loss=0.01091, over 15158.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09581, pruned_loss=0.01504, audio_tagging_loss=0.009014, over 3042700.62 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:34:04,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310600
2023-11-22 19:34:16,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.42 vs. limit=15.0
2023-11-22 19:34:39,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2070826.6666666667, ans=0.2
2023-11-22 19:34:45,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.17 vs. limit=15.0
2023-11-22 19:34:53,490 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.266e+01 8.772e+01 9.539e+01 1.390e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 19:35:03,288 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10050, loss[loss=0.06735, simple_loss=0.08626, pruned_loss=0.01223, audio_tagging_loss=0.01199, over 14968.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09448, pruned_loss=0.01469, audio_tagging_loss=0.009165, over 3034940.86 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:35:08,952 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310650
2023-11-22 19:35:11,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2070960.0, ans=0.1
2023-11-22 19:35:23,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2071026.6666666667, ans=0.0
2023-11-22 19:35:56,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2071226.6666666667, ans=0.1
2023-11-22 19:35:56,895 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.72 vs. limit=15.0
2023-11-22 19:36:08,959 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10100, loss[loss=0.07746, simple_loss=0.1086, pruned_loss=0.01501, audio_tagging_loss=0.008172, over 14959.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09364, pruned_loss=0.01454, audio_tagging_loss=0.00927, over 3037813.91 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:36:14,498 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310700
2023-11-22 19:36:21,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2071360.0, ans=0.0
2023-11-22 19:36:29,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2071360.0, ans=0.125
2023-11-22 19:36:42,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2071426.6666666667, ans=0.0
2023-11-22 19:36:58,555 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:37:02,765 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.311e+01 8.891e+01 9.853e+01 2.191e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-22 19:37:08,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2071560.0, ans=0.0
2023-11-22 19:37:11,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2071626.6666666667, ans=0.125
2023-11-22 19:37:12,762 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10150, loss[loss=0.0664, simple_loss=0.0923, pruned_loss=0.01049, audio_tagging_loss=0.009757, over 16201.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09314, pruned_loss=0.01454, audio_tagging_loss=0.009342, over 3040792.93 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:37:14,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2071626.6666666667, ans=0.95
2023-11-22 19:37:17,772 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310750
2023-11-22 19:37:41,910 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:37:51,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.25 vs. limit=15.0
2023-11-22 19:37:53,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2071826.6666666667, ans=0.0
2023-11-22 19:38:02,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.02 vs. limit=15.0
2023-11-22 19:38:02,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2071893.3333333333, ans=0.125
2023-11-22 19:38:05,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2071893.3333333333, ans=0.0
2023-11-22 19:38:15,731 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10200, loss[loss=0.0741, simple_loss=0.1041, pruned_loss=0.01385, audio_tagging_loss=0.008217, over 16036.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09356, pruned_loss=0.01466, audio_tagging_loss=0.009433, over 3043796.98 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:38:20,679 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310800
2023-11-22 19:38:25,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.65 vs. limit=22.5
2023-11-22 19:38:37,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2072026.6666666667, ans=0.0
2023-11-22 19:38:39,609 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:39:08,471 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.131e+01 8.876e+01 9.472e+01 1.289e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 19:39:16,852 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.33 vs. limit=15.0
2023-11-22 19:39:19,373 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10250, loss[loss=0.07775, simple_loss=0.09585, pruned_loss=0.02178, audio_tagging_loss=0.00804, over 15881.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09424, pruned_loss=0.01468, audio_tagging_loss=0.0093, over 3049659.62 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:39:25,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310850
2023-11-22 19:39:25,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2072293.3333333333, ans=0.0
2023-11-22 19:39:25,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2072293.3333333333, ans=0.125
2023-11-22 19:39:26,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2072293.3333333333, ans=0.1
2023-11-22 19:39:35,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2072360.0, ans=0.125
2023-11-22 19:39:52,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2072426.6666666667, ans=0.0
2023-11-22 19:39:55,509 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.54 vs. limit=12.0
2023-11-22 19:40:08,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2072493.3333333333, ans=0.125
2023-11-22 19:40:11,791 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.84 vs. limit=15.0
2023-11-22 19:40:23,793 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10300, loss[loss=0.06663, simple_loss=0.09443, pruned_loss=0.0115, audio_tagging_loss=0.007917, over 16066.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09396, pruned_loss=0.01458, audio_tagging_loss=0.009349, over 3054969.76 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:40:28,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310900
2023-11-22 19:40:29,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2072626.6666666667, ans=0.035
2023-11-22 19:40:37,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2072693.3333333333, ans=0.0
2023-11-22 19:40:47,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2072760.0, ans=0.0
2023-11-22 19:41:01,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2072826.6666666667, ans=0.0
2023-11-22 19:41:06,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2072826.6666666667, ans=0.125
2023-11-22 19:41:16,986 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.279e+01 8.744e+01 9.443e+01 1.295e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 19:41:17,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.03 vs. limit=12.0
2023-11-22 19:41:21,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2072893.3333333333, ans=0.0
2023-11-22 19:41:22,386 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.96 vs. limit=15.0
2023-11-22 19:41:26,739 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10350, loss[loss=0.07904, simple_loss=0.09965, pruned_loss=0.0172, audio_tagging_loss=0.01201, over 15319.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09519, pruned_loss=0.01486, audio_tagging_loss=0.009289, over 3050611.13 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:41:29,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2072960.0, ans=0.0
2023-11-22 19:41:30,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2072960.0, ans=0.0
2023-11-22 19:41:31,724 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 310950
2023-11-22 19:41:41,365 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.26 vs. limit=15.0
2023-11-22 19:41:51,179 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:42:14,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2073160.0, ans=0.2
2023-11-22 19:42:23,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=2073226.6666666667, ans=0.1
2023-11-22 19:42:30,615 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10400, loss[loss=0.06975, simple_loss=0.09686, pruned_loss=0.01286, audio_tagging_loss=0.008464, over 17103.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09526, pruned_loss=0.01489, audio_tagging_loss=0.009413, over 3046773.26 frames. ], batch size: 64, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:42:36,155 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311000
2023-11-22 19:42:40,774 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.55 vs. limit=15.0
2023-11-22 19:42:50,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2073360.0, ans=0.1
2023-11-22 19:42:50,536 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.03 vs. limit=22.5
2023-11-22 19:43:04,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2073426.6666666667, ans=0.0
2023-11-22 19:43:07,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2073426.6666666667, ans=0.1
2023-11-22 19:43:09,075 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.89 vs. limit=8.0
2023-11-22 19:43:15,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2073493.3333333333, ans=0.125
2023-11-22 19:43:25,682 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.233e+01 8.739e+01 9.492e+01 1.200e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 19:43:36,097 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10450, loss[loss=0.08064, simple_loss=0.1186, pruned_loss=0.01479, audio_tagging_loss=0.006529, over 14638.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09518, pruned_loss=0.01485, audio_tagging_loss=0.009369, over 3051296.78 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:43:41,077 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311050
2023-11-22 19:43:47,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2073693.3333333333, ans=0.2
2023-11-22 19:43:52,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2073693.3333333333, ans=0.1
2023-11-22 19:44:05,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2073760.0, ans=0.0
2023-11-22 19:44:18,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2073826.6666666667, ans=0.2
2023-11-22 19:44:33,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2073893.3333333333, ans=0.05
2023-11-22 19:44:37,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2073893.3333333333, ans=0.125
2023-11-22 19:44:39,303 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10500, loss[loss=0.07769, simple_loss=0.1011, pruned_loss=0.01858, audio_tagging_loss=0.00858, over 14342.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09463, pruned_loss=0.0149, audio_tagging_loss=0.009387, over 3042061.18 frames. ], batch size: 53, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:44:44,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311100
2023-11-22 19:45:33,290 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.986e+01 7.970e+01 8.549e+01 9.373e+01 1.177e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 19:45:42,200 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10550, loss[loss=0.05667, simple_loss=0.07439, pruned_loss=0.009845, audio_tagging_loss=0.009633, over 15275.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09409, pruned_loss=0.01482, audio_tagging_loss=0.009324, over 3046121.46 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:45:45,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2074293.3333333333, ans=0.125
2023-11-22 19:45:48,637 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311150
2023-11-22 19:45:52,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2074293.3333333333, ans=0.0
2023-11-22 19:46:01,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2074360.0, ans=0.1
2023-11-22 19:46:18,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.53 vs. limit=15.0
2023-11-22 19:46:47,681 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10600, loss[loss=0.07297, simple_loss=0.09291, pruned_loss=0.01569, audio_tagging_loss=0.01082, over 15337.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.0937, pruned_loss=0.01469, audio_tagging_loss=0.009309, over 3045677.70 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:46:53,172 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311200
2023-11-22 19:46:53,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2074626.6666666667, ans=0.125
2023-11-22 19:46:57,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2074626.6666666667, ans=0.0
2023-11-22 19:47:07,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2074693.3333333333, ans=0.0
2023-11-22 19:47:40,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2074893.3333333333, ans=0.09899494936611666
2023-11-22 19:47:43,209 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.586e+01 8.272e+01 8.914e+01 9.712e+01 1.198e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-22 19:47:51,573 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10650, loss[loss=0.07414, simple_loss=0.09044, pruned_loss=0.01719, audio_tagging_loss=0.01174, over 15704.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09508, pruned_loss=0.01501, audio_tagging_loss=0.009256, over 3045850.67 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:47:51,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2074960.0, ans=0.125
2023-11-22 19:47:56,522 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311250
2023-11-22 19:47:57,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2074960.0, ans=0.1
2023-11-22 19:47:58,313 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.22 vs. limit=15.0
2023-11-22 19:48:00,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2074960.0, ans=0.2
2023-11-22 19:48:01,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2074960.0, ans=0.1
2023-11-22 19:48:16,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2075093.3333333333, ans=0.0
2023-11-22 19:48:18,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2075093.3333333333, ans=0.1
2023-11-22 19:48:25,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2075093.3333333333, ans=0.05
2023-11-22 19:48:49,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2075226.6666666667, ans=0.2
2023-11-22 19:48:54,750 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10700, loss[loss=0.06215, simple_loss=0.08478, pruned_loss=0.01054, audio_tagging_loss=0.009219, over 15182.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09444, pruned_loss=0.01489, audio_tagging_loss=0.009242, over 3055046.11 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:49:00,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311300
2023-11-22 19:49:12,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2075360.0, ans=0.125
2023-11-22 19:49:14,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2075360.0, ans=0.0
2023-11-22 19:49:26,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2075426.6666666667, ans=0.2
2023-11-22 19:49:31,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2075426.6666666667, ans=0.2
2023-11-22 19:49:33,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2075493.3333333333, ans=0.1
2023-11-22 19:49:41,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2075493.3333333333, ans=0.09899494936611666
2023-11-22 19:49:43,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2075493.3333333333, ans=0.0
2023-11-22 19:49:50,182 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.233e+01 8.994e+01 9.619e+01 1.201e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 19:50:00,036 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10750, loss[loss=0.06119, simple_loss=0.07628, pruned_loss=0.01316, audio_tagging_loss=0.009891, over 15696.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09379, pruned_loss=0.01481, audio_tagging_loss=0.009283, over 3063789.32 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:50:05,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311350
2023-11-22 19:50:21,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2075693.3333333333, ans=0.0
2023-11-22 19:50:22,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2075693.3333333333, ans=0.125
2023-11-22 19:50:25,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2075760.0, ans=0.125
2023-11-22 19:50:42,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=22.5
2023-11-22 19:50:47,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2075826.6666666667, ans=0.0
2023-11-22 19:51:04,223 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10800, loss[loss=0.08469, simple_loss=0.1139, pruned_loss=0.02038, audio_tagging_loss=0.00738, over 15665.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09487, pruned_loss=0.01507, audio_tagging_loss=0.009226, over 3064586.84 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:51:09,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311400
2023-11-22 19:51:51,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2076160.0, ans=0.1
2023-11-22 19:51:59,225 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.263e+01 8.884e+01 9.522e+01 1.111e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 19:52:07,988 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10850, loss[loss=0.08124, simple_loss=0.1061, pruned_loss=0.01901, audio_tagging_loss=0.009161, over 15315.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09394, pruned_loss=0.01498, audio_tagging_loss=0.009273, over 3054948.79 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:52:13,046 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311450
2023-11-22 19:52:14,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2076293.3333333333, ans=0.125
2023-11-22 19:52:16,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2076293.3333333333, ans=0.125
2023-11-22 19:52:23,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2076360.0, ans=0.2
2023-11-22 19:52:28,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2076360.0, ans=0.125
2023-11-22 19:52:31,063 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.88 vs. limit=10.0
2023-11-22 19:52:46,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2076493.3333333333, ans=0.0
2023-11-22 19:53:07,694 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:53:13,058 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10900, loss[loss=0.06093, simple_loss=0.08179, pruned_loss=0.0117, audio_tagging_loss=0.008338, over 14237.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09446, pruned_loss=0.01501, audio_tagging_loss=0.009201, over 3056824.10 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:53:17,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311500
2023-11-22 19:53:18,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.84 vs. limit=22.5
2023-11-22 19:53:25,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2076693.3333333333, ans=0.1
2023-11-22 19:53:50,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2076826.6666666667, ans=0.2
2023-11-22 19:53:51,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.72 vs. limit=15.0
2023-11-22 19:53:54,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2076826.6666666667, ans=0.0
2023-11-22 19:54:08,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2076893.3333333333, ans=0.125
2023-11-22 19:54:09,636 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.142e+01 8.836e+01 9.377e+01 1.085e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 19:54:11,576 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.92 vs. limit=22.5
2023-11-22 19:54:13,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2076893.3333333333, ans=0.125
2023-11-22 19:54:16,919 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 10950, loss[loss=0.08885, simple_loss=0.1103, pruned_loss=0.02313, audio_tagging_loss=0.01056, over 15902.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09342, pruned_loss=0.01471, audio_tagging_loss=0.009342, over 3054915.36 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:54:20,831 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:54:21,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311550
2023-11-22 19:54:27,851 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.81 vs. limit=22.5
2023-11-22 19:54:28,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2077026.6666666667, ans=0.125
2023-11-22 19:54:41,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2077093.3333333333, ans=0.125
2023-11-22 19:55:21,225 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11000, loss[loss=0.05284, simple_loss=0.06529, pruned_loss=0.01115, audio_tagging_loss=0.009046, over 15936.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09343, pruned_loss=0.01464, audio_tagging_loss=0.009313, over 3050489.02 frames. ], batch size: 62, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:55:23,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2077293.3333333333, ans=0.125
2023-11-22 19:55:24,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=2077293.3333333333, ans=6.0
2023-11-22 19:55:26,151 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311600
2023-11-22 19:55:30,047 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:55:33,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2077360.0, ans=0.0
2023-11-22 19:55:49,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2077426.6666666667, ans=0.125
2023-11-22 19:56:00,617 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.00 vs. limit=15.0
2023-11-22 19:56:17,628 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.294e+01 8.863e+01 9.568e+01 1.311e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 19:56:26,011 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11050, loss[loss=0.05503, simple_loss=0.07529, pruned_loss=0.009181, audio_tagging_loss=0.008208, over 15129.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.0929, pruned_loss=0.01456, audio_tagging_loss=0.009387, over 3057489.47 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:56:31,562 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311650
2023-11-22 19:56:53,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2077760.0, ans=0.125
2023-11-22 19:57:00,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2077760.0, ans=0.125
2023-11-22 19:57:17,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2077893.3333333333, ans=0.1
2023-11-22 19:57:29,637 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11100, loss[loss=0.05287, simple_loss=0.06693, pruned_loss=0.006524, audio_tagging_loss=0.01288, over 14883.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.0933, pruned_loss=0.01458, audio_tagging_loss=0.009537, over 3055631.41 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:57:30,306 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.64 vs. limit=15.0
2023-11-22 19:57:34,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311700
2023-11-22 19:58:06,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2078093.3333333333, ans=0.0
2023-11-22 19:58:25,321 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.159e+01 8.862e+01 9.664e+01 1.542e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 19:58:30,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2078226.6666666667, ans=0.1
2023-11-22 19:58:32,634 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11150, loss[loss=0.07069, simple_loss=0.09566, pruned_loss=0.01294, audio_tagging_loss=0.009929, over 14805.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09363, pruned_loss=0.01461, audio_tagging_loss=0.009633, over 3051656.83 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:58:37,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311750
2023-11-22 19:58:57,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2078360.0, ans=0.125
2023-11-22 19:59:12,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.93 vs. limit=22.5
2023-11-22 19:59:12,387 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.24 vs. limit=22.5
2023-11-22 19:59:29,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2078560.0, ans=0.125
2023-11-22 19:59:37,906 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11200, loss[loss=0.08073, simple_loss=0.105, pruned_loss=0.01627, audio_tagging_loss=0.01195, over 14852.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09309, pruned_loss=0.01446, audio_tagging_loss=0.009741, over 3050975.65 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:59:42,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2078626.6666666667, ans=0.0
2023-11-22 19:59:43,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311800
2023-11-22 19:59:48,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2078626.6666666667, ans=0.0
2023-11-22 19:59:57,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2078693.3333333333, ans=0.0
2023-11-22 20:00:10,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2078760.0, ans=0.0
2023-11-22 20:00:18,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2078826.6666666667, ans=0.125
2023-11-22 20:00:34,749 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 7.967e+01 8.796e+01 9.529e+01 1.148e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 20:00:42,238 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11250, loss[loss=0.0792, simple_loss=0.1006, pruned_loss=0.01979, audio_tagging_loss=0.009092, over 14870.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09295, pruned_loss=0.01467, audio_tagging_loss=0.009668, over 3051140.28 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:00:47,387 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311850
2023-11-22 20:00:53,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2079026.6666666667, ans=0.1
2023-11-22 20:00:59,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.30 vs. limit=22.5
2023-11-22 20:01:01,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.52 vs. limit=22.5
2023-11-22 20:01:21,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2079160.0, ans=0.125
2023-11-22 20:01:28,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2079160.0, ans=0.125
2023-11-22 20:01:39,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2079226.6666666667, ans=0.0
2023-11-22 20:01:45,784 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11300, loss[loss=0.07186, simple_loss=0.09661, pruned_loss=0.01104, audio_tagging_loss=0.01251, over 15126.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09414, pruned_loss=0.01496, audio_tagging_loss=0.009466, over 3051209.94 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:01:47,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2079293.3333333333, ans=0.0
2023-11-22 20:01:50,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311900
2023-11-22 20:02:08,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2079360.0, ans=0.2
2023-11-22 20:02:26,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2079493.3333333333, ans=0.125
2023-11-22 20:02:27,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0
2023-11-22 20:02:29,085 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.07 vs. limit=15.0
2023-11-22 20:02:30,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.13 vs. limit=6.0
2023-11-22 20:02:39,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_na.min_abs, batch_count=2079560.0, ans=0.02
2023-11-22 20:02:40,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2079560.0, ans=0.07
2023-11-22 20:02:40,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2079560.0, ans=0.125
2023-11-22 20:02:41,665 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.136e+01 8.665e+01 9.544e+01 1.497e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 20:02:47,296 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.79 vs. limit=15.0
2023-11-22 20:02:47,791 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11350, loss[loss=0.07786, simple_loss=0.1067, pruned_loss=0.01627, audio_tagging_loss=0.008236, over 15711.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09357, pruned_loss=0.0149, audio_tagging_loss=0.009394, over 3046472.45 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:02:47,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2079626.6666666667, ans=0.125
2023-11-22 20:02:49,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2079626.6666666667, ans=0.125
2023-11-22 20:02:53,873 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 311950
2023-11-22 20:02:54,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2079626.6666666667, ans=0.125
2023-11-22 20:03:02,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2079693.3333333333, ans=0.125
2023-11-22 20:03:07,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2079693.3333333333, ans=0.1
2023-11-22 20:03:10,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2079693.3333333333, ans=0.125
2023-11-22 20:03:41,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2079893.3333333333, ans=0.125
2023-11-22 20:03:52,225 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11400, loss[loss=0.07244, simple_loss=0.1049, pruned_loss=0.01394, audio_tagging_loss=0.006076, over 16050.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09356, pruned_loss=0.01499, audio_tagging_loss=0.009327, over 3035059.34 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:03:57,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312000
2023-11-22 20:03:58,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2079960.0, ans=0.1
2023-11-22 20:04:32,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2080160.0, ans=0.125
2023-11-22 20:04:37,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2080160.0, ans=0.0
2023-11-22 20:04:39,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2080160.0, ans=0.0
2023-11-22 20:04:52,984 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.235e+01 8.932e+01 9.432e+01 1.169e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 20:04:59,084 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11450, loss[loss=0.05819, simple_loss=0.07795, pruned_loss=0.01027, audio_tagging_loss=0.008947, over 14876.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09328, pruned_loss=0.0149, audio_tagging_loss=0.009292, over 3034540.74 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:05:03,990 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312050
2023-11-22 20:05:04,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2080293.3333333333, ans=0.1
2023-11-22 20:05:04,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2080293.3333333333, ans=0.1
2023-11-22 20:05:50,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2080560.0, ans=0.0
2023-11-22 20:06:02,411 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11500, loss[loss=0.06444, simple_loss=0.08584, pruned_loss=0.01094, audio_tagging_loss=0.01058, over 14322.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09319, pruned_loss=0.01478, audio_tagging_loss=0.009285, over 3033164.23 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:06:07,713 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312100
2023-11-22 20:06:09,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2080626.6666666667, ans=0.125
2023-11-22 20:06:12,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2080626.6666666667, ans=0.125
2023-11-22 20:06:12,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2080626.6666666667, ans=0.125
2023-11-22 20:06:15,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2080693.3333333333, ans=0.125
2023-11-22 20:06:19,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2080693.3333333333, ans=0.125
2023-11-22 20:06:41,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2080826.6666666667, ans=0.1
2023-11-22 20:06:56,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2080893.3333333333, ans=0.125
2023-11-22 20:07:00,687 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.062e+01 8.847e+01 9.578e+01 1.218e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 20:07:06,747 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11550, loss[loss=0.03617, simple_loss=0.04627, pruned_loss=0.003067, audio_tagging_loss=0.009969, over 14774.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09355, pruned_loss=0.01479, audio_tagging_loss=0.009103, over 3036282.56 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:07:11,672 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312150
2023-11-22 20:07:36,317 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0
2023-11-22 20:07:43,832 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:07:49,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2081160.0, ans=0.2
2023-11-22 20:08:06,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2081226.6666666667, ans=0.015
2023-11-22 20:08:09,793 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11600, loss[loss=0.08334, simple_loss=0.1116, pruned_loss=0.01907, audio_tagging_loss=0.00849, over 14858.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09382, pruned_loss=0.01494, audio_tagging_loss=0.009116, over 3033579.61 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:08:14,771 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312200
2023-11-22 20:08:23,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2081360.0, ans=0.2
2023-11-22 20:08:46,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2081426.6666666667, ans=0.0
2023-11-22 20:08:48,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2081493.3333333333, ans=0.125
2023-11-22 20:09:08,466 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.935e+01 8.330e+01 9.074e+01 9.765e+01 1.554e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-22 20:09:10,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2081560.0, ans=0.09899494936611666
2023-11-22 20:09:13,418 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11650, loss[loss=0.06616, simple_loss=0.08612, pruned_loss=0.01491, audio_tagging_loss=0.008195, over 14869.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09325, pruned_loss=0.0148, audio_tagging_loss=0.009207, over 3041987.64 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:09:17,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.95 vs. limit=12.0
2023-11-22 20:09:18,981 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312250
2023-11-22 20:09:19,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2081626.6666666667, ans=0.0
2023-11-22 20:09:32,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2081693.3333333333, ans=0.1
2023-11-22 20:09:36,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2081693.3333333333, ans=0.2
2023-11-22 20:09:41,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2081760.0, ans=0.0
2023-11-22 20:09:43,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2081760.0, ans=0.0
2023-11-22 20:09:52,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2081826.6666666667, ans=0.125
2023-11-22 20:09:53,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2081826.6666666667, ans=0.1
2023-11-22 20:10:03,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2081893.3333333333, ans=0.125
2023-11-22 20:10:18,460 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11700, loss[loss=0.05305, simple_loss=0.06127, pruned_loss=0.01039, audio_tagging_loss=0.01202, over 15137.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09258, pruned_loss=0.01461, audio_tagging_loss=0.009383, over 3037559.21 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:10:23,958 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312300
2023-11-22 20:10:44,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2082093.3333333333, ans=0.2
2023-11-22 20:11:13,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2082226.6666666667, ans=0.125
2023-11-22 20:11:17,082 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.195e+01 8.722e+01 9.532e+01 1.280e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 20:11:22,126 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11750, loss[loss=0.07451, simple_loss=0.09462, pruned_loss=0.01419, audio_tagging_loss=0.01301, over 15061.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09202, pruned_loss=0.01459, audio_tagging_loss=0.009561, over 3042385.11 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:11:27,111 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312350
2023-11-22 20:11:44,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2082360.0, ans=0.1
2023-11-22 20:11:54,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2082426.6666666667, ans=0.125
2023-11-22 20:11:55,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2082426.6666666667, ans=0.2
2023-11-22 20:12:00,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.34 vs. limit=15.0
2023-11-22 20:12:02,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2082493.3333333333, ans=0.0
2023-11-22 20:12:25,274 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11800, loss[loss=0.05847, simple_loss=0.07752, pruned_loss=0.009616, audio_tagging_loss=0.01009, over 16124.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09163, pruned_loss=0.01458, audio_tagging_loss=0.009476, over 3044584.30 frames. ], batch size: 62, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:12:30,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312400
2023-11-22 20:12:44,467 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:12:49,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2082693.3333333333, ans=0.125
2023-11-22 20:13:24,248 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.495e+01 8.351e+01 8.869e+01 9.580e+01 1.621e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 20:13:30,285 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11850, loss[loss=0.03611, simple_loss=0.04766, pruned_loss=0.003484, audio_tagging_loss=0.008796, over 15808.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09136, pruned_loss=0.0146, audio_tagging_loss=0.009583, over 3045439.51 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:13:35,347 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312450
2023-11-22 20:13:39,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2082960.0, ans=0.1
2023-11-22 20:14:09,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2083160.0, ans=0.95
2023-11-22 20:14:15,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2083160.0, ans=0.1
2023-11-22 20:14:25,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2083226.6666666667, ans=0.125
2023-11-22 20:14:34,580 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11900, loss[loss=0.07934, simple_loss=0.1018, pruned_loss=0.01886, audio_tagging_loss=0.009594, over 15546.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09171, pruned_loss=0.01473, audio_tagging_loss=0.009657, over 3042175.19 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:14:39,591 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312500
2023-11-22 20:14:42,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2083293.3333333333, ans=0.125
2023-11-22 20:15:16,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2083493.3333333333, ans=0.0
2023-11-22 20:15:17,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2083493.3333333333, ans=0.0
2023-11-22 20:15:26,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2083560.0, ans=0.125
2023-11-22 20:15:29,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2083560.0, ans=10.0
2023-11-22 20:15:32,655 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.017e+01 8.415e+01 9.018e+01 9.656e+01 1.551e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-22 20:15:37,471 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 11950, loss[loss=0.06692, simple_loss=0.08975, pruned_loss=0.01305, audio_tagging_loss=0.008998, over 15464.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09182, pruned_loss=0.01484, audio_tagging_loss=0.009565, over 3042213.41 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:15:42,441 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312550
2023-11-22 20:16:17,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2083826.6666666667, ans=0.2
2023-11-22 20:16:25,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2083826.6666666667, ans=15.0
2023-11-22 20:16:38,907 INFO [train_asr.py:1221] (2/4) Epoch 26, batch 12000, loss[loss=0.0844, simple_loss=0.1082, pruned_loss=0.02177, audio_tagging_loss=0.008505, over 15855.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09225, pruned_loss=0.01492, audio_tagging_loss=0.009675, over 3040182.60 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:16:38,908 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 20:17:20,963 INFO [train_asr.py:1253] (2/4) Epoch 26, validation: loss=0.05885, simple_loss=0.05139, pruned_loss=0.00512, audio_tagging_loss=0.02803, over 4681554.00 frames. 
2023-11-22 20:17:20,964 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 20:17:25,611 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312600
2023-11-22 20:17:25,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2083960.0, ans=0.125
2023-11-22 20:17:30,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2083960.0, ans=0.1
2023-11-22 20:17:37,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2084026.6666666667, ans=0.2
2023-11-22 20:17:38,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2084026.6666666667, ans=0.0
2023-11-22 20:18:24,319 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 0, loss[loss=0.07916, simple_loss=0.08986, pruned_loss=0.01299, audio_tagging_loss=0.02123, over 14792.00 frames. ], tot_loss[loss=0.07916, simple_loss=0.08986, pruned_loss=0.01299, audio_tagging_loss=0.02123, over 14792.00 frames. ], batch size: 57, lr: 2.55e-03, grad_scale: 32.0
2023-11-22 20:18:24,319 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 20:18:40,376 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4017, 4.0206, 4.3679, 3.9901], device='cuda:2')
2023-11-22 20:19:01,963 INFO [train_asr.py:1253] (2/4) Epoch 27, validation: loss=0.05818, simple_loss=0.05133, pruned_loss=0.005046, audio_tagging_loss=0.02747, over 4681554.00 frames. 
2023-11-22 20:19:01,964 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 20:19:22,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2084180.0, ans=0.125
2023-11-22 20:19:28,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2084246.6666666667, ans=0.125
2023-11-22 20:19:32,309 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.281e+01 9.255e+01 1.009e+02 1.305e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-22 20:19:35,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2084246.6666666667, ans=0.125
2023-11-22 20:19:43,047 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312650
2023-11-22 20:19:54,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2084380.0, ans=0.1
2023-11-22 20:20:00,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2084380.0, ans=0.0
2023-11-22 20:20:06,129 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 50, loss[loss=0.07156, simple_loss=0.08833, pruned_loss=0.009793, audio_tagging_loss=0.0176, over 15126.00 frames. ], tot_loss[loss=0.07945, simple_loss=0.09397, pruned_loss=0.01472, audio_tagging_loss=0.01775, over 692000.58 frames. ], batch size: 59, lr: 2.55e-03, grad_scale: 16.0
2023-11-22 20:20:13,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2084446.6666666667, ans=0.125
2023-11-22 20:20:33,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2084580.0, ans=0.0
2023-11-22 20:20:47,028 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312700
2023-11-22 20:21:04,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2084713.3333333333, ans=0.0
2023-11-22 20:21:12,349 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 100, loss[loss=0.08317, simple_loss=0.1064, pruned_loss=0.01515, audio_tagging_loss=0.01483, over 15565.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09211, pruned_loss=0.01419, audio_tagging_loss=0.01726, over 1211408.94 frames. ], batch size: 55, lr: 2.55e-03, grad_scale: 16.0
2023-11-22 20:21:14,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2084780.0, ans=0.125
2023-11-22 20:21:31,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2084846.6666666667, ans=0.125
2023-11-22 20:21:33,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.93 vs. limit=15.0
2023-11-22 20:21:42,709 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.400e+01 8.923e+01 9.571e+01 1.006e+02 1.303e+02, threshold=1.914e+02, percent-clipped=0.0
2023-11-22 20:21:51,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312750
2023-11-22 20:21:54,263 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.93 vs. limit=10.0
2023-11-22 20:21:56,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.72 vs. limit=15.0
2023-11-22 20:21:59,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2084980.0, ans=0.0
2023-11-22 20:22:17,539 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 150, loss[loss=0.08513, simple_loss=0.1118, pruned_loss=0.01641, audio_tagging_loss=0.01281, over 15437.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09298, pruned_loss=0.01443, audio_tagging_loss=0.01546, over 1619634.18 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:22:27,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2085113.3333333333, ans=0.1
2023-11-22 20:22:32,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2085180.0, ans=0.07
2023-11-22 20:22:42,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.03 vs. limit=15.0
2023-11-22 20:22:46,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2085246.6666666667, ans=0.125
2023-11-22 20:22:49,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2085246.6666666667, ans=0.125
2023-11-22 20:22:51,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2085246.6666666667, ans=0.2
2023-11-22 20:22:54,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2085246.6666666667, ans=0.1
2023-11-22 20:22:57,507 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312800
2023-11-22 20:23:22,027 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 200, loss[loss=0.07357, simple_loss=0.0981, pruned_loss=0.01487, audio_tagging_loss=0.009643, over 15467.00 frames. ], tot_loss[loss=0.07478, simple_loss=0.09327, pruned_loss=0.0145, audio_tagging_loss=0.01364, over 1939720.99 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:23:22,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2085446.6666666667, ans=0.0
2023-11-22 20:23:26,484 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0
2023-11-22 20:23:49,320 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=15.0
2023-11-22 20:23:53,564 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.118e+01 8.443e+01 8.985e+01 9.584e+01 1.364e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 20:24:03,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312850
2023-11-22 20:24:04,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2085646.6666666667, ans=0.125
2023-11-22 20:24:22,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0
2023-11-22 20:24:25,784 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:24:28,005 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 250, loss[loss=0.08181, simple_loss=0.1106, pruned_loss=0.01679, audio_tagging_loss=0.00975, over 16304.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09258, pruned_loss=0.01435, audio_tagging_loss=0.01241, over 2190258.49 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:24:28,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2085780.0, ans=0.125
2023-11-22 20:24:35,075 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.28 vs. limit=15.0
2023-11-22 20:24:51,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2085846.6666666667, ans=0.2
2023-11-22 20:24:57,524 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:25:02,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2085913.3333333333, ans=0.125
2023-11-22 20:25:07,724 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312900
2023-11-22 20:25:31,868 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 300, loss[loss=0.08319, simple_loss=0.1114, pruned_loss=0.01699, audio_tagging_loss=0.0105, over 15297.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09352, pruned_loss=0.01458, audio_tagging_loss=0.01144, over 2382282.82 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:25:37,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2086113.3333333333, ans=0.04949747468305833
2023-11-22 20:25:51,238 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:25:58,685 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.37 vs. limit=15.0
2023-11-22 20:26:03,564 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.485e+01 8.248e+01 9.049e+01 9.988e+01 1.263e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 20:26:06,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2086246.6666666667, ans=0.125
2023-11-22 20:26:07,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=2086246.6666666667, ans=0.1
2023-11-22 20:26:12,557 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 312950
2023-11-22 20:26:17,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2086313.3333333333, ans=0.09899494936611666
2023-11-22 20:26:20,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2086313.3333333333, ans=0.125
2023-11-22 20:26:21,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2086313.3333333333, ans=0.125
2023-11-22 20:26:32,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.52 vs. limit=15.0
2023-11-22 20:26:36,513 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 350, loss[loss=0.06731, simple_loss=0.08504, pruned_loss=0.0104, audio_tagging_loss=0.01439, over 14527.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09361, pruned_loss=0.01467, audio_tagging_loss=0.01085, over 2533143.13 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:26:42,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.05 vs. limit=15.0
2023-11-22 20:26:44,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2086446.6666666667, ans=0.1
2023-11-22 20:26:53,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2086513.3333333333, ans=0.0
2023-11-22 20:26:53,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2086513.3333333333, ans=0.0
2023-11-22 20:26:58,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2086513.3333333333, ans=0.1
2023-11-22 20:27:15,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2086646.6666666667, ans=0.1
2023-11-22 20:27:16,435 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313000
2023-11-22 20:27:23,918 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=5.247e-03
2023-11-22 20:27:33,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2086713.3333333333, ans=0.2
2023-11-22 20:27:33,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2086713.3333333333, ans=0.0
2023-11-22 20:27:42,172 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 400, loss[loss=0.09513, simple_loss=0.1285, pruned_loss=0.0218, audio_tagging_loss=0.009081, over 15329.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09407, pruned_loss=0.01474, audio_tagging_loss=0.01051, over 2644971.10 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:27:57,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2086846.6666666667, ans=0.125
2023-11-22 20:27:57,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2086846.6666666667, ans=0.0
2023-11-22 20:28:05,130 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=12.0
2023-11-22 20:28:12,328 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.142e+01 8.725e+01 9.349e+01 1.086e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 20:28:21,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313050
2023-11-22 20:28:33,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2087046.6666666667, ans=0.1
2023-11-22 20:28:39,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2087046.6666666667, ans=0.0
2023-11-22 20:28:45,810 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 450, loss[loss=0.06801, simple_loss=0.09182, pruned_loss=0.01173, audio_tagging_loss=0.01036, over 16201.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09304, pruned_loss=0.01468, audio_tagging_loss=0.01023, over 2734389.58 frames. ], batch size: 60, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:28:46,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2087113.3333333333, ans=10.0
2023-11-22 20:28:52,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2087113.3333333333, ans=0.0
2023-11-22 20:29:10,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2087246.6666666667, ans=10.0
2023-11-22 20:29:14,680 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.27 vs. limit=22.5
2023-11-22 20:29:20,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2087246.6666666667, ans=0.1
2023-11-22 20:29:25,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313100
2023-11-22 20:29:49,313 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 500, loss[loss=0.08598, simple_loss=0.1153, pruned_loss=0.01851, audio_tagging_loss=0.00982, over 16178.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09326, pruned_loss=0.01476, audio_tagging_loss=0.009965, over 2808663.72 frames. ], batch size: 60, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:29:57,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2087446.6666666667, ans=0.1
2023-11-22 20:29:57,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2087446.6666666667, ans=0.125
2023-11-22 20:29:57,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2087446.6666666667, ans=0.2
2023-11-22 20:30:03,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2087513.3333333333, ans=0.0
2023-11-22 20:30:06,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2087513.3333333333, ans=0.125
2023-11-22 20:30:09,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2087513.3333333333, ans=0.125
2023-11-22 20:30:19,920 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.170e+01 8.718e+01 9.505e+01 1.331e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 20:30:28,562 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313150
2023-11-22 20:30:41,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2087713.3333333333, ans=0.2
2023-11-22 20:30:45,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2087713.3333333333, ans=0.125
2023-11-22 20:30:50,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0
2023-11-22 20:30:54,209 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 550, loss[loss=0.06001, simple_loss=0.07994, pruned_loss=0.01103, audio_tagging_loss=0.009001, over 15292.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09252, pruned_loss=0.0146, audio_tagging_loss=0.009908, over 2858437.92 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:31:00,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2087780.0, ans=0.125
2023-11-22 20:31:05,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2087846.6666666667, ans=0.0
2023-11-22 20:31:21,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2087913.3333333333, ans=0.2
2023-11-22 20:31:31,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2087980.0, ans=0.04949747468305833
2023-11-22 20:31:32,779 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313200
2023-11-22 20:31:34,349 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.74 vs. limit=15.0
2023-11-22 20:31:44,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2088046.6666666667, ans=0.2
2023-11-22 20:31:53,452 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.33 vs. limit=22.5
2023-11-22 20:31:57,882 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 600, loss[loss=0.07347, simple_loss=0.09337, pruned_loss=0.01725, audio_tagging_loss=0.009527, over 15991.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09206, pruned_loss=0.01446, audio_tagging_loss=0.009882, over 2904076.14 frames. ], batch size: 61, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:32:03,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2088113.3333333333, ans=0.125
2023-11-22 20:32:06,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2088113.3333333333, ans=0.125
2023-11-22 20:32:28,510 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.589e+01 9.222e+01 1.004e+02 1.377e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-22 20:32:35,910 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.79 vs. limit=15.0
2023-11-22 20:32:37,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313250
2023-11-22 20:32:48,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2088380.0, ans=0.0
2023-11-22 20:33:01,054 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 650, loss[loss=0.07005, simple_loss=0.08873, pruned_loss=0.01577, audio_tagging_loss=0.009918, over 15548.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09259, pruned_loss=0.01451, audio_tagging_loss=0.009768, over 2938369.09 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:33:02,840 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.36 vs. limit=6.0
2023-11-22 20:33:09,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2088446.6666666667, ans=0.2
2023-11-22 20:33:14,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2088513.3333333333, ans=0.0
2023-11-22 20:33:17,697 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.65 vs. limit=15.0
2023-11-22 20:33:19,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2088513.3333333333, ans=0.05
2023-11-22 20:33:29,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2088580.0, ans=0.2
2023-11-22 20:33:42,050 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313300
2023-11-22 20:33:45,180 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.12 vs. limit=15.0
2023-11-22 20:33:47,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2088646.6666666667, ans=0.125
2023-11-22 20:34:04,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2088780.0, ans=0.2
2023-11-22 20:34:06,057 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 700, loss[loss=0.05685, simple_loss=0.07236, pruned_loss=0.009597, audio_tagging_loss=0.01107, over 15118.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09298, pruned_loss=0.01463, audio_tagging_loss=0.009657, over 2967925.14 frames. ], batch size: 61, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:34:11,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0
2023-11-22 20:34:35,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2088913.3333333333, ans=0.125
2023-11-22 20:34:36,743 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.373e+01 8.817e+01 9.744e+01 1.427e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 20:34:37,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.55 vs. limit=15.0
2023-11-22 20:34:39,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2088913.3333333333, ans=0.125
2023-11-22 20:34:45,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313350
2023-11-22 20:34:51,339 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:34:51,853 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0
2023-11-22 20:35:11,623 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 750, loss[loss=0.08333, simple_loss=0.1117, pruned_loss=0.0194, audio_tagging_loss=0.008074, over 15520.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09355, pruned_loss=0.01488, audio_tagging_loss=0.009535, over 2991601.90 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:35:23,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2089180.0, ans=0.1
2023-11-22 20:35:24,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.76 vs. limit=15.0
2023-11-22 20:35:39,115 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=22.5
2023-11-22 20:35:52,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313400
2023-11-22 20:35:55,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2089313.3333333333, ans=0.04949747468305833
2023-11-22 20:36:03,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2089380.0, ans=0.1
2023-11-22 20:36:10,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2089380.0, ans=0.1
2023-11-22 20:36:15,365 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 800, loss[loss=0.0586, simple_loss=0.0745, pruned_loss=0.01324, audio_tagging_loss=0.008116, over 17185.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09443, pruned_loss=0.01502, audio_tagging_loss=0.009517, over 3009926.45 frames. ], batch size: 67, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:36:21,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2089446.6666666667, ans=0.1
2023-11-22 20:36:25,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2089446.6666666667, ans=0.0
2023-11-22 20:36:32,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2089513.3333333333, ans=0.0
2023-11-22 20:36:47,044 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.197e+01 8.696e+01 9.566e+01 1.219e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 20:36:48,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2089580.0, ans=0.125
2023-11-22 20:36:55,823 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313450
2023-11-22 20:36:57,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2089646.6666666667, ans=0.125
2023-11-22 20:36:59,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2089646.6666666667, ans=0.125
2023-11-22 20:37:04,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.23 vs. limit=15.0
2023-11-22 20:37:18,935 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 850, loss[loss=0.08197, simple_loss=0.1101, pruned_loss=0.01884, audio_tagging_loss=0.008087, over 15519.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09342, pruned_loss=0.01478, audio_tagging_loss=0.009582, over 3014899.64 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:37:31,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2089780.0, ans=0.2
2023-11-22 20:37:39,084 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.67 vs. limit=15.0
2023-11-22 20:37:41,112 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:37:59,134 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313500
2023-11-22 20:38:02,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2089980.0, ans=0.0
2023-11-22 20:38:10,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2090046.6666666667, ans=0.2
2023-11-22 20:38:24,650 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 900, loss[loss=0.05613, simple_loss=0.06927, pruned_loss=0.01068, audio_tagging_loss=0.01081, over 15975.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09297, pruned_loss=0.01452, audio_tagging_loss=0.009685, over 3022680.77 frames. ], batch size: 60, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:38:34,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2090113.3333333333, ans=0.125
2023-11-22 20:38:34,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.44 vs. limit=22.5
2023-11-22 20:38:37,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2090180.0, ans=0.125
2023-11-22 20:38:38,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2090180.0, ans=0.07
2023-11-22 20:38:53,876 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.151e+01 8.989e+01 9.712e+01 1.407e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-22 20:39:04,388 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313550
2023-11-22 20:39:15,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2090380.0, ans=0.0
2023-11-22 20:39:16,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.12 vs. limit=15.0
2023-11-22 20:39:27,631 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 950, loss[loss=0.07569, simple_loss=0.09661, pruned_loss=0.01938, audio_tagging_loss=0.008011, over 14728.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09259, pruned_loss=0.01442, audio_tagging_loss=0.009614, over 3030338.05 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:39:34,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2090446.6666666667, ans=0.0
2023-11-22 20:39:43,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2090513.3333333333, ans=0.1
2023-11-22 20:39:58,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2090580.0, ans=0.125
2023-11-22 20:39:59,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2090580.0, ans=0.2
2023-11-22 20:40:07,737 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313600
2023-11-22 20:40:29,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2090713.3333333333, ans=0.125
2023-11-22 20:40:31,663 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1000, loss[loss=0.05165, simple_loss=0.06529, pruned_loss=0.009847, audio_tagging_loss=0.009156, over 14718.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09225, pruned_loss=0.01438, audio_tagging_loss=0.009432, over 3033942.96 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:40:49,222 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.51 vs. limit=22.5
2023-11-22 20:40:50,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2090846.6666666667, ans=0.2
2023-11-22 20:40:57,976 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:40:59,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2090913.3333333333, ans=0.2
2023-11-22 20:41:03,830 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.703e+01 8.088e+01 8.617e+01 9.308e+01 1.248e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-22 20:41:05,714 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=15.0
2023-11-22 20:41:11,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313650
2023-11-22 20:41:36,665 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1050, loss[loss=0.06623, simple_loss=0.08513, pruned_loss=0.01509, audio_tagging_loss=0.008573, over 16171.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09275, pruned_loss=0.0145, audio_tagging_loss=0.009236, over 3043866.54 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:41:58,241 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-22 20:41:58,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2091180.0, ans=0.0
2023-11-22 20:41:59,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2091180.0, ans=0.125
2023-11-22 20:42:15,406 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313700
2023-11-22 20:42:15,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2091313.3333333333, ans=0.2
2023-11-22 20:42:18,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2091313.3333333333, ans=0.1
2023-11-22 20:42:29,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2091380.0, ans=0.125
2023-11-22 20:42:39,787 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1100, loss[loss=0.06364, simple_loss=0.08536, pruned_loss=0.01261, audio_tagging_loss=0.008348, over 15546.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09238, pruned_loss=0.0146, audio_tagging_loss=0.009234, over 3033908.78 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:42:41,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2091446.6666666667, ans=0.1
2023-11-22 20:42:42,283 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:42:46,617 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.08 vs. limit=15.0
2023-11-22 20:43:12,478 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 8.123e+01 8.810e+01 9.567e+01 1.299e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 20:43:20,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313750
2023-11-22 20:43:23,408 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.28 vs. limit=15.0
2023-11-22 20:43:31,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.62 vs. limit=15.0
2023-11-22 20:43:43,606 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1150, loss[loss=0.07988, simple_loss=0.1024, pruned_loss=0.02213, audio_tagging_loss=0.006527, over 15289.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09297, pruned_loss=0.01453, audio_tagging_loss=0.009116, over 3039206.15 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:43:49,387 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.66 vs. limit=10.0
2023-11-22 20:44:07,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.11 vs. limit=15.0
2023-11-22 20:44:08,378 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.42 vs. limit=10.0
2023-11-22 20:44:14,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2091913.3333333333, ans=0.0
2023-11-22 20:44:17,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2091913.3333333333, ans=0.125
2023-11-22 20:44:23,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313800
2023-11-22 20:44:26,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2091980.0, ans=0.125
2023-11-22 20:44:31,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2091980.0, ans=0.1
2023-11-22 20:44:33,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2092046.6666666667, ans=0.0
2023-11-22 20:44:42,916 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.13 vs. limit=15.0
2023-11-22 20:44:49,080 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1200, loss[loss=0.07189, simple_loss=0.09052, pruned_loss=0.01494, audio_tagging_loss=0.01169, over 14966.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09305, pruned_loss=0.01454, audio_tagging_loss=0.009088, over 3043599.39 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:44:58,685 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.33 vs. limit=15.0
2023-11-22 20:44:59,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2092113.3333333333, ans=0.125
2023-11-22 20:45:01,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2092180.0, ans=0.0
2023-11-22 20:45:05,806 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.27 vs. limit=15.0
2023-11-22 20:45:07,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2092180.0, ans=0.125
2023-11-22 20:45:19,786 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.331e+01 9.096e+01 9.827e+01 1.345e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-22 20:45:22,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2092246.6666666667, ans=0.05
2023-11-22 20:45:26,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2092313.3333333333, ans=0.125
2023-11-22 20:45:27,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313850
2023-11-22 20:45:30,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2092313.3333333333, ans=0.125
2023-11-22 20:45:35,705 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.32 vs. limit=15.0
2023-11-22 20:45:38,002 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.04 vs. limit=15.0
2023-11-22 20:45:52,589 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1250, loss[loss=0.07036, simple_loss=0.09067, pruned_loss=0.01566, audio_tagging_loss=0.00937, over 14885.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09287, pruned_loss=0.01473, audio_tagging_loss=0.009097, over 3037277.49 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:45:54,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2092446.6666666667, ans=0.2
2023-11-22 20:46:06,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2092513.3333333333, ans=0.0
2023-11-22 20:46:32,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313900
2023-11-22 20:46:56,870 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1300, loss[loss=0.08997, simple_loss=0.1184, pruned_loss=0.01932, audio_tagging_loss=0.01145, over 15759.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09284, pruned_loss=0.01485, audio_tagging_loss=0.009175, over 3039603.04 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:47:06,363 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.45 vs. limit=6.0
2023-11-22 20:47:19,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2092846.6666666667, ans=0.0
2023-11-22 20:47:29,475 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.264e+01 8.633e+01 9.270e+01 1.307e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-22 20:47:29,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2092913.3333333333, ans=0.0
2023-11-22 20:47:32,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2092913.3333333333, ans=0.2
2023-11-22 20:47:35,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2092980.0, ans=0.0
2023-11-22 20:47:36,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 313950
2023-11-22 20:47:51,018 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-22 20:47:57,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2093046.6666666667, ans=0.1
2023-11-22 20:47:58,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2093046.6666666667, ans=0.0
2023-11-22 20:48:01,418 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1350, loss[loss=0.08697, simple_loss=0.1152, pruned_loss=0.01929, audio_tagging_loss=0.01006, over 15275.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09265, pruned_loss=0.01466, audio_tagging_loss=0.009156, over 3040190.39 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:48:06,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2093113.3333333333, ans=0.1
2023-11-22 20:48:22,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2093180.0, ans=0.2
2023-11-22 20:48:38,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2093313.3333333333, ans=0.0
2023-11-22 20:48:40,401 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314000
2023-11-22 20:48:47,389 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:49:00,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2093380.0, ans=0.125
2023-11-22 20:49:05,277 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1400, loss[loss=0.08137, simple_loss=0.1003, pruned_loss=0.02018, audio_tagging_loss=0.01106, over 15355.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09343, pruned_loss=0.01463, audio_tagging_loss=0.00928, over 3041962.64 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:49:19,888 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.85 vs. limit=15.0
2023-11-22 20:49:25,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2093513.3333333333, ans=0.0
2023-11-22 20:49:33,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2093580.0, ans=0.125
2023-11-22 20:49:38,500 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.151e+01 8.829e+01 9.680e+01 1.142e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 20:49:44,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314050
2023-11-22 20:50:06,958 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=12.0
2023-11-22 20:50:08,774 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1450, loss[loss=0.07811, simple_loss=0.1066, pruned_loss=0.01727, audio_tagging_loss=0.007535, over 14699.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09426, pruned_loss=0.01465, audio_tagging_loss=0.009333, over 3045226.15 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:50:16,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2093780.0, ans=0.2
2023-11-22 20:50:29,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2093846.6666666667, ans=0.0
2023-11-22 20:50:34,445 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.55 vs. limit=22.5
2023-11-22 20:50:49,270 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314100
2023-11-22 20:50:54,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2093980.0, ans=0.035
2023-11-22 20:51:11,990 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.81 vs. limit=15.0
2023-11-22 20:51:12,969 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1500, loss[loss=0.07706, simple_loss=0.1086, pruned_loss=0.01382, audio_tagging_loss=0.008939, over 15403.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09484, pruned_loss=0.01472, audio_tagging_loss=0.009387, over 3041529.71 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:51:22,901 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.28 vs. limit=10.0
2023-11-22 20:51:31,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2094180.0, ans=0.1
2023-11-22 20:51:40,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.whiten.whitening_limit, batch_count=2094246.6666666667, ans=12.0
2023-11-22 20:51:46,474 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.220e+01 8.857e+01 9.629e+01 1.228e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 20:51:47,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2094246.6666666667, ans=0.5
2023-11-22 20:51:52,742 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314150
2023-11-22 20:52:04,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2094380.0, ans=15.0
2023-11-22 20:52:12,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2094380.0, ans=0.05
2023-11-22 20:52:17,414 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1550, loss[loss=0.08966, simple_loss=0.1215, pruned_loss=0.02066, audio_tagging_loss=0.008249, over 15290.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09542, pruned_loss=0.01493, audio_tagging_loss=0.009341, over 3039219.23 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:52:29,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2094513.3333333333, ans=0.0
2023-11-22 20:52:46,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2094580.0, ans=0.1
2023-11-22 20:52:49,709 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.04 vs. limit=15.0
2023-11-22 20:52:51,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2094580.0, ans=0.125
2023-11-22 20:52:53,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2094580.0, ans=0.125
2023-11-22 20:52:55,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2094646.6666666667, ans=0.125
2023-11-22 20:52:57,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314200
2023-11-22 20:53:04,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2094646.6666666667, ans=0.0
2023-11-22 20:53:05,419 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:53:09,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2094713.3333333333, ans=0.125
2023-11-22 20:53:15,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2094713.3333333333, ans=0.1
2023-11-22 20:53:21,774 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1600, loss[loss=0.07691, simple_loss=0.1069, pruned_loss=0.01366, audio_tagging_loss=0.009804, over 15749.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09584, pruned_loss=0.01508, audio_tagging_loss=0.00936, over 3043097.61 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:53:38,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2094846.6666666667, ans=0.0
2023-11-22 20:53:49,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2094913.3333333333, ans=0.125
2023-11-22 20:53:54,980 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.130e+01 8.806e+01 9.580e+01 1.162e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 20:53:59,120 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:54:01,162 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314250
2023-11-22 20:54:14,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2095046.6666666667, ans=0.125
2023-11-22 20:54:14,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2095046.6666666667, ans=0.2
2023-11-22 20:54:25,234 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1650, loss[loss=0.06616, simple_loss=0.08127, pruned_loss=0.01549, audio_tagging_loss=0.01003, over 15333.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09552, pruned_loss=0.01486, audio_tagging_loss=0.009394, over 3046322.34 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:54:30,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2095113.3333333333, ans=0.0
2023-11-22 20:55:02,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2095246.6666666667, ans=0.125
2023-11-22 20:55:05,967 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314300
2023-11-22 20:55:14,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2095313.3333333333, ans=0.1
2023-11-22 20:55:30,011 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1700, loss[loss=0.06697, simple_loss=0.08855, pruned_loss=0.01361, audio_tagging_loss=0.009085, over 13971.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09509, pruned_loss=0.01485, audio_tagging_loss=0.009455, over 3050381.46 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:55:31,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2095446.6666666667, ans=0.125
2023-11-22 20:55:38,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2095446.6666666667, ans=0.2
2023-11-22 20:55:55,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2095580.0, ans=0.125
2023-11-22 20:55:59,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2095580.0, ans=0.1
2023-11-22 20:56:02,456 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.152e+01 8.274e+01 8.793e+01 9.384e+01 1.139e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 20:56:04,575 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.20 vs. limit=15.0
2023-11-22 20:56:10,147 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314350
2023-11-22 20:56:16,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2095646.6666666667, ans=0.1
2023-11-22 20:56:19,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2095646.6666666667, ans=0.125
2023-11-22 20:56:33,706 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1750, loss[loss=0.0612, simple_loss=0.08308, pruned_loss=0.01049, audio_tagging_loss=0.009171, over 14666.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.0941, pruned_loss=0.01465, audio_tagging_loss=0.009445, over 3046092.79 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:56:49,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2095846.6666666667, ans=0.125
2023-11-22 20:56:52,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2095846.6666666667, ans=0.0
2023-11-22 20:56:55,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2095846.6666666667, ans=0.5
2023-11-22 20:56:55,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2095846.6666666667, ans=0.2
2023-11-22 20:57:04,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2095913.3333333333, ans=0.125
2023-11-22 20:57:08,455 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.29 vs. limit=22.5
2023-11-22 20:57:14,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314400
2023-11-22 20:57:15,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2095980.0, ans=0.2
2023-11-22 20:57:38,157 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1800, loss[loss=0.08118, simple_loss=0.1117, pruned_loss=0.01468, audio_tagging_loss=0.01064, over 16077.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09436, pruned_loss=0.0147, audio_tagging_loss=0.009281, over 3042015.47 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:57:38,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2096113.3333333333, ans=0.95
2023-11-22 20:57:54,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2096180.0, ans=0.125
2023-11-22 20:58:11,555 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 8.025e+01 8.684e+01 9.480e+01 1.547e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 20:58:17,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314450
2023-11-22 20:58:22,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2096313.3333333333, ans=0.2
2023-11-22 20:58:31,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2096380.0, ans=0.125
2023-11-22 20:58:42,927 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1850, loss[loss=0.06226, simple_loss=0.0831, pruned_loss=0.01238, audio_tagging_loss=0.008333, over 15280.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09341, pruned_loss=0.01452, audio_tagging_loss=0.009259, over 3039055.76 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:58:49,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.84 vs. limit=22.5
2023-11-22 20:58:50,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2096446.6666666667, ans=0.0
2023-11-22 20:59:14,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2096580.0, ans=0.5
2023-11-22 20:59:21,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2096646.6666666667, ans=0.2
2023-11-22 20:59:22,189 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.84 vs. limit=22.5
2023-11-22 20:59:22,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314500
2023-11-22 20:59:42,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2096713.3333333333, ans=0.1
2023-11-22 20:59:46,110 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1900, loss[loss=0.06618, simple_loss=0.0853, pruned_loss=0.01271, audio_tagging_loss=0.01082, over 15511.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09216, pruned_loss=0.01439, audio_tagging_loss=0.009294, over 3039171.55 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:00:15,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2096913.3333333333, ans=0.125
2023-11-22 21:00:19,874 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 7.994e+01 8.463e+01 9.067e+01 1.382e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-22 21:00:26,039 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314550
2023-11-22 21:00:27,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2096980.0, ans=0.1
2023-11-22 21:00:39,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2097046.6666666667, ans=0.0
2023-11-22 21:00:44,405 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:00:49,514 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 1950, loss[loss=0.07006, simple_loss=0.08956, pruned_loss=0.01424, audio_tagging_loss=0.01105, over 14866.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09149, pruned_loss=0.01425, audio_tagging_loss=0.009353, over 3038696.16 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:00:52,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2097113.3333333333, ans=0.1
2023-11-22 21:01:25,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2097246.6666666665, ans=0.125
2023-11-22 21:01:28,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314600
2023-11-22 21:01:32,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2097313.3333333335, ans=0.1
2023-11-22 21:01:52,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2097380.0, ans=0.2
2023-11-22 21:01:54,204 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2000, loss[loss=0.05605, simple_loss=0.07559, pruned_loss=0.008263, audio_tagging_loss=0.00999, over 14555.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09099, pruned_loss=0.01425, audio_tagging_loss=0.009466, over 3035457.33 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:02:03,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2097446.6666666665, ans=10.0
2023-11-22 21:02:06,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2097513.3333333335, ans=0.0
2023-11-22 21:02:09,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2097513.3333333335, ans=0.0
2023-11-22 21:02:15,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2097513.3333333335, ans=0.125
2023-11-22 21:02:26,126 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.310e+01 8.293e+01 8.864e+01 9.588e+01 1.359e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 21:02:27,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2097580.0, ans=0.125
2023-11-22 21:02:32,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314650
2023-11-22 21:02:38,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2097646.6666666665, ans=0.1
2023-11-22 21:02:42,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2097646.6666666665, ans=0.125
2023-11-22 21:02:47,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2097713.3333333335, ans=0.2
2023-11-22 21:02:53,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2097713.3333333335, ans=0.0
2023-11-22 21:02:57,057 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2050, loss[loss=0.09535, simple_loss=0.122, pruned_loss=0.02437, audio_tagging_loss=0.009974, over 14572.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09204, pruned_loss=0.01442, audio_tagging_loss=0.009417, over 3034340.93 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:02:58,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2097780.0, ans=0.125
2023-11-22 21:03:05,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2097780.0, ans=0.1
2023-11-22 21:03:13,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2097846.6666666665, ans=0.0
2023-11-22 21:03:33,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2097913.3333333335, ans=0.1
2023-11-22 21:03:34,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.41 vs. limit=22.5
2023-11-22 21:03:36,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2097980.0, ans=0.0
2023-11-22 21:03:37,196 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314700
2023-11-22 21:03:44,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2097980.0, ans=0.125
2023-11-22 21:03:50,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2098046.6666666665, ans=0.125
2023-11-22 21:03:55,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2098046.6666666665, ans=0.125
2023-11-22 21:04:00,567 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2100, loss[loss=0.05302, simple_loss=0.06755, pruned_loss=0.008343, audio_tagging_loss=0.0109, over 15657.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09238, pruned_loss=0.01437, audio_tagging_loss=0.009333, over 3037849.35 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:04:10,836 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.58 vs. limit=15.0
2023-11-22 21:04:30,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2098246.6666666665, ans=10.0
2023-11-22 21:04:34,484 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.684e+01 9.132e+01 9.760e+01 1.245e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-22 21:04:40,794 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314750
2023-11-22 21:04:44,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2098313.3333333335, ans=0.0
2023-11-22 21:04:48,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2098313.3333333335, ans=0.125
2023-11-22 21:04:53,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2098380.0, ans=0.2
2023-11-22 21:05:02,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2098380.0, ans=0.125
2023-11-22 21:05:05,787 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2150, loss[loss=0.094, simple_loss=0.1321, pruned_loss=0.02241, audio_tagging_loss=0.005545, over 15525.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09332, pruned_loss=0.01449, audio_tagging_loss=0.009285, over 3039731.56 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:05:25,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2098513.3333333335, ans=0.125
2023-11-22 21:05:40,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2098580.0, ans=0.125
2023-11-22 21:05:42,523 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:05:43,778 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314800
2023-11-22 21:05:53,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2098646.6666666665, ans=0.2
2023-11-22 21:06:08,667 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2200, loss[loss=0.07299, simple_loss=0.09912, pruned_loss=0.01342, audio_tagging_loss=0.01001, over 14751.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.0936, pruned_loss=0.01462, audio_tagging_loss=0.009236, over 3040256.97 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:06:16,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2098780.0, ans=0.0
2023-11-22 21:06:28,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2098846.6666666665, ans=0.2
2023-11-22 21:06:42,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.590e+01 8.202e+01 8.834e+01 9.532e+01 1.382e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 21:06:48,393 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314850
2023-11-22 21:06:52,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1.whitening_limit, batch_count=2098980.0, ans=10.0
2023-11-22 21:07:04,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2099046.6666666665, ans=0.125
2023-11-22 21:07:09,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2099046.6666666665, ans=0.5
2023-11-22 21:07:11,832 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2250, loss[loss=0.0583, simple_loss=0.07352, pruned_loss=0.0124, audio_tagging_loss=0.00915, over 16163.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09335, pruned_loss=0.01464, audio_tagging_loss=0.009235, over 3038149.38 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:07:13,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2099113.3333333335, ans=0.05
2023-11-22 21:07:18,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2099113.3333333335, ans=0.125
2023-11-22 21:07:25,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2099180.0, ans=0.125
2023-11-22 21:07:51,677 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314900
2023-11-22 21:08:16,695 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2300, loss[loss=0.06471, simple_loss=0.08586, pruned_loss=0.01203, audio_tagging_loss=0.009749, over 15455.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09372, pruned_loss=0.01468, audio_tagging_loss=0.009383, over 3042866.52 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:08:18,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2099446.6666666665, ans=0.0
2023-11-22 21:08:23,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2099446.6666666665, ans=0.2
2023-11-22 21:08:24,214 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.38 vs. limit=15.0
2023-11-22 21:08:33,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2099513.3333333335, ans=0.125
2023-11-22 21:08:47,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2099580.0, ans=0.0
2023-11-22 21:08:50,528 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.386e+01 8.086e+01 8.698e+01 9.161e+01 1.218e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 21:08:52,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2099580.0, ans=0.125
2023-11-22 21:08:55,683 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 314950
2023-11-22 21:09:01,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2099646.6666666665, ans=0.0
2023-11-22 21:09:06,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.24 vs. limit=10.0
2023-11-22 21:09:13,521 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:09:17,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2099713.3333333335, ans=0.1
2023-11-22 21:09:19,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2099780.0, ans=0.1
2023-11-22 21:09:20,820 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2350, loss[loss=0.05227, simple_loss=0.06185, pruned_loss=0.009376, audio_tagging_loss=0.01197, over 14585.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.0942, pruned_loss=0.01484, audio_tagging_loss=0.009352, over 3043864.35 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:09:44,616 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.33 vs. limit=10.0
2023-11-22 21:10:00,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315000
2023-11-22 21:10:16,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2100046.6666666665, ans=0.0
2023-11-22 21:10:24,666 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2400, loss[loss=0.07786, simple_loss=0.1039, pruned_loss=0.01743, audio_tagging_loss=0.008491, over 14974.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09371, pruned_loss=0.01477, audio_tagging_loss=0.009425, over 3039739.29 frames. ], batch size: 53, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:10:52,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2100246.6666666665, ans=0.0
2023-11-22 21:11:00,962 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.260e+01 8.417e+01 9.057e+01 9.650e+01 1.239e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 21:11:04,805 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315050
2023-11-22 21:11:28,999 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2450, loss[loss=0.06703, simple_loss=0.08458, pruned_loss=0.01486, audio_tagging_loss=0.00988, over 15095.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09378, pruned_loss=0.01475, audio_tagging_loss=0.009417, over 3049889.89 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:11:32,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2100446.6666666665, ans=0.0
2023-11-22 21:11:36,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2100446.6666666665, ans=0.125
2023-11-22 21:11:39,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2100446.6666666665, ans=0.2
2023-11-22 21:11:51,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2100513.3333333335, ans=0.0
2023-11-22 21:12:08,051 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315100
2023-11-22 21:12:19,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2100713.3333333335, ans=0.1
2023-11-22 21:12:20,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2100713.3333333335, ans=0.04949747468305833
2023-11-22 21:12:33,348 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2500, loss[loss=0.07467, simple_loss=0.09182, pruned_loss=0.01953, audio_tagging_loss=0.009227, over 15587.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09367, pruned_loss=0.01474, audio_tagging_loss=0.009426, over 3049049.23 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:12:49,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2100846.6666666665, ans=0.125
2023-11-22 21:13:08,897 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.214e+01 8.930e+01 9.863e+01 1.228e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 21:13:13,298 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315150
2023-11-22 21:13:14,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2100980.0, ans=0.125
2023-11-22 21:13:36,981 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2550, loss[loss=0.05997, simple_loss=0.08491, pruned_loss=0.009062, audio_tagging_loss=0.008454, over 15523.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09322, pruned_loss=0.01461, audio_tagging_loss=0.009404, over 3051790.38 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:14:17,279 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315200
2023-11-22 21:14:40,765 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2600, loss[loss=0.06358, simple_loss=0.08286, pruned_loss=0.01043, audio_tagging_loss=0.01172, over 13390.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09303, pruned_loss=0.01458, audio_tagging_loss=0.009286, over 3043458.50 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:15:11,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2101580.0, ans=0.1
2023-11-22 21:15:17,707 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 5.989e+01 8.300e+01 8.753e+01 9.403e+01 1.364e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 21:15:21,573 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315250
2023-11-22 21:15:24,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2101646.6666666665, ans=0.0
2023-11-22 21:15:24,245 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:15:33,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2101713.3333333335, ans=0.125
2023-11-22 21:15:34,134 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-22 21:15:43,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2101713.3333333335, ans=0.0
2023-11-22 21:15:45,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2101780.0, ans=0.125
2023-11-22 21:15:46,430 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2650, loss[loss=0.07023, simple_loss=0.1092, pruned_loss=0.01175, audio_tagging_loss=0.003867, over 14844.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09303, pruned_loss=0.01459, audio_tagging_loss=0.00913, over 3044792.38 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:15:54,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.55 vs. limit=15.0
2023-11-22 21:16:03,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2101846.6666666665, ans=0.2
2023-11-22 21:16:17,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2101913.3333333335, ans=0.035
2023-11-22 21:16:25,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2101980.0, ans=0.125
2023-11-22 21:16:26,918 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315300
2023-11-22 21:16:30,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2101980.0, ans=0.04949747468305833
2023-11-22 21:16:42,098 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.39 vs. limit=15.0
2023-11-22 21:16:47,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2102046.6666666665, ans=0.125
2023-11-22 21:16:50,640 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2700, loss[loss=0.06716, simple_loss=0.08821, pruned_loss=0.009417, audio_tagging_loss=0.01364, over 15877.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09236, pruned_loss=0.01445, audio_tagging_loss=0.009162, over 3045019.45 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:16:53,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2102113.3333333335, ans=0.125
2023-11-22 21:17:17,614 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.91 vs. limit=15.0
2023-11-22 21:17:26,861 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.486e+01 8.121e+01 8.686e+01 9.401e+01 1.913e+02, threshold=1.737e+02, percent-clipped=1.0
2023-11-22 21:17:30,742 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315350
2023-11-22 21:17:36,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2102313.3333333335, ans=0.1
2023-11-22 21:17:54,227 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2750, loss[loss=0.05828, simple_loss=0.07283, pruned_loss=0.01149, audio_tagging_loss=0.01037, over 15706.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09193, pruned_loss=0.01442, audio_tagging_loss=0.009235, over 3046128.84 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:18:26,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2102580.0, ans=0.1
2023-11-22 21:18:27,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.25 vs. limit=15.0
2023-11-22 21:18:28,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2102580.0, ans=0.0
2023-11-22 21:18:34,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315400
2023-11-22 21:18:43,189 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.63 vs. limit=6.0
2023-11-22 21:18:50,950 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:19:00,005 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2800, loss[loss=0.08445, simple_loss=0.1196, pruned_loss=0.01841, audio_tagging_loss=0.006259, over 15302.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09198, pruned_loss=0.01443, audio_tagging_loss=0.00918, over 3046842.70 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:19:30,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2102913.3333333335, ans=0.125
2023-11-22 21:19:36,993 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.680e+01 8.174e+01 8.854e+01 9.674e+01 2.365e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-22 21:19:40,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315450
2023-11-22 21:19:42,909 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-22 21:20:03,932 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2850, loss[loss=0.07023, simple_loss=0.1001, pruned_loss=0.01318, audio_tagging_loss=0.006982, over 16603.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09334, pruned_loss=0.01465, audio_tagging_loss=0.009177, over 3045209.36 frames. ], batch size: 62, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:20:14,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2103113.3333333335, ans=0.0
2023-11-22 21:20:38,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2103246.6666666665, ans=0.125
2023-11-22 21:20:42,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2103313.3333333335, ans=0.125
2023-11-22 21:20:44,682 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315500
2023-11-22 21:21:07,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2103446.6666666665, ans=0.2
2023-11-22 21:21:08,288 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2900, loss[loss=0.07659, simple_loss=0.1014, pruned_loss=0.01742, audio_tagging_loss=0.008467, over 13809.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.0936, pruned_loss=0.01474, audio_tagging_loss=0.009122, over 3037804.19 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:21:13,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2103446.6666666665, ans=0.125
2023-11-22 21:21:44,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2103580.0, ans=0.125
2023-11-22 21:21:45,630 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.319e+01 9.006e+01 9.775e+01 1.156e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 21:21:48,832 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315550
2023-11-22 21:21:49,489 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.83 vs. limit=15.0
2023-11-22 21:21:50,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2103646.6666666665, ans=0.125
2023-11-22 21:22:00,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2103713.3333333335, ans=0.125
2023-11-22 21:22:08,766 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:22:13,401 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 2950, loss[loss=0.07175, simple_loss=0.0938, pruned_loss=0.0173, audio_tagging_loss=0.007549, over 15493.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09282, pruned_loss=0.01453, audio_tagging_loss=0.009185, over 3038603.89 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:22:14,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2103780.0, ans=0.125
2023-11-22 21:22:33,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2103846.6666666665, ans=0.125
2023-11-22 21:22:33,621 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.91 vs. limit=15.0
2023-11-22 21:22:52,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315600
2023-11-22 21:23:17,769 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3000, loss[loss=0.06594, simple_loss=0.09928, pruned_loss=0.00954, audio_tagging_loss=0.00676, over 14741.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09329, pruned_loss=0.01448, audio_tagging_loss=0.009135, over 3039078.57 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:23:17,770 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 21:23:37,333 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.8613, 2.2191, 3.2254, 2.5333], device='cuda:2')
2023-11-22 21:23:59,589 INFO [train_asr.py:1253] (2/4) Epoch 27, validation: loss=0.058, simple_loss=0.05133, pruned_loss=0.005079, audio_tagging_loss=0.02726, over 4681554.00 frames. 
2023-11-22 21:23:59,590 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 21:24:09,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2104113.3333333335, ans=0.125
2023-11-22 21:24:09,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2104113.3333333335, ans=0.125
2023-11-22 21:24:11,919 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.64 vs. limit=15.0
2023-11-22 21:24:12,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2104180.0, ans=0.0
2023-11-22 21:24:17,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2104180.0, ans=0.2
2023-11-22 21:24:35,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.21 vs. limit=15.0
2023-11-22 21:24:36,799 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.281e+01 8.900e+01 9.558e+01 2.196e+02, threshold=1.780e+02, percent-clipped=1.0
2023-11-22 21:24:39,378 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315650
2023-11-22 21:24:54,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2104380.0, ans=0.1
2023-11-22 21:24:56,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2104380.0, ans=0.2
2023-11-22 21:25:04,721 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3050, loss[loss=0.04751, simple_loss=0.05819, pruned_loss=0.007595, audio_tagging_loss=0.01083, over 15153.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09342, pruned_loss=0.01452, audio_tagging_loss=0.009269, over 3048298.96 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:25:09,099 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.29 vs. limit=15.0
2023-11-22 21:25:15,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.44 vs. limit=6.0
2023-11-22 21:25:23,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2104513.3333333335, ans=0.125
2023-11-22 21:25:35,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2104580.0, ans=0.1
2023-11-22 21:25:41,338 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:25:43,934 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315700
2023-11-22 21:25:46,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2104646.6666666665, ans=0.1
2023-11-22 21:26:07,724 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3100, loss[loss=0.08279, simple_loss=0.1133, pruned_loss=0.01794, audio_tagging_loss=0.008203, over 15623.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09487, pruned_loss=0.01483, audio_tagging_loss=0.009196, over 3044665.88 frames. ], batch size: 61, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:26:18,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2104846.6666666665, ans=0.1
2023-11-22 21:26:23,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2104846.6666666665, ans=0.125
2023-11-22 21:26:40,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2104913.3333333335, ans=10.0
2023-11-22 21:26:44,230 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.175e+01 8.886e+01 9.764e+01 1.279e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 21:26:45,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.63 vs. limit=15.0
2023-11-22 21:26:46,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315750
2023-11-22 21:27:01,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2105046.6666666665, ans=0.0
2023-11-22 21:27:09,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.18 vs. limit=15.0
2023-11-22 21:27:09,962 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3150, loss[loss=0.08955, simple_loss=0.1276, pruned_loss=0.01768, audio_tagging_loss=0.008075, over 15235.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09502, pruned_loss=0.01474, audio_tagging_loss=0.009304, over 3042721.25 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:27:29,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.92 vs. limit=15.0
2023-11-22 21:27:50,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315800
2023-11-22 21:27:51,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2105313.3333333335, ans=0.125
2023-11-22 21:27:51,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=12.0
2023-11-22 21:28:01,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2105380.0, ans=0.125
2023-11-22 21:28:16,053 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3200, loss[loss=0.07369, simple_loss=0.09277, pruned_loss=0.01681, audio_tagging_loss=0.01049, over 15363.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09443, pruned_loss=0.01474, audio_tagging_loss=0.009395, over 3039824.83 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:28:21,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2105446.6666666665, ans=0.125
2023-11-22 21:28:31,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2105513.3333333335, ans=0.125
2023-11-22 21:28:32,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2105513.3333333335, ans=0.1
2023-11-22 21:28:38,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2105513.3333333335, ans=0.0
2023-11-22 21:28:44,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2105580.0, ans=0.1
2023-11-22 21:28:53,852 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.364e+01 8.971e+01 9.873e+01 1.649e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 21:28:55,861 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315850
2023-11-22 21:29:01,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2105646.6666666665, ans=0.2
2023-11-22 21:29:20,013 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3250, loss[loss=0.07642, simple_loss=0.1015, pruned_loss=0.0166, audio_tagging_loss=0.009072, over 15758.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09366, pruned_loss=0.01459, audio_tagging_loss=0.009388, over 3033653.42 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:29:37,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2105846.6666666665, ans=0.0
2023-11-22 21:29:54,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2105913.3333333335, ans=0.125
2023-11-22 21:29:56,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.10 vs. limit=15.0
2023-11-22 21:29:56,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.20 vs. limit=8.0
2023-11-22 21:30:00,781 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315900
2023-11-22 21:30:08,448 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=5.028e-03
2023-11-22 21:30:13,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2106046.6666666665, ans=0.07
2023-11-22 21:30:19,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2106046.6666666665, ans=0.2
2023-11-22 21:30:21,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2106046.6666666665, ans=0.2
2023-11-22 21:30:24,074 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3300, loss[loss=0.06891, simple_loss=0.08102, pruned_loss=0.01673, audio_tagging_loss=0.01167, over 15322.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09248, pruned_loss=0.01449, audio_tagging_loss=0.009649, over 3034724.51 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:30:24,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2106113.3333333335, ans=0.125
2023-11-22 21:30:37,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2106180.0, ans=0.07
2023-11-22 21:30:39,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2106180.0, ans=0.1
2023-11-22 21:30:56,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2106246.6666666665, ans=0.125
2023-11-22 21:31:02,504 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.403e+01 9.072e+01 9.867e+01 1.352e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-22 21:31:03,947 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 315950
2023-11-22 21:31:08,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2106313.3333333335, ans=0.0
2023-11-22 21:31:26,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2106446.6666666665, ans=0.0
2023-11-22 21:31:27,784 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3350, loss[loss=0.0644, simple_loss=0.08763, pruned_loss=0.0151, audio_tagging_loss=0.005487, over 15075.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09197, pruned_loss=0.01441, audio_tagging_loss=0.009507, over 3030692.14 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:32:07,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316000
2023-11-22 21:32:18,945 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.46 vs. limit=15.0
2023-11-22 21:32:27,712 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=24.99 vs. limit=22.5
2023-11-22 21:32:35,461 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3400, loss[loss=0.09154, simple_loss=0.1111, pruned_loss=0.02636, audio_tagging_loss=0.009632, over 14940.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.0922, pruned_loss=0.01448, audio_tagging_loss=0.009465, over 3033961.27 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:32:44,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2106780.0, ans=0.0
2023-11-22 21:33:01,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.61 vs. limit=15.0
2023-11-22 21:33:03,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2106913.3333333335, ans=0.95
2023-11-22 21:33:04,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2106913.3333333335, ans=0.125
2023-11-22 21:33:06,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=12.0
2023-11-22 21:33:14,296 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.192e+01 8.214e+01 8.729e+01 9.303e+01 1.198e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 21:33:15,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316050
2023-11-22 21:33:18,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2106980.0, ans=0.125
2023-11-22 21:33:27,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2107046.6666666665, ans=0.125
2023-11-22 21:33:27,714 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.22 vs. limit=15.0
2023-11-22 21:33:39,068 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3450, loss[loss=0.06416, simple_loss=0.08362, pruned_loss=0.01415, audio_tagging_loss=0.00821, over 13802.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09221, pruned_loss=0.01447, audio_tagging_loss=0.009398, over 3034167.64 frames. ], batch size: 53, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:33:48,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2107113.3333333335, ans=0.125
2023-11-22 21:33:58,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2107180.0, ans=0.125
2023-11-22 21:34:18,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316100
2023-11-22 21:34:26,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2107313.3333333335, ans=0.2
2023-11-22 21:34:43,091 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3500, loss[loss=0.05075, simple_loss=0.0615, pruned_loss=0.009517, audio_tagging_loss=0.01049, over 15110.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09243, pruned_loss=0.01449, audio_tagging_loss=0.009237, over 3032785.87 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:35:16,219 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:35:20,933 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 8.106e+01 8.631e+01 9.458e+01 1.678e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 21:35:22,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316150
2023-11-22 21:35:29,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2107646.6666666665, ans=0.1
2023-11-22 21:35:30,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.71 vs. limit=15.0
2023-11-22 21:35:30,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2107646.6666666665, ans=0.125
2023-11-22 21:35:44,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2107713.3333333335, ans=0.125
2023-11-22 21:35:47,501 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3550, loss[loss=0.08181, simple_loss=0.1227, pruned_loss=0.01323, audio_tagging_loss=0.007223, over 16128.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09199, pruned_loss=0.01428, audio_tagging_loss=0.009189, over 3033536.43 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:35:48,246 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.64 vs. limit=15.0
2023-11-22 21:36:18,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2107913.3333333335, ans=0.2
2023-11-22 21:36:26,431 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:36:27,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316200
2023-11-22 21:36:32,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2107980.0, ans=0.05
2023-11-22 21:36:34,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2107980.0, ans=0.0
2023-11-22 21:36:51,777 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3600, loss[loss=0.09776, simple_loss=0.1394, pruned_loss=0.0201, audio_tagging_loss=0.007945, over 16097.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.0925, pruned_loss=0.01431, audio_tagging_loss=0.009125, over 3031589.10 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:37:00,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2108113.3333333335, ans=0.125
2023-11-22 21:37:07,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2108180.0, ans=0.0
2023-11-22 21:37:29,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2108313.3333333335, ans=0.0
2023-11-22 21:37:29,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2108313.3333333335, ans=0.1
2023-11-22 21:37:30,461 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.329e+01 8.258e+01 8.715e+01 9.542e+01 1.505e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 21:37:31,917 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316250
2023-11-22 21:37:37,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2108313.3333333335, ans=10.0
2023-11-22 21:37:43,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2108380.0, ans=0.0
2023-11-22 21:37:46,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2108380.0, ans=0.0
2023-11-22 21:37:56,468 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3650, loss[loss=0.09871, simple_loss=0.1332, pruned_loss=0.02391, audio_tagging_loss=0.008204, over 14615.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09311, pruned_loss=0.01451, audio_tagging_loss=0.009128, over 3034848.57 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:37:59,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2108446.6666666665, ans=0.0
2023-11-22 21:38:00,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2108446.6666666665, ans=0.2
2023-11-22 21:38:04,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2108446.6666666665, ans=0.125
2023-11-22 21:38:27,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2108580.0, ans=0.125
2023-11-22 21:38:35,920 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316300
2023-11-22 21:38:36,484 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.58 vs. limit=15.0
2023-11-22 21:38:38,843 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.58 vs. limit=22.5
2023-11-22 21:38:42,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2108646.6666666665, ans=0.0
2023-11-22 21:38:43,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2108646.6666666665, ans=0.125
2023-11-22 21:39:00,755 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3700, loss[loss=0.07549, simple_loss=0.1037, pruned_loss=0.01508, audio_tagging_loss=0.008568, over 16468.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09323, pruned_loss=0.0146, audio_tagging_loss=0.009064, over 3041230.94 frames. ], batch size: 61, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:39:06,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2108780.0, ans=0.0
2023-11-22 21:39:09,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2108780.0, ans=0.125
2023-11-22 21:39:24,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2108913.3333333335, ans=0.025
2023-11-22 21:39:40,495 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.949e+01 8.416e+01 9.143e+01 1.024e+02 1.510e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-22 21:39:41,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316350
2023-11-22 21:39:42,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2108980.0, ans=0.1
2023-11-22 21:39:50,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2108980.0, ans=0.1
2023-11-22 21:39:52,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2109046.6666666665, ans=0.0
2023-11-22 21:39:54,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2109046.6666666665, ans=0.0
2023-11-22 21:40:01,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2109046.6666666665, ans=0.0
2023-11-22 21:40:05,177 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3750, loss[loss=0.06711, simple_loss=0.08804, pruned_loss=0.01226, audio_tagging_loss=0.01083, over 15346.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09451, pruned_loss=0.0148, audio_tagging_loss=0.008971, over 3050223.91 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:40:05,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.68 vs. limit=15.0
2023-11-22 21:40:16,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2109180.0, ans=0.125
2023-11-22 21:40:32,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2109246.6666666665, ans=0.1
2023-11-22 21:40:45,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316400
2023-11-22 21:40:45,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2109313.3333333335, ans=0.125
2023-11-22 21:40:50,964 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:40:56,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2109380.0, ans=0.0
2023-11-22 21:41:03,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2109380.0, ans=0.125
2023-11-22 21:41:05,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2109380.0, ans=0.0
2023-11-22 21:41:05,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2109380.0, ans=0.2
2023-11-22 21:41:07,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2109380.0, ans=0.125
2023-11-22 21:41:10,534 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3800, loss[loss=0.06998, simple_loss=0.08976, pruned_loss=0.01607, audio_tagging_loss=0.009028, over 15278.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09461, pruned_loss=0.01479, audio_tagging_loss=0.009068, over 3044088.30 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:41:11,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.92 vs. limit=6.0
2023-11-22 21:41:12,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2109446.6666666665, ans=10.0
2023-11-22 21:41:22,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2109513.3333333335, ans=0.025
2023-11-22 21:41:32,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2109513.3333333335, ans=0.0
2023-11-22 21:41:37,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2109580.0, ans=0.125
2023-11-22 21:41:38,070 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=7.82 vs. limit=15.0
2023-11-22 21:41:45,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.33 vs. limit=22.5
2023-11-22 21:41:46,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2109580.0, ans=0.0
2023-11-22 21:41:50,083 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.525e+01 8.963e+01 9.873e+01 1.311e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-22 21:41:50,229 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316450
2023-11-22 21:41:52,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2109646.6666666665, ans=0.2
2023-11-22 21:41:58,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2109646.6666666665, ans=0.125
2023-11-22 21:42:14,508 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3850, loss[loss=0.07231, simple_loss=0.0931, pruned_loss=0.01662, audio_tagging_loss=0.009143, over 15868.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.0948, pruned_loss=0.01487, audio_tagging_loss=0.009149, over 3045524.12 frames. ], batch size: 61, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:42:19,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2109780.0, ans=0.125
2023-11-22 21:42:49,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2109913.3333333335, ans=0.125
2023-11-22 21:42:53,802 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316500
2023-11-22 21:42:54,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=14.13 vs. limit=15.0
2023-11-22 21:43:02,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.52 vs. limit=15.0
2023-11-22 21:43:06,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2110046.6666666665, ans=0.125
2023-11-22 21:43:17,560 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3900, loss[loss=0.07937, simple_loss=0.1063, pruned_loss=0.0179, audio_tagging_loss=0.008292, over 14588.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09403, pruned_loss=0.01466, audio_tagging_loss=0.009264, over 3047820.04 frames. ], batch size: 53, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:43:22,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2110113.3333333335, ans=0.125
2023-11-22 21:43:26,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2110113.3333333335, ans=0.125
2023-11-22 21:43:29,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2110180.0, ans=0.0
2023-11-22 21:43:37,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2110180.0, ans=0.125
2023-11-22 21:43:38,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2110180.0, ans=0.125
2023-11-22 21:43:48,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2110246.6666666665, ans=0.0
2023-11-22 21:43:57,193 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.364e+01 8.359e+01 8.893e+01 9.456e+01 1.423e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 21:43:57,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316550
2023-11-22 21:44:21,502 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 3950, loss[loss=0.0739, simple_loss=0.09083, pruned_loss=0.01673, audio_tagging_loss=0.01176, over 14613.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09293, pruned_loss=0.01451, audio_tagging_loss=0.009476, over 3042114.03 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:44:35,411 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.83 vs. limit=15.0
2023-11-22 21:44:38,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2110513.3333333335, ans=0.125
2023-11-22 21:44:48,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2110580.0, ans=0.125
2023-11-22 21:45:00,667 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316600
2023-11-22 21:45:07,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2110646.6666666665, ans=0.125
2023-11-22 21:45:25,352 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4000, loss[loss=0.06461, simple_loss=0.08529, pruned_loss=0.01024, audio_tagging_loss=0.01172, over 14383.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09314, pruned_loss=0.0147, audio_tagging_loss=0.009537, over 3046889.88 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:45:30,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.65 vs. limit=15.0
2023-11-22 21:45:46,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2110846.6666666665, ans=0.2
2023-11-22 21:45:51,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2110913.3333333335, ans=0.125
2023-11-22 21:46:04,310 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.309e+01 8.921e+01 9.654e+01 1.157e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 21:46:04,452 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316650
2023-11-22 21:46:06,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2110980.0, ans=0.1
2023-11-22 21:46:18,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.77 vs. limit=22.5
2023-11-22 21:46:26,281 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:46:27,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2111113.3333333335, ans=0.125
2023-11-22 21:46:28,464 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4050, loss[loss=0.04656, simple_loss=0.05082, pruned_loss=0.01205, audio_tagging_loss=0.009098, over 15516.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09355, pruned_loss=0.01476, audio_tagging_loss=0.009523, over 3051970.43 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:46:31,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2111113.3333333335, ans=0.125
2023-11-22 21:46:31,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2111113.3333333335, ans=0.2
2023-11-22 21:46:32,053 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:46:37,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2111113.3333333335, ans=0.0
2023-11-22 21:46:42,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2111180.0, ans=0.125
2023-11-22 21:46:57,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2111246.6666666665, ans=0.0
2023-11-22 21:47:03,719 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.21 vs. limit=15.0
2023-11-22 21:47:07,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2111313.3333333335, ans=0.0
2023-11-22 21:47:07,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316700
2023-11-22 21:47:16,040 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.53 vs. limit=15.0
2023-11-22 21:47:26,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2111380.0, ans=0.125
2023-11-22 21:47:26,846 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.70 vs. limit=10.0
2023-11-22 21:47:30,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2111446.6666666665, ans=0.0
2023-11-22 21:47:31,742 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4100, loss[loss=0.05933, simple_loss=0.08294, pruned_loss=0.01042, audio_tagging_loss=0.007445, over 15749.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.0941, pruned_loss=0.01494, audio_tagging_loss=0.009523, over 3054255.78 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:47:45,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2111513.3333333335, ans=0.125
2023-11-22 21:48:09,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2111646.6666666665, ans=0.0
2023-11-22 21:48:10,805 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.312e+01 9.101e+01 1.011e+02 1.374e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-22 21:48:10,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316750
2023-11-22 21:48:31,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2111713.3333333335, ans=0.1
2023-11-22 21:48:36,939 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4150, loss[loss=0.07896, simple_loss=0.1057, pruned_loss=0.01758, audio_tagging_loss=0.00856, over 13754.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09461, pruned_loss=0.01489, audio_tagging_loss=0.009349, over 3052656.80 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:48:38,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2111780.0, ans=0.125
2023-11-22 21:48:39,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2111780.0, ans=0.125
2023-11-22 21:48:49,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2111846.6666666665, ans=0.0
2023-11-22 21:48:57,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2111846.6666666665, ans=0.125
2023-11-22 21:48:58,178 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.88 vs. limit=15.0
2023-11-22 21:49:06,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.79 vs. limit=15.0
2023-11-22 21:49:15,888 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.38 vs. limit=15.0
2023-11-22 21:49:16,577 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316800
2023-11-22 21:49:18,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.13 vs. limit=22.5
2023-11-22 21:49:21,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.87 vs. limit=22.5
2023-11-22 21:49:23,119 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:49:32,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2112046.6666666665, ans=0.0
2023-11-22 21:49:32,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2112046.6666666665, ans=0.125
2023-11-22 21:49:40,869 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4200, loss[loss=0.05961, simple_loss=0.07989, pruned_loss=0.01099, audio_tagging_loss=0.008671, over 15402.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.09506, pruned_loss=0.01494, audio_tagging_loss=0.009178, over 3048777.73 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:49:48,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2112113.3333333335, ans=0.125
2023-11-22 21:49:49,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2112113.3333333335, ans=0.0
2023-11-22 21:50:04,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2112180.0, ans=0.0
2023-11-22 21:50:08,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.36 vs. limit=22.5
2023-11-22 21:50:09,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2112246.6666666665, ans=0.0
2023-11-22 21:50:19,501 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:50:20,404 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.143e+01 8.631e+01 9.392e+01 1.233e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 21:50:20,552 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316850
2023-11-22 21:50:29,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2112313.3333333335, ans=0.125
2023-11-22 21:50:34,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2112380.0, ans=0.1
2023-11-22 21:50:43,572 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4250, loss[loss=0.06706, simple_loss=0.1001, pruned_loss=0.009509, audio_tagging_loss=0.007498, over 14786.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09455, pruned_loss=0.01473, audio_tagging_loss=0.009158, over 3052901.81 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:51:01,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2112513.3333333335, ans=0.125
2023-11-22 21:51:05,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2112513.3333333335, ans=0.125
2023-11-22 21:51:10,431 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=12.0
2023-11-22 21:51:16,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2112580.0, ans=0.2
2023-11-22 21:51:17,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2112580.0, ans=0.125
2023-11-22 21:51:23,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316900
2023-11-22 21:51:32,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2112646.6666666665, ans=0.125
2023-11-22 21:51:47,698 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4300, loss[loss=0.08562, simple_loss=0.109, pruned_loss=0.02149, audio_tagging_loss=0.009642, over 15885.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09459, pruned_loss=0.01469, audio_tagging_loss=0.009118, over 3055323.17 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:51:48,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.78 vs. limit=15.0
2023-11-22 21:51:59,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2112846.6666666665, ans=0.2
2023-11-22 21:52:26,441 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.241e+01 8.941e+01 9.538e+01 1.182e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 21:52:26,578 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 316950
2023-11-22 21:52:29,430 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.64 vs. limit=12.0
2023-11-22 21:52:51,090 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4350, loss[loss=0.07324, simple_loss=0.1034, pruned_loss=0.01418, audio_tagging_loss=0.007356, over 15180.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09439, pruned_loss=0.01461, audio_tagging_loss=0.009065, over 3052135.78 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:53:03,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2113180.0, ans=0.2
2023-11-22 21:53:04,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2113180.0, ans=0.125
2023-11-22 21:53:10,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.84 vs. limit=15.0
2023-11-22 21:53:15,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2113246.6666666665, ans=0.125
2023-11-22 21:53:28,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2113246.6666666665, ans=0.07
2023-11-22 21:53:31,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317000
2023-11-22 21:53:33,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2113313.3333333335, ans=0.125
2023-11-22 21:53:39,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2113313.3333333335, ans=0.1
2023-11-22 21:53:54,887 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.63 vs. limit=15.0
2023-11-22 21:53:55,194 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4400, loss[loss=0.06626, simple_loss=0.08951, pruned_loss=0.01399, audio_tagging_loss=0.007507, over 14136.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.0936, pruned_loss=0.01442, audio_tagging_loss=0.009008, over 3049314.27 frames. ], batch size: 53, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:54:02,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2113446.6666666665, ans=0.125
2023-11-22 21:54:34,938 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.698e+01 8.507e+01 9.216e+01 9.885e+01 1.286e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-22 21:54:35,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317050
2023-11-22 21:54:59,471 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4450, loss[loss=0.07492, simple_loss=0.09256, pruned_loss=0.01903, audio_tagging_loss=0.009603, over 14590.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09454, pruned_loss=0.01486, audio_tagging_loss=0.008936, over 3047375.41 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:55:11,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2113846.6666666665, ans=0.0
2023-11-22 21:55:12,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2113846.6666666665, ans=0.0
2023-11-22 21:55:19,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2113846.6666666665, ans=0.95
2023-11-22 21:55:36,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2113980.0, ans=0.0
2023-11-22 21:55:39,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317100
2023-11-22 21:55:59,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2114046.6666666665, ans=0.0
2023-11-22 21:56:04,044 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4500, loss[loss=0.05731, simple_loss=0.07581, pruned_loss=0.009739, audio_tagging_loss=0.00967, over 16131.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.095, pruned_loss=0.01486, audio_tagging_loss=0.008941, over 3049018.62 frames. ], batch size: 61, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:56:13,724 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.66 vs. limit=15.0
2023-11-22 21:56:26,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2114180.0, ans=0.0
2023-11-22 21:56:44,917 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.228e+01 8.995e+01 9.708e+01 1.219e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 21:56:45,126 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317150
2023-11-22 21:57:07,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2114446.6666666665, ans=0.0
2023-11-22 21:57:08,309 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4550, loss[loss=0.0587, simple_loss=0.07088, pruned_loss=0.01271, audio_tagging_loss=0.01055, over 15137.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.0953, pruned_loss=0.015, audio_tagging_loss=0.008927, over 3047312.65 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:57:48,026 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.29 vs. limit=15.0
2023-11-22 21:57:48,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317200
2023-11-22 21:57:48,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2114646.6666666665, ans=0.1
2023-11-22 21:57:57,917 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:57:58,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2114646.6666666665, ans=0.0
2023-11-22 21:58:07,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=2114713.3333333335, ans=0.02
2023-11-22 21:58:10,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2114713.3333333335, ans=0.125
2023-11-22 21:58:13,854 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4600, loss[loss=0.07638, simple_loss=0.1012, pruned_loss=0.01838, audio_tagging_loss=0.007406, over 14638.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09482, pruned_loss=0.01486, audio_tagging_loss=0.009022, over 3045150.21 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:58:36,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2114846.6666666665, ans=0.125
2023-11-22 21:58:40,414 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.72 vs. limit=10.0
2023-11-22 21:58:53,245 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317250
2023-11-22 21:58:54,294 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.032e+01 8.605e+01 9.322e+01 1.270e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-22 21:59:03,549 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.73 vs. limit=22.5
2023-11-22 21:59:12,473 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.23 vs. limit=15.0
2023-11-22 21:59:18,573 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4650, loss[loss=0.08176, simple_loss=0.1152, pruned_loss=0.01884, audio_tagging_loss=0.005321, over 15586.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09445, pruned_loss=0.01489, audio_tagging_loss=0.009164, over 3047329.55 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:59:33,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2115180.0, ans=0.0
2023-11-22 21:59:43,175 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=12.0
2023-11-22 21:59:57,314 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317300
2023-11-22 22:00:21,171 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4700, loss[loss=0.09659, simple_loss=0.1248, pruned_loss=0.0242, audio_tagging_loss=0.01, over 15938.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09375, pruned_loss=0.01479, audio_tagging_loss=0.009361, over 3048922.35 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:00:21,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2115446.6666666665, ans=0.125
2023-11-22 22:00:23,055 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.16 vs. limit=15.0
2023-11-22 22:00:26,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2115446.6666666665, ans=0.1
2023-11-22 22:00:34,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2115513.3333333335, ans=0.125
2023-11-22 22:01:00,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317350
2023-11-22 22:01:01,992 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.187e+01 8.736e+01 9.527e+01 1.154e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 22:01:03,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2115646.6666666665, ans=0.125
2023-11-22 22:01:06,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2115646.6666666665, ans=0.0
2023-11-22 22:01:06,267 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.15 vs. limit=22.5
2023-11-22 22:01:08,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2115646.6666666665, ans=0.0
2023-11-22 22:01:19,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2115713.3333333335, ans=0.125
2023-11-22 22:01:21,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2115713.3333333335, ans=0.125
2023-11-22 22:01:25,027 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4750, loss[loss=0.07888, simple_loss=0.1009, pruned_loss=0.01869, audio_tagging_loss=0.009734, over 15427.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09341, pruned_loss=0.01471, audio_tagging_loss=0.009372, over 3035102.33 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:02:03,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317400
2023-11-22 22:02:26,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2116046.6666666665, ans=0.125
2023-11-22 22:02:29,388 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4800, loss[loss=0.0553, simple_loss=0.06869, pruned_loss=0.008227, audio_tagging_loss=0.01273, over 14819.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09337, pruned_loss=0.01475, audio_tagging_loss=0.009444, over 3042009.72 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 22:02:30,240 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.84 vs. limit=15.0
2023-11-22 22:03:07,657 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.35 vs. limit=15.0
2023-11-22 22:03:09,300 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317450
2023-11-22 22:03:10,367 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.616e+01 8.423e+01 9.043e+01 9.752e+01 1.234e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 22:03:16,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2116313.3333333335, ans=0.0
2023-11-22 22:03:21,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2116380.0, ans=0.0
2023-11-22 22:03:26,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2116380.0, ans=0.04949747468305833
2023-11-22 22:03:33,802 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4850, loss[loss=0.05296, simple_loss=0.0676, pruned_loss=0.007786, audio_tagging_loss=0.01137, over 15821.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09321, pruned_loss=0.01493, audio_tagging_loss=0.009471, over 3037868.46 frames. ], batch size: 61, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 22:03:43,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2116446.6666666665, ans=0.2
2023-11-22 22:04:13,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317500
2023-11-22 22:04:15,394 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.25 vs. limit=10.0
2023-11-22 22:04:25,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2116713.3333333335, ans=0.125
2023-11-22 22:04:29,639 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.36 vs. limit=22.5
2023-11-22 22:04:32,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.22 vs. limit=15.0
2023-11-22 22:04:35,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2116713.3333333335, ans=0.125
2023-11-22 22:04:38,101 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4900, loss[loss=0.06415, simple_loss=0.08309, pruned_loss=0.01189, audio_tagging_loss=0.01072, over 14826.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09319, pruned_loss=0.01497, audio_tagging_loss=0.009473, over 3037958.61 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:05:10,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2116913.3333333335, ans=0.0
2023-11-22 22:05:18,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317550
2023-11-22 22:05:20,625 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.184e+01 8.772e+01 9.580e+01 1.151e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 22:05:35,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2117046.6666666665, ans=0.125
2023-11-22 22:05:39,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2117046.6666666665, ans=0.125
2023-11-22 22:05:42,997 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 4950, loss[loss=0.05832, simple_loss=0.07299, pruned_loss=0.01026, audio_tagging_loss=0.01156, over 14963.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09299, pruned_loss=0.01474, audio_tagging_loss=0.00932, over 3028930.75 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:05:56,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.90 vs. limit=15.0
2023-11-22 22:06:03,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2117180.0, ans=0.0
2023-11-22 22:06:12,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2117246.6666666665, ans=0.0
2023-11-22 22:06:17,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2117246.6666666665, ans=0.125
2023-11-22 22:06:19,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2117246.6666666665, ans=0.125
2023-11-22 22:06:22,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317600
2023-11-22 22:06:30,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.41 vs. limit=6.0
2023-11-22 22:06:30,576 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.11 vs. limit=12.0
2023-11-22 22:06:46,925 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5000, loss[loss=0.07577, simple_loss=0.1058, pruned_loss=0.01217, audio_tagging_loss=0.01069, over 15901.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09362, pruned_loss=0.01469, audio_tagging_loss=0.009095, over 3037094.00 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:07:27,303 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317650
2023-11-22 22:07:29,597 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.065e+01 8.747e+01 9.520e+01 1.397e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 22:07:36,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2117646.6666666665, ans=0.0
2023-11-22 22:07:40,110 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.45 vs. limit=15.0
2023-11-22 22:07:49,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2117780.0, ans=0.125
2023-11-22 22:07:51,019 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5050, loss[loss=0.05856, simple_loss=0.07275, pruned_loss=0.01101, audio_tagging_loss=0.01117, over 14245.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09279, pruned_loss=0.01467, audio_tagging_loss=0.009126, over 3036181.69 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:08:06,896 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.37 vs. limit=22.5
2023-11-22 22:08:28,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2117980.0, ans=0.0
2023-11-22 22:08:31,696 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317700
2023-11-22 22:08:42,251 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:08:44,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2118046.6666666665, ans=0.1
2023-11-22 22:08:45,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2118046.6666666665, ans=0.125
2023-11-22 22:08:55,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2118113.3333333335, ans=0.125
2023-11-22 22:08:56,517 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5100, loss[loss=0.05287, simple_loss=0.07286, pruned_loss=0.007289, audio_tagging_loss=0.009154, over 15853.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09238, pruned_loss=0.01454, audio_tagging_loss=0.009182, over 3040584.36 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:09:01,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2118113.3333333335, ans=0.125
2023-11-22 22:09:06,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2118113.3333333335, ans=0.1
2023-11-22 22:09:10,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2118180.0, ans=0.1
2023-11-22 22:09:13,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2118180.0, ans=0.125
2023-11-22 22:09:15,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2118180.0, ans=0.125
2023-11-22 22:09:23,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2118246.6666666665, ans=0.125
2023-11-22 22:09:31,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2118246.6666666665, ans=0.1
2023-11-22 22:09:36,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317750
2023-11-22 22:09:38,769 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.041e+01 8.209e+01 8.781e+01 9.360e+01 1.201e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 22:09:55,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2118380.0, ans=0.125
2023-11-22 22:09:56,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2118380.0, ans=0.0
2023-11-22 22:09:57,096 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.70 vs. limit=22.5
2023-11-22 22:10:00,178 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5150, loss[loss=0.07809, simple_loss=0.1071, pruned_loss=0.01695, audio_tagging_loss=0.007594, over 14573.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09351, pruned_loss=0.01465, audio_tagging_loss=0.009106, over 3046045.03 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:10:04,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2118446.6666666665, ans=0.125
2023-11-22 22:10:21,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2118513.3333333335, ans=0.0
2023-11-22 22:10:40,788 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317800
2023-11-22 22:10:44,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.00 vs. limit=10.0
2023-11-22 22:10:48,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2118646.6666666665, ans=0.125
2023-11-22 22:10:50,996 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:10:58,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2118713.3333333335, ans=0.125
2023-11-22 22:11:04,849 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5200, loss[loss=0.08182, simple_loss=0.1171, pruned_loss=0.01862, audio_tagging_loss=0.004652, over 15175.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09416, pruned_loss=0.01477, audio_tagging_loss=0.009005, over 3051786.83 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:11:17,445 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.00 vs. limit=15.0
2023-11-22 22:11:31,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2118913.3333333335, ans=0.2
2023-11-22 22:11:44,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317850
2023-11-22 22:11:45,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2118980.0, ans=0.125
2023-11-22 22:11:46,663 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.652e+01 9.262e+01 9.962e+01 1.336e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-22 22:12:09,251 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5250, loss[loss=0.05229, simple_loss=0.05996, pruned_loss=0.009202, audio_tagging_loss=0.0131, over 14846.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09474, pruned_loss=0.01484, audio_tagging_loss=0.009028, over 3055547.03 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:12:16,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.58 vs. limit=15.0
2023-11-22 22:12:22,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2119180.0, ans=0.0
2023-11-22 22:12:27,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2119180.0, ans=0.125
2023-11-22 22:12:33,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2119246.6666666665, ans=0.2
2023-11-22 22:12:37,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2119246.6666666665, ans=0.125
2023-11-22 22:12:37,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2119246.6666666665, ans=0.0
2023-11-22 22:12:38,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2119246.6666666665, ans=0.125
2023-11-22 22:12:42,309 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=12.0
2023-11-22 22:12:48,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317900
2023-11-22 22:12:56,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2119313.3333333335, ans=0.125
2023-11-22 22:13:12,504 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5300, loss[loss=0.06477, simple_loss=0.0837, pruned_loss=0.0147, audio_tagging_loss=0.008215, over 16434.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09434, pruned_loss=0.01483, audio_tagging_loss=0.00906, over 3057627.76 frames. ], batch size: 62, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:13:17,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2119446.6666666665, ans=0.125
2023-11-22 22:13:53,074 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 317950
2023-11-22 22:13:56,549 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.353e+01 8.983e+01 9.482e+01 1.198e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 22:14:16,244 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5350, loss[loss=0.09644, simple_loss=0.1354, pruned_loss=0.02157, audio_tagging_loss=0.007151, over 15199.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09486, pruned_loss=0.01504, audio_tagging_loss=0.009097, over 3049234.87 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:14:22,938 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.07 vs. limit=15.0
2023-11-22 22:14:24,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2119780.0, ans=0.125
2023-11-22 22:14:34,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2119846.6666666665, ans=0.125
2023-11-22 22:14:54,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.93 vs. limit=10.0
2023-11-22 22:14:56,195 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318000
2023-11-22 22:15:01,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2119980.0, ans=0.125
2023-11-22 22:15:21,772 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5400, loss[loss=0.07579, simple_loss=0.09869, pruned_loss=0.0169, audio_tagging_loss=0.009546, over 15322.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09428, pruned_loss=0.0148, audio_tagging_loss=0.009211, over 3044267.57 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:15:28,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2120113.3333333335, ans=0.125
2023-11-22 22:15:33,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2120180.0, ans=0.0
2023-11-22 22:15:42,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2120180.0, ans=0.0
2023-11-22 22:16:00,389 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318050
2023-11-22 22:16:04,454 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.392e+01 8.997e+01 9.617e+01 1.276e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 22:16:13,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2120380.0, ans=0.125
2023-11-22 22:16:25,802 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5450, loss[loss=0.06275, simple_loss=0.09035, pruned_loss=0.00896, audio_tagging_loss=0.008615, over 13859.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09514, pruned_loss=0.01494, audio_tagging_loss=0.009192, over 3041637.04 frames. ], batch size: 52, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:16:41,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2120513.3333333335, ans=0.0
2023-11-22 22:16:52,191 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.56 vs. limit=22.5
2023-11-22 22:16:52,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2120580.0, ans=0.0
2023-11-22 22:17:06,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318100
2023-11-22 22:17:21,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.89 vs. limit=10.0
2023-11-22 22:17:28,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2120780.0, ans=0.125
2023-11-22 22:17:29,790 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5500, loss[loss=0.07128, simple_loss=0.09514, pruned_loss=0.01473, audio_tagging_loss=0.008981, over 14508.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09463, pruned_loss=0.01484, audio_tagging_loss=0.0092, over 3042299.03 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:17:30,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2120780.0, ans=0.125
2023-11-22 22:17:41,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2120846.6666666665, ans=0.0
2023-11-22 22:17:41,702 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.02 vs. limit=15.0
2023-11-22 22:17:59,213 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:18:02,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2120913.3333333335, ans=0.0
2023-11-22 22:18:02,742 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:18:09,911 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318150
2023-11-22 22:18:13,411 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.175e+01 8.992e+01 9.514e+01 1.142e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-22 22:18:15,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2120980.0, ans=0.0
2023-11-22 22:18:20,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2121046.6666666665, ans=0.125
2023-11-22 22:18:23,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2121046.6666666665, ans=0.125
2023-11-22 22:18:24,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2121046.6666666665, ans=0.1
2023-11-22 22:18:34,015 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5550, loss[loss=0.0793, simple_loss=0.1028, pruned_loss=0.01776, audio_tagging_loss=0.01013, over 16516.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09461, pruned_loss=0.01471, audio_tagging_loss=0.009275, over 3045695.16 frames. ], batch size: 62, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:18:52,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2121180.0, ans=0.2
2023-11-22 22:19:02,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2121246.6666666665, ans=0.125
2023-11-22 22:19:09,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=2121246.6666666665, ans=0.1
2023-11-22 22:19:09,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2121246.6666666665, ans=0.125
2023-11-22 22:19:11,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2121313.3333333335, ans=0.125
2023-11-22 22:19:13,412 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318200
2023-11-22 22:19:22,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2121313.3333333335, ans=0.125
2023-11-22 22:19:32,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2121380.0, ans=0.1
2023-11-22 22:19:39,033 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5600, loss[loss=0.09179, simple_loss=0.1226, pruned_loss=0.0231, audio_tagging_loss=0.007381, over 15483.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09328, pruned_loss=0.0145, audio_tagging_loss=0.009479, over 3051634.74 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:20:01,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2121513.3333333335, ans=0.09899494936611666
2023-11-22 22:20:09,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2121580.0, ans=0.0
2023-11-22 22:20:18,199 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318250
2023-11-22 22:20:22,787 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.364e+01 9.057e+01 9.975e+01 1.357e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 22:20:25,266 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 22:20:27,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2121646.6666666665, ans=0.125
2023-11-22 22:20:42,426 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5650, loss[loss=0.07419, simple_loss=0.1016, pruned_loss=0.01541, audio_tagging_loss=0.007996, over 15681.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.0938, pruned_loss=0.01462, audio_tagging_loss=0.009429, over 3057016.78 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:20:55,697 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.55 vs. limit=15.0
2023-11-22 22:21:00,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2121846.6666666665, ans=0.0
2023-11-22 22:21:10,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2121913.3333333335, ans=0.125
2023-11-22 22:21:22,544 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318300
2023-11-22 22:21:32,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2122046.6666666665, ans=0.0
2023-11-22 22:21:46,022 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5700, loss[loss=0.07953, simple_loss=0.1007, pruned_loss=0.01871, audio_tagging_loss=0.01046, over 14522.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09368, pruned_loss=0.01451, audio_tagging_loss=0.00952, over 3048985.64 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:21:46,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.09 vs. limit=12.0
2023-11-22 22:21:48,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2122113.3333333335, ans=0.2
2023-11-22 22:21:51,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2122113.3333333335, ans=0.0
2023-11-22 22:21:56,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2122113.3333333335, ans=0.2
2023-11-22 22:22:01,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.52 vs. limit=6.0
2023-11-22 22:22:02,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2122180.0, ans=0.125
2023-11-22 22:22:23,738 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.17 vs. limit=12.0
2023-11-22 22:22:25,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318350
2023-11-22 22:22:29,256 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.246e+01 8.801e+01 9.549e+01 1.194e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 22:22:29,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2122313.3333333335, ans=0.1
2023-11-22 22:22:48,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2122380.0, ans=0.0
2023-11-22 22:22:50,410 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5750, loss[loss=0.03903, simple_loss=0.05176, pruned_loss=0.004543, audio_tagging_loss=0.008613, over 15674.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09324, pruned_loss=0.01451, audio_tagging_loss=0.009313, over 3045152.47 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:22:57,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2122446.6666666665, ans=0.125
2023-11-22 22:23:10,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2122513.3333333335, ans=0.125
2023-11-22 22:23:29,911 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318400
2023-11-22 22:23:31,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2122646.6666666665, ans=0.125
2023-11-22 22:23:52,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2122713.3333333335, ans=0.0
2023-11-22 22:23:54,256 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5800, loss[loss=0.06624, simple_loss=0.08359, pruned_loss=0.01589, audio_tagging_loss=0.008555, over 13834.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.0923, pruned_loss=0.01428, audio_tagging_loss=0.009213, over 3055289.02 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:23:54,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.33 vs. limit=22.5
2023-11-22 22:24:14,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2122846.6666666665, ans=0.2
2023-11-22 22:24:21,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2122913.3333333335, ans=0.0
2023-11-22 22:24:26,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2122913.3333333335, ans=0.125
2023-11-22 22:24:29,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2122913.3333333335, ans=0.125
2023-11-22 22:24:29,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2122913.3333333335, ans=0.0
2023-11-22 22:24:29,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2122913.3333333335, ans=0.125
2023-11-22 22:24:34,760 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318450
2023-11-22 22:24:38,377 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.593e+01 8.198e+01 8.735e+01 9.439e+01 1.113e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 22:24:45,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.13 vs. limit=15.0
2023-11-22 22:24:58,471 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5850, loss[loss=0.07077, simple_loss=0.08835, pruned_loss=0.01421, audio_tagging_loss=0.01238, over 14944.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09265, pruned_loss=0.0144, audio_tagging_loss=0.009162, over 3056838.94 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:25:04,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2123113.3333333335, ans=0.04949747468305833
2023-11-22 22:25:04,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2123113.3333333335, ans=0.1
2023-11-22 22:25:33,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2123246.6666666665, ans=0.2
2023-11-22 22:25:38,289 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318500
2023-11-22 22:26:03,118 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5900, loss[loss=0.08351, simple_loss=0.1181, pruned_loss=0.01762, audio_tagging_loss=0.006846, over 15878.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09259, pruned_loss=0.01425, audio_tagging_loss=0.009146, over 3054649.38 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:26:05,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2123446.6666666665, ans=0.05
2023-11-22 22:26:15,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn2.whiten.whitening_limit, batch_count=2123513.3333333335, ans=22.5
2023-11-22 22:26:42,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318550
2023-11-22 22:26:48,714 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.253e+01 8.842e+01 9.499e+01 1.797e+02, threshold=1.768e+02, percent-clipped=1.0
2023-11-22 22:26:53,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2123713.3333333335, ans=0.125
2023-11-22 22:27:03,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2123713.3333333335, ans=0.2
2023-11-22 22:27:06,795 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 5950, loss[loss=0.09786, simple_loss=0.1146, pruned_loss=0.03134, audio_tagging_loss=0.009217, over 16043.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09234, pruned_loss=0.01425, audio_tagging_loss=0.009275, over 3053582.94 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:27:43,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2123913.3333333335, ans=0.2
2023-11-22 22:27:46,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318600
2023-11-22 22:27:56,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2123980.0, ans=0.0
2023-11-22 22:28:10,925 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6000, loss[loss=0.07584, simple_loss=0.1017, pruned_loss=0.01548, audio_tagging_loss=0.00953, over 15934.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.0924, pruned_loss=0.01432, audio_tagging_loss=0.009298, over 3053630.18 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:28:10,925 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 22:28:31,993 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.7400, 5.4350, 5.1632, 5.2438], device='cuda:2')
2023-11-22 22:28:42,974 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.2015, 3.0237, 3.2457, 3.0101, 3.6888, 3.8195, 3.2340, 3.1728],
       device='cuda:2')
2023-11-22 22:28:54,215 INFO [train_asr.py:1253] (2/4) Epoch 27, validation: loss=0.05853, simple_loss=0.05134, pruned_loss=0.005103, audio_tagging_loss=0.02775, over 4681554.00 frames. 
2023-11-22 22:28:54,216 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 22:29:08,463 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.90 vs. limit=15.0
2023-11-22 22:29:32,112 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.49 vs. limit=15.0
2023-11-22 22:29:34,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318650
2023-11-22 22:29:34,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2124313.3333333335, ans=0.125
2023-11-22 22:29:40,609 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.043e+01 8.303e+01 8.890e+01 9.466e+01 1.522e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 22:29:41,913 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 22:29:51,216 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=22.5
2023-11-22 22:29:57,715 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6050, loss[loss=0.06922, simple_loss=0.09474, pruned_loss=0.01629, audio_tagging_loss=0.005561, over 15372.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09336, pruned_loss=0.01428, audio_tagging_loss=0.009109, over 3058374.18 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:30:21,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2124513.3333333335, ans=0.2
2023-11-22 22:30:38,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318700
2023-11-22 22:30:43,716 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.11 vs. limit=15.0
2023-11-22 22:30:46,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2124646.6666666665, ans=0.2
2023-11-22 22:31:02,039 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6100, loss[loss=0.1013, simple_loss=0.1407, pruned_loss=0.02448, audio_tagging_loss=0.006469, over 14947.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09466, pruned_loss=0.0146, audio_tagging_loss=0.009104, over 3061293.69 frames. ], batch size: 52, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:31:41,929 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318750
2023-11-22 22:31:47,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2124980.0, ans=0.2
2023-11-22 22:31:48,031 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.696e+01 8.282e+01 8.867e+01 9.593e+01 1.255e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 22:32:06,859 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6150, loss[loss=0.07552, simple_loss=0.1032, pruned_loss=0.01778, audio_tagging_loss=0.006138, over 15934.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09481, pruned_loss=0.01485, audio_tagging_loss=0.00912, over 3055245.54 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:32:08,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2125113.3333333335, ans=0.2
2023-11-22 22:32:23,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2125180.0, ans=0.1
2023-11-22 22:32:24,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2125180.0, ans=0.0
2023-11-22 22:32:47,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318800
2023-11-22 22:32:54,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2125313.3333333335, ans=0.125
2023-11-22 22:33:11,339 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6200, loss[loss=0.07155, simple_loss=0.09546, pruned_loss=0.01653, audio_tagging_loss=0.00729, over 15443.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09356, pruned_loss=0.01463, audio_tagging_loss=0.009228, over 3057952.77 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:33:38,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2125580.0, ans=0.125
2023-11-22 22:33:38,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2125580.0, ans=0.0
2023-11-22 22:33:52,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318850
2023-11-22 22:33:57,956 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.282e+01 8.300e+01 8.948e+01 9.929e+01 3.016e+02, threshold=1.790e+02, percent-clipped=1.0
2023-11-22 22:34:06,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2125713.3333333335, ans=0.07
2023-11-22 22:34:15,990 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6250, loss[loss=0.04958, simple_loss=0.06454, pruned_loss=0.006721, audio_tagging_loss=0.01059, over 14479.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09317, pruned_loss=0.01454, audio_tagging_loss=0.009375, over 3061411.08 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:34:30,780 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:34:42,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2125913.3333333335, ans=0.125
2023-11-22 22:34:48,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2125913.3333333335, ans=0.0
2023-11-22 22:34:54,077 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.22 vs. limit=15.0
2023-11-22 22:34:54,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2125980.0, ans=0.0
2023-11-22 22:34:55,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318900
2023-11-22 22:35:04,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2125980.0, ans=0.0
2023-11-22 22:35:07,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2126046.6666666665, ans=0.1
2023-11-22 22:35:20,688 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6300, loss[loss=0.06672, simple_loss=0.1001, pruned_loss=0.01018, audio_tagging_loss=0.00652, over 13219.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09341, pruned_loss=0.01464, audio_tagging_loss=0.009371, over 3056672.10 frames. ], batch size: 52, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:35:28,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2126113.3333333335, ans=0.125
2023-11-22 22:35:41,546 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.77 vs. limit=15.0
2023-11-22 22:35:46,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2126246.6666666665, ans=0.1
2023-11-22 22:35:58,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2126313.3333333335, ans=0.2
2023-11-22 22:35:59,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2126313.3333333335, ans=0.0
2023-11-22 22:36:01,574 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 318950
2023-11-22 22:36:07,424 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.041e+01 8.820e+01 9.703e+01 1.205e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 22:36:12,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2126380.0, ans=0.0
2023-11-22 22:36:23,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.32 vs. limit=22.5
2023-11-22 22:36:25,208 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6350, loss[loss=0.06515, simple_loss=0.08623, pruned_loss=0.01277, audio_tagging_loss=0.00927, over 15541.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09375, pruned_loss=0.01456, audio_tagging_loss=0.009376, over 3054856.01 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:36:26,673 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:36:32,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2126446.6666666665, ans=0.5
2023-11-22 22:36:53,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2126580.0, ans=0.2
2023-11-22 22:37:05,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319000
2023-11-22 22:37:22,729 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.49 vs. limit=15.0
2023-11-22 22:37:29,138 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6400, loss[loss=0.1056, simple_loss=0.1491, pruned_loss=0.0237, audio_tagging_loss=0.007367, over 15429.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09439, pruned_loss=0.01463, audio_tagging_loss=0.009449, over 3058902.82 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:37:30,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2126780.0, ans=0.125
2023-11-22 22:37:35,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2126780.0, ans=0.0
2023-11-22 22:37:37,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2126780.0, ans=0.125
2023-11-22 22:37:41,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2126846.6666666665, ans=0.125
2023-11-22 22:37:59,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2126913.3333333335, ans=0.125
2023-11-22 22:38:08,556 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319050
2023-11-22 22:38:15,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.479e+01 8.884e+01 9.505e+01 1.366e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 22:38:17,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2126980.0, ans=10.0
2023-11-22 22:38:23,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0
2023-11-22 22:38:32,789 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6450, loss[loss=0.08107, simple_loss=0.1044, pruned_loss=0.01947, audio_tagging_loss=0.009398, over 14884.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09378, pruned_loss=0.01453, audio_tagging_loss=0.009558, over 3063738.71 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:38:37,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2127113.3333333335, ans=0.125
2023-11-22 22:38:39,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2127113.3333333335, ans=0.125
2023-11-22 22:38:43,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2127113.3333333335, ans=0.125
2023-11-22 22:38:53,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2127180.0, ans=0.125
2023-11-22 22:38:53,337 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.51 vs. limit=15.0
2023-11-22 22:39:08,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2127246.6666666665, ans=15.0
2023-11-22 22:39:11,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319100
2023-11-22 22:39:36,764 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6500, loss[loss=0.06952, simple_loss=0.09993, pruned_loss=0.0116, audio_tagging_loss=0.007955, over 15852.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09345, pruned_loss=0.01447, audio_tagging_loss=0.009488, over 3061080.03 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:39:42,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2127446.6666666665, ans=0.1
2023-11-22 22:39:45,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2127446.6666666665, ans=0.2
2023-11-22 22:39:46,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2127446.6666666665, ans=0.025
2023-11-22 22:40:09,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2127580.0, ans=0.0
2023-11-22 22:40:17,228 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319150
2023-11-22 22:40:18,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2127646.6666666665, ans=0.125
2023-11-22 22:40:19,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2127646.6666666665, ans=0.0
2023-11-22 22:40:24,329 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.580e+01 8.470e+01 8.995e+01 9.586e+01 1.282e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 22:40:27,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2127713.3333333335, ans=0.09899494936611666
2023-11-22 22:40:40,338 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6550, loss[loss=0.07728, simple_loss=0.09549, pruned_loss=0.01801, audio_tagging_loss=0.01152, over 15455.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09376, pruned_loss=0.01451, audio_tagging_loss=0.009332, over 3054458.42 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:40:51,559 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0
2023-11-22 22:40:59,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2127846.6666666665, ans=0.1
2023-11-22 22:41:13,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2127913.3333333335, ans=0.1
2023-11-22 22:41:21,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319200
2023-11-22 22:41:30,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2127980.0, ans=0.2
2023-11-22 22:41:45,847 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6600, loss[loss=0.07349, simple_loss=0.1007, pruned_loss=0.0142, audio_tagging_loss=0.008928, over 15181.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09486, pruned_loss=0.01481, audio_tagging_loss=0.009149, over 3056203.33 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:42:02,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2128180.0, ans=0.025
2023-11-22 22:42:03,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2128180.0, ans=0.05
2023-11-22 22:42:17,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2128246.6666666665, ans=0.2
2023-11-22 22:42:24,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2128313.3333333335, ans=0.125
2023-11-22 22:42:25,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319250
2023-11-22 22:42:31,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2128313.3333333335, ans=0.1
2023-11-22 22:42:33,337 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.441e+01 8.485e+01 9.023e+01 9.901e+01 1.531e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-22 22:42:49,976 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6650, loss[loss=0.0606, simple_loss=0.08153, pruned_loss=0.01145, audio_tagging_loss=0.008383, over 14882.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09426, pruned_loss=0.0146, audio_tagging_loss=0.009034, over 3050214.07 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:42:56,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2128446.6666666665, ans=0.125
2023-11-22 22:43:01,818 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.07 vs. limit=15.0
2023-11-22 22:43:25,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2128580.0, ans=0.0
2023-11-22 22:43:29,487 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319300
2023-11-22 22:43:36,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2128646.6666666665, ans=0.0
2023-11-22 22:43:37,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2128646.6666666665, ans=0.125
2023-11-22 22:43:40,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2128713.3333333335, ans=0.125
2023-11-22 22:43:53,715 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6700, loss[loss=0.06108, simple_loss=0.07788, pruned_loss=0.01275, audio_tagging_loss=0.009385, over 14706.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.0934, pruned_loss=0.01452, audio_tagging_loss=0.009039, over 3039641.53 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:43:55,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2128780.0, ans=0.125
2023-11-22 22:44:18,325 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0
2023-11-22 22:44:34,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319350
2023-11-22 22:44:41,538 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.124e+01 8.630e+01 9.282e+01 1.213e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 22:44:54,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2129046.6666666665, ans=0.125
2023-11-22 22:44:56,575 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.91 vs. limit=15.0
2023-11-22 22:44:58,811 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6750, loss[loss=0.06016, simple_loss=0.07712, pruned_loss=0.01235, audio_tagging_loss=0.009258, over 14396.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09315, pruned_loss=0.0145, audio_tagging_loss=0.008995, over 3039040.93 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:45:09,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2129113.3333333335, ans=0.125
2023-11-22 22:45:30,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2129246.6666666665, ans=0.125
2023-11-22 22:45:31,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2129246.6666666665, ans=0.0
2023-11-22 22:45:37,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319400
2023-11-22 22:45:42,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2129313.3333333335, ans=0.05
2023-11-22 22:45:43,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2129313.3333333335, ans=0.125
2023-11-22 22:45:50,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2129380.0, ans=0.2
2023-11-22 22:45:58,699 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=15.0
2023-11-22 22:46:03,693 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6800, loss[loss=0.07301, simple_loss=0.09924, pruned_loss=0.0135, audio_tagging_loss=0.009897, over 15941.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09327, pruned_loss=0.01471, audio_tagging_loss=0.009038, over 3034142.17 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:46:15,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2129513.3333333335, ans=0.125
2023-11-22 22:46:23,869 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.61 vs. limit=22.5
2023-11-22 22:46:28,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2129580.0, ans=0.125
2023-11-22 22:46:29,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2129580.0, ans=0.125
2023-11-22 22:46:42,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2129646.6666666665, ans=0.1
2023-11-22 22:46:43,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319450
2023-11-22 22:46:50,903 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.253e+01 9.028e+01 9.739e+01 1.351e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 22:46:54,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2129713.3333333335, ans=0.0
2023-11-22 22:46:56,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2129713.3333333335, ans=0.0
2023-11-22 22:47:07,611 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6850, loss[loss=0.07456, simple_loss=0.09446, pruned_loss=0.0183, audio_tagging_loss=0.009024, over 15517.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09371, pruned_loss=0.01481, audio_tagging_loss=0.009006, over 3027955.63 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:47:07,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2129780.0, ans=0.125
2023-11-22 22:47:11,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2129780.0, ans=0.025
2023-11-22 22:47:20,502 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.87 vs. limit=15.0
2023-11-22 22:47:36,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2129913.3333333335, ans=0.0
2023-11-22 22:47:37,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2129913.3333333335, ans=0.125
2023-11-22 22:47:46,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2129980.0, ans=0.125
2023-11-22 22:47:47,742 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319500
2023-11-22 22:47:56,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2129980.0, ans=0.125
2023-11-22 22:48:02,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2130046.6666666665, ans=0.125
2023-11-22 22:48:10,842 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.67 vs. limit=15.0
2023-11-22 22:48:11,920 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6900, loss[loss=0.06403, simple_loss=0.07896, pruned_loss=0.01372, audio_tagging_loss=0.01083, over 16621.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.0935, pruned_loss=0.01481, audio_tagging_loss=0.009067, over 3042045.02 frames. ], batch size: 63, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:48:12,824 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=13.44 vs. limit=15.0
2023-11-22 22:48:39,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2130246.6666666665, ans=0.2
2023-11-22 22:48:41,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2130246.6666666665, ans=0.125
2023-11-22 22:48:51,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319550
2023-11-22 22:48:59,083 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.191e+01 8.934e+01 9.580e+01 1.102e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 22:49:02,189 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 22:49:04,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2130380.0, ans=0.125
2023-11-22 22:49:15,980 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 6950, loss[loss=0.07797, simple_loss=0.1066, pruned_loss=0.01628, audio_tagging_loss=0.008387, over 15125.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09398, pruned_loss=0.01474, audio_tagging_loss=0.009085, over 3047173.94 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:49:44,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2130580.0, ans=0.0
2023-11-22 22:49:51,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2130580.0, ans=0.1
2023-11-22 22:49:55,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319600
2023-11-22 22:49:55,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2130646.6666666665, ans=0.125
2023-11-22 22:50:07,504 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.09 vs. limit=15.0
2023-11-22 22:50:19,642 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7000, loss[loss=0.05545, simple_loss=0.07136, pruned_loss=0.01069, audio_tagging_loss=0.009078, over 16203.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09394, pruned_loss=0.01472, audio_tagging_loss=0.00916, over 3051018.72 frames. ], batch size: 61, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:50:33,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2130846.6666666665, ans=0.125
2023-11-22 22:50:41,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2130846.6666666665, ans=0.0
2023-11-22 22:50:44,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2130913.3333333335, ans=0.2
2023-11-22 22:50:54,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2130913.3333333335, ans=0.0
2023-11-22 22:50:58,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2130980.0, ans=0.125
2023-11-22 22:50:59,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319650
2023-11-22 22:51:07,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0
2023-11-22 22:51:07,650 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.087e+01 8.954e+01 9.728e+01 1.359e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 22:51:16,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2131046.6666666665, ans=0.125
2023-11-22 22:51:17,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2131046.6666666665, ans=0.125
2023-11-22 22:51:21,895 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.55 vs. limit=15.0
2023-11-22 22:51:23,713 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7050, loss[loss=0.06261, simple_loss=0.07904, pruned_loss=0.01307, audio_tagging_loss=0.01001, over 14890.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09334, pruned_loss=0.01451, audio_tagging_loss=0.009189, over 3048183.49 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:52:03,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319700
2023-11-22 22:52:10,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2131313.3333333335, ans=0.0
2023-11-22 22:52:23,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2131380.0, ans=0.0
2023-11-22 22:52:28,767 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7100, loss[loss=0.06094, simple_loss=0.08453, pruned_loss=0.007338, audio_tagging_loss=0.01134, over 15797.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09245, pruned_loss=0.01442, audio_tagging_loss=0.009295, over 3044811.48 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:52:37,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2131446.6666666665, ans=0.0
2023-11-22 22:52:42,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2131513.3333333335, ans=0.05
2023-11-22 22:52:49,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2131513.3333333335, ans=0.0
2023-11-22 22:53:07,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319750
2023-11-22 22:53:15,647 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 7.978e+01 8.785e+01 9.427e+01 1.365e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 22:53:31,761 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7150, loss[loss=0.08439, simple_loss=0.1048, pruned_loss=0.02011, audio_tagging_loss=0.01187, over 14823.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09244, pruned_loss=0.01441, audio_tagging_loss=0.009299, over 3048844.19 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:53:41,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2131780.0, ans=0.125
2023-11-22 22:53:57,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2131913.3333333335, ans=0.125
2023-11-22 22:54:00,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2131913.3333333335, ans=0.2
2023-11-22 22:54:09,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2131980.0, ans=0.125
2023-11-22 22:54:11,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319800
2023-11-22 22:54:15,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.72 vs. limit=15.0
2023-11-22 22:54:15,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2131980.0, ans=0.025
2023-11-22 22:54:23,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2132046.6666666665, ans=0.0
2023-11-22 22:54:35,959 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7200, loss[loss=0.06834, simple_loss=0.08559, pruned_loss=0.01253, audio_tagging_loss=0.01302, over 15295.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09247, pruned_loss=0.01424, audio_tagging_loss=0.009441, over 3046308.28 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:54:52,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2132180.0, ans=0.125
2023-11-22 22:55:15,839 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319850
2023-11-22 22:55:16,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2132313.3333333335, ans=10.0
2023-11-22 22:55:23,099 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.344e+01 8.207e+01 8.744e+01 9.394e+01 1.258e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 22:55:23,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2132313.3333333335, ans=0.125
2023-11-22 22:55:27,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2132380.0, ans=0.2
2023-11-22 22:55:35,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2132380.0, ans=0.125
2023-11-22 22:55:36,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2132380.0, ans=0.125
2023-11-22 22:55:39,841 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7250, loss[loss=0.05929, simple_loss=0.08018, pruned_loss=0.008167, audio_tagging_loss=0.01103, over 16110.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09184, pruned_loss=0.01411, audio_tagging_loss=0.009546, over 3050179.74 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:55:48,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2132446.6666666665, ans=0.125
2023-11-22 22:55:55,277 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:56:00,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2132513.3333333335, ans=0.2
2023-11-22 22:56:07,487 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.53 vs. limit=15.0
2023-11-22 22:56:18,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319900
2023-11-22 22:56:23,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2132646.6666666665, ans=0.1
2023-11-22 22:56:38,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2132713.3333333335, ans=0.125
2023-11-22 22:56:43,077 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7300, loss[loss=0.07676, simple_loss=0.1058, pruned_loss=0.01611, audio_tagging_loss=0.007747, over 16162.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09125, pruned_loss=0.01407, audio_tagging_loss=0.009517, over 3048178.51 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:57:15,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2132913.3333333335, ans=0.0
2023-11-22 22:57:16,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2132913.3333333335, ans=0.125
2023-11-22 22:57:22,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 319950
2023-11-22 22:57:32,779 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.257e+01 8.750e+01 9.620e+01 1.168e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 22:57:42,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2133046.6666666665, ans=0.0
2023-11-22 22:57:46,965 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7350, loss[loss=0.0547, simple_loss=0.06656, pruned_loss=0.01055, audio_tagging_loss=0.01087, over 15147.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09131, pruned_loss=0.01413, audio_tagging_loss=0.009383, over 3042603.20 frames. ], batch size: 63, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:57:51,381 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.63 vs. limit=15.0
2023-11-22 22:58:21,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2133246.6666666665, ans=0.5
2023-11-22 22:58:22,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-22 22:58:26,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320000
2023-11-22 22:58:37,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=2133313.3333333335, ans=0.1
2023-11-22 22:58:53,952 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7400, loss[loss=0.07022, simple_loss=0.09663, pruned_loss=0.01469, audio_tagging_loss=0.007219, over 16900.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09152, pruned_loss=0.01422, audio_tagging_loss=0.009162, over 3037333.40 frames. ], batch size: 61, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:58:55,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2133446.6666666665, ans=0.125
2023-11-22 22:59:02,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2133446.6666666665, ans=0.0
2023-11-22 22:59:05,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2133513.3333333335, ans=0.1
2023-11-22 22:59:15,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2133513.3333333335, ans=0.0
2023-11-22 22:59:20,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2133580.0, ans=0.1
2023-11-22 22:59:25,724 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.57 vs. limit=15.0
2023-11-22 22:59:26,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2133580.0, ans=0.0
2023-11-22 22:59:29,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2133580.0, ans=0.125
2023-11-22 22:59:33,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320050
2023-11-22 22:59:43,443 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.110e+01 8.793e+01 9.338e+01 3.637e+02, threshold=1.759e+02, percent-clipped=1.0
2023-11-22 22:59:54,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2133713.3333333335, ans=0.07
2023-11-22 22:59:57,617 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7450, loss[loss=0.07363, simple_loss=0.1008, pruned_loss=0.01493, audio_tagging_loss=0.00829, over 15449.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.0907, pruned_loss=0.01417, audio_tagging_loss=0.009137, over 3037508.80 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 23:00:00,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.15 vs. limit=22.5
2023-11-22 23:00:34,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2133913.3333333335, ans=0.125
2023-11-22 23:00:38,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320100
2023-11-22 23:01:01,248 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7500, loss[loss=0.07166, simple_loss=0.09883, pruned_loss=0.01226, audio_tagging_loss=0.009991, over 15585.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09125, pruned_loss=0.01417, audio_tagging_loss=0.009072, over 3043463.71 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 23:01:01,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2134113.3333333335, ans=0.0
2023-11-22 23:01:12,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2134113.3333333335, ans=0.0
2023-11-22 23:01:35,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2134246.6666666665, ans=0.07
2023-11-22 23:01:41,521 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320150
2023-11-22 23:01:45,793 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-22 23:01:51,157 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.201e+01 8.778e+01 9.335e+01 1.224e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 23:02:05,940 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7550, loss[loss=0.07179, simple_loss=0.09703, pruned_loss=0.01465, audio_tagging_loss=0.008619, over 15170.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09234, pruned_loss=0.01435, audio_tagging_loss=0.009041, over 3046176.47 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 23:02:26,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2134513.3333333335, ans=0.0
2023-11-22 23:02:29,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2134513.3333333335, ans=0.0
2023-11-22 23:02:37,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2134580.0, ans=0.125
2023-11-22 23:02:40,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2134580.0, ans=10.0
2023-11-22 23:02:45,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320200
2023-11-22 23:03:10,694 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7600, loss[loss=0.08605, simple_loss=0.1245, pruned_loss=0.018, audio_tagging_loss=0.005799, over 16319.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09175, pruned_loss=0.01419, audio_tagging_loss=0.009109, over 3045332.87 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 23:03:31,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2134846.6666666665, ans=0.125
2023-11-22 23:03:43,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2134913.3333333335, ans=0.0
2023-11-22 23:03:44,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2134913.3333333335, ans=0.0
2023-11-22 23:03:48,201 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.45 vs. limit=15.0
2023-11-22 23:03:50,873 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320250
2023-11-22 23:03:59,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2134980.0, ans=0.125
2023-11-22 23:04:00,565 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.533e+01 8.089e+01 8.781e+01 9.511e+01 1.232e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 23:04:09,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2135046.6666666665, ans=0.125
2023-11-22 23:04:13,892 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7650, loss[loss=0.0865, simple_loss=0.1171, pruned_loss=0.01973, audio_tagging_loss=0.008196, over 15168.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09164, pruned_loss=0.01418, audio_tagging_loss=0.009186, over 3046446.67 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:04:19,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2135113.3333333335, ans=0.09899494936611666
2023-11-22 23:04:20,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2135113.3333333335, ans=0.05
2023-11-22 23:04:36,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2135180.0, ans=0.125
2023-11-22 23:04:40,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2135246.6666666665, ans=0.2
2023-11-22 23:04:49,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2135246.6666666665, ans=0.125
2023-11-22 23:04:53,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320300
2023-11-22 23:04:53,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2135313.3333333335, ans=0.125
2023-11-22 23:05:17,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2135446.6666666665, ans=10.0
2023-11-22 23:05:17,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2135446.6666666665, ans=0.0
2023-11-22 23:05:18,670 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7700, loss[loss=0.07629, simple_loss=0.1068, pruned_loss=0.01546, audio_tagging_loss=0.007434, over 15992.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09209, pruned_loss=0.01412, audio_tagging_loss=0.00923, over 3045877.92 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:05:22,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2135446.6666666665, ans=0.0
2023-11-22 23:05:48,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2135580.0, ans=0.0
2023-11-22 23:05:49,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2135580.0, ans=0.0
2023-11-22 23:05:59,433 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320350
2023-11-22 23:06:10,430 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.326e+01 9.020e+01 9.879e+01 1.267e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-22 23:06:16,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2135713.3333333335, ans=0.125
2023-11-22 23:06:24,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2135780.0, ans=0.125
2023-11-22 23:06:25,278 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7750, loss[loss=0.06061, simple_loss=0.07833, pruned_loss=0.009998, audio_tagging_loss=0.01145, over 16482.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09218, pruned_loss=0.01413, audio_tagging_loss=0.009197, over 3054144.83 frames. ], batch size: 63, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:06:34,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2135780.0, ans=0.125
2023-11-22 23:06:50,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2135913.3333333335, ans=0.2
2023-11-22 23:06:51,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2135913.3333333335, ans=0.125
2023-11-22 23:07:05,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320400
2023-11-22 23:07:08,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2135980.0, ans=0.125
2023-11-22 23:07:20,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2136046.6666666665, ans=0.0
2023-11-22 23:07:21,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2136046.6666666665, ans=0.1
2023-11-22 23:07:29,750 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7800, loss[loss=0.07084, simple_loss=0.1038, pruned_loss=0.01157, audio_tagging_loss=0.0074, over 16399.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09326, pruned_loss=0.01434, audio_tagging_loss=0.009116, over 3054078.05 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:07:42,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2136180.0, ans=0.0
2023-11-22 23:07:47,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2136180.0, ans=0.1
2023-11-22 23:07:49,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2136180.0, ans=0.125
2023-11-22 23:08:08,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2136313.3333333335, ans=0.125
2023-11-22 23:08:10,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320450
2023-11-22 23:08:14,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2136313.3333333335, ans=0.125
2023-11-22 23:08:17,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2136313.3333333335, ans=0.125
2023-11-22 23:08:19,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.253e+01 8.929e+01 9.795e+01 1.214e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 23:08:21,673 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.01 vs. limit=10.0
2023-11-22 23:08:23,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2136380.0, ans=0.025
2023-11-22 23:08:33,304 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7850, loss[loss=0.0815, simple_loss=0.1146, pruned_loss=0.0148, audio_tagging_loss=0.009391, over 15698.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09339, pruned_loss=0.01435, audio_tagging_loss=0.009111, over 3050738.11 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:08:35,246 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.12 vs. limit=15.0
2023-11-22 23:08:47,543 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.52 vs. limit=22.5
2023-11-22 23:09:14,146 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320500
2023-11-22 23:09:39,727 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7900, loss[loss=0.06277, simple_loss=0.08284, pruned_loss=0.01132, audio_tagging_loss=0.01003, over 15126.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09354, pruned_loss=0.01455, audio_tagging_loss=0.009232, over 3046471.68 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:09:46,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2136780.0, ans=0.2
2023-11-22 23:10:03,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2136913.3333333335, ans=0.0
2023-11-22 23:10:04,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2136913.3333333335, ans=0.125
2023-11-22 23:10:08,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2136913.3333333335, ans=0.125
2023-11-22 23:10:09,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2136913.3333333335, ans=0.025
2023-11-22 23:10:19,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320550
2023-11-22 23:10:29,991 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.096e+01 8.900e+01 9.590e+01 1.237e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 23:10:32,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2137046.6666666665, ans=0.125
2023-11-22 23:10:33,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.33 vs. limit=15.0
2023-11-22 23:10:39,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.28 vs. limit=22.5
2023-11-22 23:10:41,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2137046.6666666665, ans=0.1
2023-11-22 23:10:43,599 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 7950, loss[loss=0.06895, simple_loss=0.09753, pruned_loss=0.01041, audio_tagging_loss=0.009775, over 15572.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09307, pruned_loss=0.01459, audio_tagging_loss=0.009311, over 3051641.86 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:10:51,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2137113.3333333335, ans=0.2
2023-11-22 23:10:58,094 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:11:11,982 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=12.0
2023-11-22 23:11:20,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2137246.6666666665, ans=0.2
2023-11-22 23:11:23,992 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320600
2023-11-22 23:11:27,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.38 vs. limit=15.0
2023-11-22 23:11:29,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2137313.3333333335, ans=0.0
2023-11-22 23:11:31,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2137313.3333333335, ans=0.125
2023-11-22 23:11:43,146 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:11:43,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.12 vs. limit=15.0
2023-11-22 23:11:47,661 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8000, loss[loss=0.07532, simple_loss=0.1002, pruned_loss=0.01459, audio_tagging_loss=0.01063, over 14880.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09259, pruned_loss=0.01457, audio_tagging_loss=0.009385, over 3046659.61 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:12:01,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.88 vs. limit=15.0
2023-11-22 23:12:03,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2137513.3333333335, ans=0.2
2023-11-22 23:12:19,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2137580.0, ans=0.125
2023-11-22 23:12:24,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2137580.0, ans=0.125
2023-11-22 23:12:26,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2137646.6666666665, ans=0.1
2023-11-22 23:12:27,867 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320650
2023-11-22 23:12:37,609 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.271e+01 8.710e+01 9.655e+01 1.155e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 23:12:47,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2137713.3333333335, ans=0.125
2023-11-22 23:12:50,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2137713.3333333335, ans=0.04949747468305833
2023-11-22 23:12:52,947 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8050, loss[loss=0.06671, simple_loss=0.0938, pruned_loss=0.01292, audio_tagging_loss=0.00689, over 15414.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09271, pruned_loss=0.0147, audio_tagging_loss=0.009418, over 3044248.74 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:13:24,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2137913.3333333335, ans=0.2
2023-11-22 23:13:32,230 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320700
2023-11-22 23:13:37,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2137980.0, ans=0.0
2023-11-22 23:13:39,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2137980.0, ans=0.0
2023-11-22 23:13:57,270 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8100, loss[loss=0.04128, simple_loss=0.04619, pruned_loss=0.005722, audio_tagging_loss=0.01246, over 14686.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09357, pruned_loss=0.01506, audio_tagging_loss=0.009324, over 3046483.63 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:13:59,271 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0
2023-11-22 23:14:37,229 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320750
2023-11-22 23:14:40,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.16 vs. limit=12.0
2023-11-22 23:14:43,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2138313.3333333335, ans=0.5
2023-11-22 23:14:47,689 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.347e+01 8.289e+01 8.980e+01 9.464e+01 1.714e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-22 23:15:00,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2138446.6666666665, ans=0.125
2023-11-22 23:15:01,304 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8150, loss[loss=0.05083, simple_loss=0.06733, pruned_loss=0.007458, audio_tagging_loss=0.0097, over 15655.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09368, pruned_loss=0.01494, audio_tagging_loss=0.009211, over 3046615.24 frames. ], batch size: 62, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:15:02,730 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:15:08,074 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.58 vs. limit=15.0
2023-11-22 23:15:09,423 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:15:16,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2138513.3333333335, ans=0.125
2023-11-22 23:15:33,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2138580.0, ans=0.2
2023-11-22 23:15:40,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2138646.6666666665, ans=0.125
2023-11-22 23:15:41,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320800
2023-11-22 23:15:55,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2138713.3333333335, ans=0.125
2023-11-22 23:16:06,075 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8200, loss[loss=0.05712, simple_loss=0.07364, pruned_loss=0.01223, audio_tagging_loss=0.00807, over 15285.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09391, pruned_loss=0.01495, audio_tagging_loss=0.008975, over 3052210.17 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:16:06,133 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:16:08,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-22 23:16:15,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2138780.0, ans=0.125
2023-11-22 23:16:18,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.84 vs. limit=10.0
2023-11-22 23:16:27,683 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.59 vs. limit=12.0
2023-11-22 23:16:31,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2138913.3333333335, ans=0.0
2023-11-22 23:16:31,013 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:16:34,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2138913.3333333335, ans=0.0
2023-11-22 23:16:45,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320850
2023-11-22 23:16:56,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2138980.0, ans=0.1
2023-11-22 23:16:56,801 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.281e+01 9.046e+01 9.469e+01 1.170e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 23:17:02,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2139046.6666666665, ans=0.05
2023-11-22 23:17:07,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2139046.6666666665, ans=0.2
2023-11-22 23:17:11,171 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8250, loss[loss=0.06288, simple_loss=0.08651, pruned_loss=0.01086, audio_tagging_loss=0.00876, over 17038.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09349, pruned_loss=0.01485, audio_tagging_loss=0.008894, over 3061784.01 frames. ], batch size: 65, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:17:21,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2139113.3333333335, ans=0.1
2023-11-22 23:17:30,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2139180.0, ans=0.125
2023-11-22 23:17:34,207 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:17:35,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2139246.6666666665, ans=0.0
2023-11-22 23:17:45,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2139246.6666666665, ans=0.2
2023-11-22 23:17:51,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320900
2023-11-22 23:17:58,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2139313.3333333335, ans=0.0
2023-11-22 23:18:13,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2139380.0, ans=0.125
2023-11-22 23:18:15,853 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8300, loss[loss=0.06677, simple_loss=0.08842, pruned_loss=0.01222, audio_tagging_loss=0.01034, over 15397.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.0934, pruned_loss=0.01483, audio_tagging_loss=0.008899, over 3056410.08 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:18:47,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2139580.0, ans=0.125
2023-11-22 23:18:56,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 320950
2023-11-22 23:18:56,742 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:19:05,948 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.400e+01 9.007e+01 9.767e+01 1.180e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 23:19:20,882 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8350, loss[loss=0.06219, simple_loss=0.07647, pruned_loss=0.01136, audio_tagging_loss=0.01259, over 15063.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09316, pruned_loss=0.01462, audio_tagging_loss=0.008912, over 3051383.74 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:19:22,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2139780.0, ans=0.0
2023-11-22 23:19:29,524 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.17 vs. limit=10.0
2023-11-22 23:19:38,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2139846.6666666665, ans=0.125
2023-11-22 23:19:46,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2139913.3333333335, ans=0.1
2023-11-22 23:19:59,970 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321000
2023-11-22 23:20:14,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.53 vs. limit=12.0
2023-11-22 23:20:24,977 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8400, loss[loss=0.05537, simple_loss=0.07282, pruned_loss=0.00843, audio_tagging_loss=0.01053, over 16038.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09266, pruned_loss=0.01447, audio_tagging_loss=0.008923, over 3051262.87 frames. ], batch size: 60, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:20:28,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2140113.3333333335, ans=0.035
2023-11-22 23:20:30,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2140113.3333333335, ans=0.0
2023-11-22 23:20:45,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2140180.0, ans=0.125
2023-11-22 23:21:06,710 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321050
2023-11-22 23:21:08,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2140313.3333333335, ans=0.0
2023-11-22 23:21:17,702 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.024e+01 8.884e+01 9.515e+01 1.445e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 23:21:19,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2140380.0, ans=0.0
2023-11-22 23:21:29,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2140446.6666666665, ans=0.2
2023-11-22 23:21:30,750 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8450, loss[loss=0.06843, simple_loss=0.08805, pruned_loss=0.01558, audio_tagging_loss=0.008831, over 16063.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09208, pruned_loss=0.01431, audio_tagging_loss=0.009014, over 3051492.07 frames. ], batch size: 61, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:21:42,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2140513.3333333335, ans=0.125
2023-11-22 23:21:53,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2140513.3333333335, ans=0.0
2023-11-22 23:21:59,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2140580.0, ans=0.04949747468305833
2023-11-22 23:22:11,048 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321100
2023-11-22 23:22:18,530 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.23 vs. limit=22.5
2023-11-22 23:22:36,200 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8500, loss[loss=0.07954, simple_loss=0.1169, pruned_loss=0.01472, audio_tagging_loss=0.006368, over 15572.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09248, pruned_loss=0.01427, audio_tagging_loss=0.009067, over 3049271.15 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:23:04,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2140913.3333333335, ans=0.125
2023-11-22 23:23:15,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321150
2023-11-22 23:23:15,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2140980.0, ans=0.125
2023-11-22 23:23:27,508 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.243e+01 8.956e+01 9.576e+01 1.335e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 23:23:33,425 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:23:38,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2141046.6666666665, ans=0.0
2023-11-22 23:23:40,533 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8550, loss[loss=0.05483, simple_loss=0.06743, pruned_loss=0.0114, audio_tagging_loss=0.00972, over 13831.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09208, pruned_loss=0.01427, audio_tagging_loss=0.009115, over 3046682.08 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:24:08,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2141246.6666666665, ans=0.0
2023-11-22 23:24:09,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2141246.6666666665, ans=0.125
2023-11-22 23:24:15,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2141246.6666666665, ans=0.1
2023-11-22 23:24:20,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321200
2023-11-22 23:24:22,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2141313.3333333335, ans=0.0
2023-11-22 23:24:30,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2141313.3333333335, ans=0.0
2023-11-22 23:24:31,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.86 vs. limit=12.0
2023-11-22 23:24:44,465 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8600, loss[loss=0.08524, simple_loss=0.1163, pruned_loss=0.02144, audio_tagging_loss=0.005621, over 15087.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09243, pruned_loss=0.01435, audio_tagging_loss=0.009205, over 3049801.84 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:24:51,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2141446.6666666665, ans=0.04949747468305833
2023-11-22 23:25:16,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2141580.0, ans=0.0
2023-11-22 23:25:25,087 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321250
2023-11-22 23:25:25,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.94 vs. limit=22.5
2023-11-22 23:25:32,996 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.38 vs. limit=15.0
2023-11-22 23:25:33,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2141646.6666666665, ans=0.125
2023-11-22 23:25:36,062 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.429e+01 8.395e+01 8.985e+01 9.646e+01 1.201e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 23:25:49,421 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8650, loss[loss=0.07032, simple_loss=0.09187, pruned_loss=0.0162, audio_tagging_loss=0.008195, over 14868.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09297, pruned_loss=0.0144, audio_tagging_loss=0.009227, over 3050557.74 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:25:54,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2141780.0, ans=0.2
2023-11-22 23:25:55,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2141780.0, ans=0.125
2023-11-22 23:26:09,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2141846.6666666665, ans=0.1
2023-11-22 23:26:25,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2141913.3333333335, ans=0.125
2023-11-22 23:26:28,826 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321300
2023-11-22 23:26:39,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2141980.0, ans=0.125
2023-11-22 23:26:54,816 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8700, loss[loss=0.08294, simple_loss=0.1129, pruned_loss=0.01885, audio_tagging_loss=0.007622, over 14963.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09367, pruned_loss=0.01449, audio_tagging_loss=0.009262, over 3043465.83 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:27:02,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2142113.3333333335, ans=0.125
2023-11-22 23:27:12,385 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.37 vs. limit=22.5
2023-11-22 23:27:18,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2142246.6666666665, ans=0.125
2023-11-22 23:27:19,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2142246.6666666665, ans=0.0
2023-11-22 23:27:31,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2142246.6666666665, ans=0.125
2023-11-22 23:27:33,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2142313.3333333335, ans=0.09899494936611666
2023-11-22 23:27:34,513 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321350
2023-11-22 23:27:45,844 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 8.323e+01 8.903e+01 9.633e+01 3.250e+02, threshold=1.781e+02, percent-clipped=1.0
2023-11-22 23:27:46,421 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.10 vs. limit=15.0
2023-11-22 23:27:58,127 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8750, loss[loss=0.07344, simple_loss=0.09911, pruned_loss=0.01504, audio_tagging_loss=0.008845, over 15366.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09431, pruned_loss=0.01472, audio_tagging_loss=0.009328, over 3038176.52 frames. ], batch size: 61, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:28:34,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2142580.0, ans=0.2
2023-11-22 23:28:39,047 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321400
2023-11-22 23:29:03,163 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8800, loss[loss=0.07923, simple_loss=0.1042, pruned_loss=0.0176, audio_tagging_loss=0.009534, over 14328.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09444, pruned_loss=0.01482, audio_tagging_loss=0.009423, over 3037712.04 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:29:15,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2142780.0, ans=0.1
2023-11-22 23:29:35,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2142913.3333333335, ans=0.0
2023-11-22 23:29:44,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321450
2023-11-22 23:29:55,933 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.814e+01 8.244e+01 8.962e+01 9.489e+01 1.229e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 23:30:10,435 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8850, loss[loss=0.07748, simple_loss=0.1035, pruned_loss=0.01917, audio_tagging_loss=0.006563, over 15669.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09347, pruned_loss=0.01469, audio_tagging_loss=0.009461, over 3038291.42 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:30:21,718 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:30:36,362 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.56 vs. limit=5.0
2023-11-22 23:30:43,364 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.75 vs. limit=15.0
2023-11-22 23:30:50,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321500
2023-11-22 23:31:07,680 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.29 vs. limit=15.0
2023-11-22 23:31:11,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2143380.0, ans=0.125
2023-11-22 23:31:14,523 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8900, loss[loss=0.08967, simple_loss=0.1175, pruned_loss=0.02097, audio_tagging_loss=0.00996, over 16024.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09286, pruned_loss=0.01456, audio_tagging_loss=0.009303, over 3038680.25 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:31:18,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2143446.6666666665, ans=0.125
2023-11-22 23:31:27,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2143513.3333333335, ans=0.0
2023-11-22 23:31:53,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.06 vs. limit=10.0
2023-11-22 23:31:55,126 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321550
2023-11-22 23:32:05,882 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 8.181e+01 8.758e+01 9.474e+01 1.266e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 23:32:12,777 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=15.0
2023-11-22 23:32:18,173 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 8950, loss[loss=0.08274, simple_loss=0.1066, pruned_loss=0.02236, audio_tagging_loss=0.007072, over 15127.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09326, pruned_loss=0.01453, audio_tagging_loss=0.009123, over 3046961.86 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:32:20,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2143780.0, ans=0.1
2023-11-22 23:32:53,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2143913.3333333335, ans=0.125
2023-11-22 23:33:00,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321600
2023-11-22 23:33:01,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2143980.0, ans=0.125
2023-11-22 23:33:03,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2143980.0, ans=0.125
2023-11-22 23:33:14,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.10 vs. limit=6.0
2023-11-22 23:33:21,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.98 vs. limit=15.0
2023-11-22 23:33:26,993 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9000, loss[loss=0.07064, simple_loss=0.09751, pruned_loss=0.01424, audio_tagging_loss=0.007641, over 16000.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09402, pruned_loss=0.0147, audio_tagging_loss=0.009002, over 3053982.43 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:33:26,993 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-22 23:34:05,740 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4703, 3.2311, 3.7185, 3.4625], device='cuda:2')
2023-11-22 23:34:08,307 INFO [train_asr.py:1253] (2/4) Epoch 27, validation: loss=0.05906, simple_loss=0.05129, pruned_loss=0.005052, audio_tagging_loss=0.02836, over 4681554.00 frames. 
2023-11-22 23:34:08,308 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-22 23:34:12,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2144113.3333333335, ans=0.125
2023-11-22 23:34:25,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2144180.0, ans=0.025
2023-11-22 23:34:49,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321650
2023-11-22 23:34:53,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2144313.3333333335, ans=0.125
2023-11-22 23:34:55,317 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:35:01,332 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.041e+01 8.419e+01 9.018e+01 9.927e+01 1.306e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-22 23:35:11,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2144380.0, ans=0.125
2023-11-22 23:35:11,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2144380.0, ans=0.05
2023-11-22 23:35:13,818 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9050, loss[loss=0.06658, simple_loss=0.08951, pruned_loss=0.01294, audio_tagging_loss=0.008887, over 15356.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09301, pruned_loss=0.01455, audio_tagging_loss=0.008948, over 3051626.46 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:35:20,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.22 vs. limit=15.0
2023-11-22 23:35:26,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2144446.6666666665, ans=0.125
2023-11-22 23:35:33,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.71 vs. limit=10.0
2023-11-22 23:35:52,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2144646.6666666665, ans=0.0
2023-11-22 23:35:54,703 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321700
2023-11-22 23:36:06,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2144713.3333333335, ans=0.125
2023-11-22 23:36:09,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2144713.3333333335, ans=0.125
2023-11-22 23:36:19,469 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9100, loss[loss=0.08103, simple_loss=0.1133, pruned_loss=0.01595, audio_tagging_loss=0.00844, over 15958.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09299, pruned_loss=0.01447, audio_tagging_loss=0.008928, over 3059001.04 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:36:49,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2144913.3333333335, ans=0.125
2023-11-22 23:36:58,361 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321750
2023-11-22 23:37:00,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=2144980.0, ans=22.5
2023-11-22 23:37:01,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2144980.0, ans=0.0
2023-11-22 23:37:09,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2145046.6666666665, ans=0.2
2023-11-22 23:37:11,333 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.80 vs. limit=15.0
2023-11-22 23:37:11,730 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.202e+01 8.707e+01 9.347e+01 1.081e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 23:37:22,636 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9150, loss[loss=0.08038, simple_loss=0.1009, pruned_loss=0.02068, audio_tagging_loss=0.009248, over 15693.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09279, pruned_loss=0.01438, audio_tagging_loss=0.008956, over 3053596.82 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:37:37,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2145180.0, ans=0.0
2023-11-22 23:37:37,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2145180.0, ans=0.0
2023-11-22 23:37:51,673 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.06 vs. limit=15.0
2023-11-22 23:38:02,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321800
2023-11-22 23:38:04,829 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.41 vs. limit=12.0
2023-11-22 23:38:09,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2145313.3333333335, ans=0.05
2023-11-22 23:38:19,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2145380.0, ans=0.125
2023-11-22 23:38:25,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2145446.6666666665, ans=0.2
2023-11-22 23:38:26,083 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9200, loss[loss=0.06996, simple_loss=0.08874, pruned_loss=0.01694, audio_tagging_loss=0.008646, over 14494.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09212, pruned_loss=0.01415, audio_tagging_loss=0.009075, over 3048527.46 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:38:43,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2145513.3333333335, ans=0.125
2023-11-22 23:38:57,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2145580.0, ans=0.125
2023-11-22 23:39:06,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321850
2023-11-22 23:39:09,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2145646.6666666665, ans=0.2
2023-11-22 23:39:12,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2145646.6666666665, ans=0.0
2023-11-22 23:39:18,715 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.155e+01 8.815e+01 9.566e+01 1.382e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 23:39:30,945 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9250, loss[loss=0.0806, simple_loss=0.1023, pruned_loss=0.02317, audio_tagging_loss=0.006256, over 15502.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09207, pruned_loss=0.01411, audio_tagging_loss=0.009098, over 3054327.05 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:39:31,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2145780.0, ans=0.1
2023-11-22 23:39:32,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2145780.0, ans=0.1
2023-11-22 23:39:38,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2145780.0, ans=0.125
2023-11-22 23:39:38,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.52 vs. limit=10.0
2023-11-22 23:39:45,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2145846.6666666665, ans=0.125
2023-11-22 23:40:05,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2145913.3333333335, ans=0.125
2023-11-22 23:40:10,926 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321900
2023-11-22 23:40:35,064 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9300, loss[loss=0.05939, simple_loss=0.07727, pruned_loss=0.01381, audio_tagging_loss=0.006954, over 15559.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09195, pruned_loss=0.01424, audio_tagging_loss=0.009154, over 3055551.64 frames. ], batch size: 61, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:40:43,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2146113.3333333335, ans=0.1
2023-11-22 23:40:52,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2146180.0, ans=0.125
2023-11-22 23:41:10,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2146246.6666666665, ans=0.125
2023-11-22 23:41:14,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 321950
2023-11-22 23:41:19,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2146313.3333333335, ans=0.2
2023-11-22 23:41:26,761 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.225e+01 8.828e+01 9.380e+01 1.148e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 23:41:37,955 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9350, loss[loss=0.08845, simple_loss=0.1097, pruned_loss=0.02281, audio_tagging_loss=0.01078, over 15477.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09261, pruned_loss=0.0144, audio_tagging_loss=0.009175, over 3053055.99 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:41:58,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2146513.3333333335, ans=0.2
2023-11-22 23:42:17,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322000
2023-11-22 23:42:22,047 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:42:43,213 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9400, loss[loss=0.07225, simple_loss=0.08917, pruned_loss=0.01693, audio_tagging_loss=0.01073, over 14713.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09321, pruned_loss=0.01451, audio_tagging_loss=0.009243, over 3046905.66 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:42:50,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2146780.0, ans=0.0
2023-11-22 23:43:05,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2146846.6666666665, ans=0.125
2023-11-22 23:43:12,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2146913.3333333335, ans=0.125
2023-11-22 23:43:12,798 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.02 vs. limit=15.0
2023-11-22 23:43:18,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=12.0
2023-11-22 23:43:20,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2146980.0, ans=0.125
2023-11-22 23:43:22,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2146980.0, ans=0.05
2023-11-22 23:43:23,200 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322050
2023-11-22 23:43:25,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2146980.0, ans=0.1
2023-11-22 23:43:36,909 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.391e+01 8.949e+01 9.601e+01 1.196e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 23:43:46,457 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:43:49,027 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9450, loss[loss=0.07668, simple_loss=0.1032, pruned_loss=0.01471, audio_tagging_loss=0.01038, over 15853.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09415, pruned_loss=0.01454, audio_tagging_loss=0.009308, over 3049900.20 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:43:49,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2147113.3333333335, ans=0.125
2023-11-22 23:43:50,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2147113.3333333335, ans=0.125
2023-11-22 23:44:00,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.52 vs. limit=15.0
2023-11-22 23:44:12,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0
2023-11-22 23:44:21,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2147246.6666666665, ans=0.125
2023-11-22 23:44:29,027 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322100
2023-11-22 23:44:33,343 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.85 vs. limit=15.0
2023-11-22 23:44:41,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2147380.0, ans=0.125
2023-11-22 23:44:51,303 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.97 vs. limit=22.5
2023-11-22 23:44:52,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2147446.6666666665, ans=0.1
2023-11-22 23:44:53,035 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9500, loss[loss=0.06301, simple_loss=0.08649, pruned_loss=0.01243, audio_tagging_loss=0.007334, over 16066.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09402, pruned_loss=0.0146, audio_tagging_loss=0.009248, over 3047173.08 frames. ], batch size: 64, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:45:01,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2147446.6666666665, ans=0.0
2023-11-22 23:45:11,760 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0
2023-11-22 23:45:31,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2147646.6666666665, ans=0.0
2023-11-22 23:45:33,389 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322150
2023-11-22 23:45:35,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.83 vs. limit=15.0
2023-11-22 23:45:36,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2147646.6666666665, ans=0.125
2023-11-22 23:45:38,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2147646.6666666665, ans=0.125
2023-11-22 23:45:45,479 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.349e+01 8.991e+01 9.634e+01 1.651e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-22 23:45:57,764 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9550, loss[loss=0.08158, simple_loss=0.109, pruned_loss=0.01649, audio_tagging_loss=0.01061, over 15481.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09438, pruned_loss=0.0147, audio_tagging_loss=0.009367, over 3046461.84 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:45:58,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2147780.0, ans=0.125
2023-11-22 23:46:09,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.29 vs. limit=15.0
2023-11-22 23:46:20,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2147846.6666666665, ans=0.0
2023-11-22 23:46:29,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2147913.3333333335, ans=0.1
2023-11-22 23:46:31,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2147913.3333333335, ans=0.0
2023-11-22 23:46:35,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2147980.0, ans=0.0
2023-11-22 23:46:37,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322200
2023-11-22 23:46:37,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2147980.0, ans=0.125
2023-11-22 23:46:43,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2147980.0, ans=0.0
2023-11-22 23:47:03,433 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9600, loss[loss=0.06484, simple_loss=0.08956, pruned_loss=0.01063, audio_tagging_loss=0.009438, over 14745.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09411, pruned_loss=0.01457, audio_tagging_loss=0.009424, over 3049602.20 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:47:06,722 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.66 vs. limit=15.0
2023-11-22 23:47:30,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2148246.6666666665, ans=0.1
2023-11-22 23:47:33,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2148246.6666666665, ans=0.0
2023-11-22 23:47:33,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2148246.6666666665, ans=0.0
2023-11-22 23:47:43,913 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322250
2023-11-22 23:47:51,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2148313.3333333335, ans=0.0
2023-11-22 23:47:57,166 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.150e+01 8.727e+01 9.513e+01 1.269e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 23:48:07,658 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9650, loss[loss=0.08095, simple_loss=0.1106, pruned_loss=0.01774, audio_tagging_loss=0.007925, over 15520.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09305, pruned_loss=0.01432, audio_tagging_loss=0.009474, over 3050491.78 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:48:28,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.82 vs. limit=10.0
2023-11-22 23:48:28,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2148513.3333333335, ans=0.2
2023-11-22 23:48:47,536 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322300
2023-11-22 23:48:47,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2148646.6666666665, ans=0.1
2023-11-22 23:48:48,075 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.80 vs. limit=15.0
2023-11-22 23:48:50,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2148646.6666666665, ans=0.125
2023-11-22 23:49:07,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2148713.3333333335, ans=0.125
2023-11-22 23:49:12,275 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9700, loss[loss=0.07139, simple_loss=0.09162, pruned_loss=0.01374, audio_tagging_loss=0.01184, over 14875.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09323, pruned_loss=0.0144, audio_tagging_loss=0.009354, over 3050975.20 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:49:15,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2148780.0, ans=0.125
2023-11-22 23:49:22,123 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.28 vs. limit=15.0
2023-11-22 23:49:23,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.15 vs. limit=15.0
2023-11-22 23:49:26,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2148846.6666666665, ans=0.0
2023-11-22 23:49:30,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2148846.6666666665, ans=0.125
2023-11-22 23:49:52,129 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322350
2023-11-22 23:49:57,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2148980.0, ans=0.1
2023-11-22 23:50:07,998 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.299e+01 9.052e+01 9.780e+01 1.137e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 23:50:16,518 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9750, loss[loss=0.07548, simple_loss=0.09077, pruned_loss=0.0193, audio_tagging_loss=0.0108, over 14506.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09391, pruned_loss=0.01446, audio_tagging_loss=0.009197, over 3047970.13 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:50:27,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2149180.0, ans=0.0
2023-11-22 23:50:38,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2149180.0, ans=0.0
2023-11-22 23:50:45,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.79 vs. limit=22.5
2023-11-22 23:50:56,590 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322400
2023-11-22 23:51:20,622 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9800, loss[loss=0.06744, simple_loss=0.08679, pruned_loss=0.0164, audio_tagging_loss=0.007646, over 14821.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09418, pruned_loss=0.0146, audio_tagging_loss=0.009095, over 3043035.10 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:51:26,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2149446.6666666665, ans=0.125
2023-11-22 23:51:28,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.51 vs. limit=15.0
2023-11-22 23:51:43,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2149513.3333333335, ans=0.5
2023-11-22 23:52:01,159 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322450
2023-11-22 23:52:05,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2149646.6666666665, ans=0.125
2023-11-22 23:52:16,548 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.583e+01 9.205e+01 9.862e+01 1.172e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-22 23:52:17,924 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:52:25,259 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9850, loss[loss=0.07202, simple_loss=0.09733, pruned_loss=0.01337, audio_tagging_loss=0.009986, over 14899.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09503, pruned_loss=0.01482, audio_tagging_loss=0.008981, over 3036502.34 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:52:45,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2149846.6666666665, ans=0.2
2023-11-22 23:52:49,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2149846.6666666665, ans=0.1
2023-11-22 23:52:55,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2149913.3333333335, ans=0.0
2023-11-22 23:53:06,084 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322500
2023-11-22 23:53:10,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=2149980.0, ans=15.0
2023-11-22 23:53:13,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2149980.0, ans=0.0
2023-11-22 23:53:21,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2150046.6666666665, ans=0.0
2023-11-22 23:53:31,655 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9900, loss[loss=0.06935, simple_loss=0.09087, pruned_loss=0.01558, audio_tagging_loss=0.008332, over 14388.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09511, pruned_loss=0.01483, audio_tagging_loss=0.008988, over 3042599.51 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:53:47,768 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:54:12,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322550
2023-11-22 23:54:27,616 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.162e+01 8.931e+01 9.625e+01 1.127e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 23:54:34,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2150380.0, ans=0.125
2023-11-22 23:54:36,335 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 9950, loss[loss=0.05902, simple_loss=0.07792, pruned_loss=0.011, audio_tagging_loss=0.009056, over 15715.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.0932, pruned_loss=0.01455, audio_tagging_loss=0.009117, over 3042928.33 frames. ], batch size: 61, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:54:42,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2150446.6666666665, ans=0.125
2023-11-22 23:54:52,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2150513.3333333335, ans=0.5
2023-11-22 23:54:53,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2150513.3333333335, ans=0.125
2023-11-22 23:55:13,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2150580.0, ans=0.2
2023-11-22 23:55:15,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2150646.6666666665, ans=0.1
2023-11-22 23:55:16,714 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322600
2023-11-22 23:55:19,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2150646.6666666665, ans=0.125
2023-11-22 23:55:24,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2150646.6666666665, ans=0.125
2023-11-22 23:55:28,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2150713.3333333335, ans=0.125
2023-11-22 23:55:41,022 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10000, loss[loss=0.0827, simple_loss=0.109, pruned_loss=0.01858, audio_tagging_loss=0.009624, over 14694.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09327, pruned_loss=0.01461, audio_tagging_loss=0.009111, over 3036435.25 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:55:42,851 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.07 vs. limit=12.0
2023-11-22 23:55:56,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2150846.6666666665, ans=0.5
2023-11-22 23:56:01,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2150846.6666666665, ans=0.125
2023-11-22 23:56:15,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2150913.3333333335, ans=10.0
2023-11-22 23:56:21,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322650
2023-11-22 23:56:38,548 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.648e+01 8.099e+01 8.716e+01 9.532e+01 1.465e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 23:56:43,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2151046.6666666665, ans=0.1
2023-11-22 23:56:47,894 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10050, loss[loss=0.08059, simple_loss=0.1168, pruned_loss=0.01533, audio_tagging_loss=0.00687, over 15252.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09366, pruned_loss=0.01465, audio_tagging_loss=0.009078, over 3037763.37 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:56:59,162 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:57:00,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2151180.0, ans=0.04949747468305833
2023-11-22 23:57:08,385 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.35 vs. limit=15.0
2023-11-22 23:57:27,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322700
2023-11-22 23:57:42,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2151380.0, ans=0.0
2023-11-22 23:57:52,566 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10100, loss[loss=0.05473, simple_loss=0.07278, pruned_loss=0.008508, audio_tagging_loss=0.009826, over 14965.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09364, pruned_loss=0.01456, audio_tagging_loss=0.009043, over 3037521.09 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:57:53,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2151446.6666666665, ans=0.125
2023-11-22 23:58:00,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2151446.6666666665, ans=15.0
2023-11-22 23:58:07,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2151513.3333333335, ans=0.0
2023-11-22 23:58:08,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2151513.3333333335, ans=0.0
2023-11-22 23:58:10,986 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.55 vs. limit=8.0
2023-11-22 23:58:26,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2151580.0, ans=0.1
2023-11-22 23:58:30,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2151580.0, ans=0.125
2023-11-22 23:58:33,887 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322750
2023-11-22 23:58:44,952 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:58:48,685 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.658e+01 8.081e+01 8.819e+01 9.576e+01 1.489e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 23:58:57,828 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10150, loss[loss=0.08218, simple_loss=0.1095, pruned_loss=0.01827, audio_tagging_loss=0.00916, over 15316.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09486, pruned_loss=0.01491, audio_tagging_loss=0.009032, over 3042474.40 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:59:06,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2151780.0, ans=0.125
2023-11-22 23:59:28,835 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:59:38,982 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322800
2023-11-22 23:59:58,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2152046.6666666665, ans=0.04949747468305833
2023-11-23 00:00:04,821 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10200, loss[loss=0.07725, simple_loss=0.1028, pruned_loss=0.01789, audio_tagging_loss=0.007958, over 14532.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09407, pruned_loss=0.01482, audio_tagging_loss=0.00915, over 3048570.54 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:00:17,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2152180.0, ans=0.125
2023-11-23 00:00:27,270 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:00:34,006 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.05 vs. limit=15.0
2023-11-23 00:00:38,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2152246.6666666665, ans=0.2
2023-11-23 00:00:43,322 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322850
2023-11-23 00:00:59,979 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.282e+01 8.917e+01 9.345e+01 1.171e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 00:01:06,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2152380.0, ans=0.2
2023-11-23 00:01:08,760 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10250, loss[loss=0.06271, simple_loss=0.08646, pruned_loss=0.008988, audio_tagging_loss=0.01049, over 13602.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09372, pruned_loss=0.01483, audio_tagging_loss=0.009167, over 3038245.35 frames. ], batch size: 51, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:01:11,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0
2023-11-23 00:01:16,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2152446.6666666665, ans=0.0
2023-11-23 00:01:17,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2152446.6666666665, ans=0.1
2023-11-23 00:01:19,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2152446.6666666665, ans=0.0
2023-11-23 00:01:37,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2152580.0, ans=0.5
2023-11-23 00:01:49,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322900
2023-11-23 00:01:58,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2152646.6666666665, ans=0.125
2023-11-23 00:02:13,726 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10300, loss[loss=0.07188, simple_loss=0.09085, pruned_loss=0.01615, audio_tagging_loss=0.0103, over 15545.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09314, pruned_loss=0.0147, audio_tagging_loss=0.009325, over 3047656.25 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:02:24,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2152780.0, ans=0.125
2023-11-23 00:02:53,979 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 322950
2023-11-23 00:03:01,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2152980.0, ans=0.0
2023-11-23 00:03:10,238 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.589e+01 8.288e+01 8.993e+01 9.603e+01 1.175e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 00:03:19,090 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10350, loss[loss=0.06465, simple_loss=0.09058, pruned_loss=0.009897, audio_tagging_loss=0.009466, over 15025.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09352, pruned_loss=0.01479, audio_tagging_loss=0.009402, over 3050015.05 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:03:22,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2153113.3333333335, ans=0.125
2023-11-23 00:03:27,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2153113.3333333335, ans=0.125
2023-11-23 00:03:32,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2153180.0, ans=0.1
2023-11-23 00:03:58,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323000
2023-11-23 00:04:01,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2153313.3333333335, ans=0.2
2023-11-23 00:04:04,053 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.80 vs. limit=15.0
2023-11-23 00:04:04,608 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:04:13,426 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:04:23,904 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.37 vs. limit=15.0
2023-11-23 00:04:24,478 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10400, loss[loss=0.05014, simple_loss=0.06524, pruned_loss=0.008835, audio_tagging_loss=0.00869, over 15034.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.093, pruned_loss=0.01472, audio_tagging_loss=0.00945, over 3046897.36 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:04:33,343 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:04:39,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2153513.3333333335, ans=0.0
2023-11-23 00:04:40,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2153513.3333333335, ans=0.125
2023-11-23 00:04:49,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2153580.0, ans=0.0
2023-11-23 00:04:54,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2153580.0, ans=0.04949747468305833
2023-11-23 00:05:05,650 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323050
2023-11-23 00:05:07,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2153646.6666666665, ans=0.125
2023-11-23 00:05:13,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2153646.6666666665, ans=0.1
2023-11-23 00:05:20,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2153713.3333333335, ans=0.2
2023-11-23 00:05:21,770 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 8.434e+01 9.021e+01 9.797e+01 1.156e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 00:05:23,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.65 vs. limit=15.0
2023-11-23 00:05:29,010 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10450, loss[loss=0.07279, simple_loss=0.1021, pruned_loss=0.0157, audio_tagging_loss=0.006065, over 15609.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09268, pruned_loss=0.01444, audio_tagging_loss=0.009445, over 3039954.80 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:05:32,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2153780.0, ans=0.2
2023-11-23 00:06:05,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2153913.3333333335, ans=0.0
2023-11-23 00:06:06,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2153913.3333333335, ans=0.125
2023-11-23 00:06:08,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2153980.0, ans=0.125
2023-11-23 00:06:09,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323100
2023-11-23 00:06:14,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2153980.0, ans=0.0
2023-11-23 00:06:34,663 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10500, loss[loss=0.05479, simple_loss=0.0697, pruned_loss=0.008018, audio_tagging_loss=0.01192, over 14197.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09265, pruned_loss=0.01428, audio_tagging_loss=0.009231, over 3038397.67 frames. ], batch size: 54, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:06:47,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2154180.0, ans=0.0
2023-11-23 00:07:13,614 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.79 vs. limit=10.0
2023-11-23 00:07:14,233 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323150
2023-11-23 00:07:17,123 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.90 vs. limit=15.0
2023-11-23 00:07:26,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2154380.0, ans=0.2
2023-11-23 00:07:32,617 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.185e+01 8.836e+01 9.458e+01 1.353e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 00:07:39,212 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10550, loss[loss=0.0662, simple_loss=0.09091, pruned_loss=0.01238, audio_tagging_loss=0.008362, over 14373.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09208, pruned_loss=0.01425, audio_tagging_loss=0.009185, over 3033966.33 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:08:03,212 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.62 vs. limit=10.0
2023-11-23 00:08:19,421 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323200
2023-11-23 00:08:21,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.22 vs. limit=6.0
2023-11-23 00:08:30,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2154713.3333333335, ans=0.125
2023-11-23 00:08:30,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.27 vs. limit=15.0
2023-11-23 00:08:43,848 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10600, loss[loss=0.07891, simple_loss=0.1055, pruned_loss=0.01768, audio_tagging_loss=0.008505, over 14687.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09301, pruned_loss=0.01437, audio_tagging_loss=0.009069, over 3037362.51 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:08:45,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2154780.0, ans=0.125
2023-11-23 00:09:14,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2154913.3333333335, ans=0.125
2023-11-23 00:09:24,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323250
2023-11-23 00:09:41,921 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 7.982e+01 8.705e+01 9.514e+01 1.291e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-23 00:09:48,203 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10650, loss[loss=0.06578, simple_loss=0.0783, pruned_loss=0.01576, audio_tagging_loss=0.01088, over 14885.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09313, pruned_loss=0.01439, audio_tagging_loss=0.009022, over 3036217.67 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:10:01,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.14 vs. limit=15.0
2023-11-23 00:10:28,637 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323300
2023-11-23 00:10:44,731 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:10:50,025 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0
2023-11-23 00:10:52,978 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10700, loss[loss=0.06531, simple_loss=0.08806, pruned_loss=0.0127, audio_tagging_loss=0.008581, over 14005.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09343, pruned_loss=0.01442, audio_tagging_loss=0.009078, over 3038603.81 frames. ], batch size: 53, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:10:59,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2155446.6666666665, ans=0.125
2023-11-23 00:10:59,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2155446.6666666665, ans=0.0
2023-11-23 00:11:01,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2155446.6666666665, ans=0.0
2023-11-23 00:11:15,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2155513.3333333335, ans=0.0
2023-11-23 00:11:33,530 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323350
2023-11-23 00:11:50,553 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.154e+01 8.901e+01 9.766e+01 1.298e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 00:11:57,405 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10750, loss[loss=0.0681, simple_loss=0.08946, pruned_loss=0.01463, audio_tagging_loss=0.008744, over 15344.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09321, pruned_loss=0.01434, audio_tagging_loss=0.009178, over 3038598.03 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:11:57,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2155780.0, ans=0.0
2023-11-23 00:12:11,934 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.37 vs. limit=15.0
2023-11-23 00:12:12,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2155846.6666666665, ans=0.07
2023-11-23 00:12:16,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2155846.6666666665, ans=0.125
2023-11-23 00:12:37,269 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323400
2023-11-23 00:12:42,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2155980.0, ans=0.0
2023-11-23 00:12:49,916 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.15 vs. limit=12.0
2023-11-23 00:13:00,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2156113.3333333335, ans=0.125
2023-11-23 00:13:01,567 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10800, loss[loss=0.07067, simple_loss=0.09706, pruned_loss=0.01353, audio_tagging_loss=0.008614, over 15301.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09333, pruned_loss=0.01445, audio_tagging_loss=0.009167, over 3046551.00 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:13:01,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2156113.3333333335, ans=0.125
2023-11-23 00:13:11,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2156113.3333333335, ans=0.125
2023-11-23 00:13:41,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323450
2023-11-23 00:13:48,879 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0
2023-11-23 00:13:49,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2156313.3333333335, ans=0.125
2023-11-23 00:13:52,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2156380.0, ans=0.0
2023-11-23 00:14:00,555 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.240e+01 8.825e+01 9.425e+01 1.162e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 00:14:00,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2156380.0, ans=0.125
2023-11-23 00:14:06,736 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10850, loss[loss=0.04597, simple_loss=0.06276, pruned_loss=0.006564, audio_tagging_loss=0.00803, over 13584.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09332, pruned_loss=0.01445, audio_tagging_loss=0.009161, over 3048598.81 frames. ], batch size: 53, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:14:10,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2156446.6666666665, ans=0.125
2023-11-23 00:14:20,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2156513.3333333335, ans=0.125
2023-11-23 00:14:33,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2156580.0, ans=0.125
2023-11-23 00:14:38,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2156580.0, ans=0.125
2023-11-23 00:14:46,697 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323500
2023-11-23 00:14:56,078 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:15:04,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2156713.3333333335, ans=0.0
2023-11-23 00:15:06,841 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:15:10,535 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10900, loss[loss=0.08795, simple_loss=0.1161, pruned_loss=0.01847, audio_tagging_loss=0.01144, over 15237.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09273, pruned_loss=0.01417, audio_tagging_loss=0.0092, over 3047974.40 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:15:19,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2156780.0, ans=0.0
2023-11-23 00:15:40,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2156913.3333333335, ans=0.0
2023-11-23 00:15:51,557 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323550
2023-11-23 00:16:09,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2157046.6666666665, ans=0.05
2023-11-23 00:16:10,512 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.063e+01 8.887e+01 9.492e+01 1.128e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 00:16:15,453 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 10950, loss[loss=0.07156, simple_loss=0.1022, pruned_loss=0.0159, audio_tagging_loss=0.004558, over 15409.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09262, pruned_loss=0.01408, audio_tagging_loss=0.009167, over 3044210.79 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:16:15,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2157113.3333333335, ans=0.125
2023-11-23 00:16:15,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2157113.3333333335, ans=0.2
2023-11-23 00:16:26,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2157113.3333333335, ans=0.125
2023-11-23 00:16:41,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.17 vs. limit=22.5
2023-11-23 00:16:43,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2157246.6666666665, ans=0.125
2023-11-23 00:16:50,337 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.14 vs. limit=15.0
2023-11-23 00:16:55,451 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323600
2023-11-23 00:16:58,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2157313.3333333335, ans=0.0
2023-11-23 00:17:00,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2157313.3333333335, ans=0.1
2023-11-23 00:17:01,122 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.53 vs. limit=12.0
2023-11-23 00:17:05,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2157380.0, ans=0.09899494936611666
2023-11-23 00:17:17,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2157380.0, ans=15.0
2023-11-23 00:17:20,239 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11000, loss[loss=0.068, simple_loss=0.08445, pruned_loss=0.01501, audio_tagging_loss=0.01076, over 15229.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09275, pruned_loss=0.01423, audio_tagging_loss=0.009264, over 3053061.91 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:17:20,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2157446.6666666665, ans=0.1
2023-11-23 00:17:24,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2157446.6666666665, ans=0.0
2023-11-23 00:17:25,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.73 vs. limit=15.0
2023-11-23 00:17:26,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2157446.6666666665, ans=0.0
2023-11-23 00:17:30,168 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:17:48,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2157580.0, ans=0.05
2023-11-23 00:17:58,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2157646.6666666665, ans=0.125
2023-11-23 00:17:59,838 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323650
2023-11-23 00:18:01,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2157646.6666666665, ans=0.0
2023-11-23 00:18:10,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2157713.3333333335, ans=0.2
2023-11-23 00:18:18,923 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.419e+01 9.027e+01 1.014e+02 1.259e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 00:18:24,005 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11050, loss[loss=0.08817, simple_loss=0.1133, pruned_loss=0.02061, audio_tagging_loss=0.01091, over 15693.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09321, pruned_loss=0.01435, audio_tagging_loss=0.009342, over 3045777.67 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:19:03,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.09 vs. limit=10.0
2023-11-23 00:19:04,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323700
2023-11-23 00:19:05,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2157980.0, ans=0.05
2023-11-23 00:19:13,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2157980.0, ans=0.07
2023-11-23 00:19:14,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2158046.6666666665, ans=0.2
2023-11-23 00:19:20,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2158046.6666666665, ans=0.2
2023-11-23 00:19:28,745 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11100, loss[loss=0.03879, simple_loss=0.04457, pruned_loss=0.004661, audio_tagging_loss=0.01184, over 14812.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.092, pruned_loss=0.01419, audio_tagging_loss=0.009552, over 3051745.82 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:19:32,991 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.77 vs. limit=12.0
2023-11-23 00:19:52,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2158180.0, ans=0.2
2023-11-23 00:20:02,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.whiten.whitening_limit, batch_count=2158246.6666666665, ans=12.0
2023-11-23 00:20:08,641 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323750
2023-11-23 00:20:12,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2158313.3333333335, ans=0.125
2023-11-23 00:20:21,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2158380.0, ans=0.1
2023-11-23 00:20:23,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2158380.0, ans=0.0
2023-11-23 00:20:28,950 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.251e+01 8.340e+01 9.049e+01 9.831e+01 1.206e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 00:20:34,615 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11150, loss[loss=0.06691, simple_loss=0.08248, pruned_loss=0.01664, audio_tagging_loss=0.009034, over 15129.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09265, pruned_loss=0.01442, audio_tagging_loss=0.009618, over 3049375.59 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:20:37,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2158446.6666666665, ans=0.125
2023-11-23 00:20:43,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2158446.6666666665, ans=0.05
2023-11-23 00:21:05,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2158580.0, ans=0.125
2023-11-23 00:21:14,166 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323800
2023-11-23 00:21:18,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2158646.6666666665, ans=0.0
2023-11-23 00:21:20,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2158646.6666666665, ans=0.125
2023-11-23 00:21:24,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2158646.6666666665, ans=0.2
2023-11-23 00:21:37,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.82 vs. limit=22.5
2023-11-23 00:21:39,506 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11200, loss[loss=0.0582, simple_loss=0.06962, pruned_loss=0.01072, audio_tagging_loss=0.01267, over 14394.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09291, pruned_loss=0.01435, audio_tagging_loss=0.009718, over 3055095.59 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:21:43,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.58 vs. limit=15.0
2023-11-23 00:21:46,395 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.68 vs. limit=15.0
2023-11-23 00:21:53,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2158846.6666666665, ans=0.125
2023-11-23 00:22:01,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2158846.6666666665, ans=0.0
2023-11-23 00:22:16,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2158913.3333333335, ans=0.125
2023-11-23 00:22:20,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323850
2023-11-23 00:22:31,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2159046.6666666665, ans=0.2
2023-11-23 00:22:39,531 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.459e+01 8.191e+01 8.566e+01 9.520e+01 1.333e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-23 00:22:44,552 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11250, loss[loss=0.05434, simple_loss=0.07744, pruned_loss=0.007515, audio_tagging_loss=0.008109, over 14572.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09197, pruned_loss=0.01421, audio_tagging_loss=0.009663, over 3048295.73 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:23:04,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2159180.0, ans=0.125
2023-11-23 00:23:21,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2159246.6666666665, ans=0.0
2023-11-23 00:23:25,462 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323900
2023-11-23 00:23:35,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2159380.0, ans=0.0
2023-11-23 00:23:40,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2159380.0, ans=0.125
2023-11-23 00:23:50,758 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11300, loss[loss=0.07612, simple_loss=0.1092, pruned_loss=0.01438, audio_tagging_loss=0.007156, over 15831.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09196, pruned_loss=0.0142, audio_tagging_loss=0.009496, over 3056010.43 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:24:19,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2159580.0, ans=0.125
2023-11-23 00:24:25,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2159580.0, ans=0.125
2023-11-23 00:24:29,380 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 323950
2023-11-23 00:24:33,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2159646.6666666665, ans=0.0
2023-11-23 00:24:44,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2159713.3333333335, ans=0.0
2023-11-23 00:24:51,664 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.358e+01 9.001e+01 9.797e+01 1.218e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 00:24:55,562 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11350, loss[loss=0.08487, simple_loss=0.1064, pruned_loss=0.02182, audio_tagging_loss=0.009824, over 14452.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09318, pruned_loss=0.01454, audio_tagging_loss=0.00926, over 3060483.04 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:25:26,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2159913.3333333335, ans=0.0
2023-11-23 00:25:30,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2159913.3333333335, ans=0.125
2023-11-23 00:25:34,374 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:25:36,762 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324000
2023-11-23 00:25:36,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2159980.0, ans=0.125
2023-11-23 00:25:43,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2159980.0, ans=0.1
2023-11-23 00:26:03,547 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11400, loss[loss=0.06914, simple_loss=0.0901, pruned_loss=0.01476, audio_tagging_loss=0.009331, over 15072.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09374, pruned_loss=0.01465, audio_tagging_loss=0.009252, over 3062574.33 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:26:39,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.23 vs. limit=12.0
2023-11-23 00:26:44,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324050
2023-11-23 00:26:51,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2160313.3333333335, ans=0.125
2023-11-23 00:27:05,065 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.089e+01 8.744e+01 9.515e+01 1.376e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 00:27:10,220 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11450, loss[loss=0.08325, simple_loss=0.1099, pruned_loss=0.02132, audio_tagging_loss=0.006973, over 16437.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09457, pruned_loss=0.01494, audio_tagging_loss=0.009135, over 3060077.82 frames. ], batch size: 62, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:27:22,578 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.54 vs. limit=22.5
2023-11-23 00:27:35,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2160580.0, ans=0.125
2023-11-23 00:27:47,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.01 vs. limit=15.0
2023-11-23 00:27:48,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2160646.6666666665, ans=0.1
2023-11-23 00:27:50,522 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324100
2023-11-23 00:28:16,252 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11500, loss[loss=0.05799, simple_loss=0.08207, pruned_loss=0.009706, audio_tagging_loss=0.007246, over 14544.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09435, pruned_loss=0.01486, audio_tagging_loss=0.00912, over 3060864.99 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:28:18,224 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.32 vs. limit=22.5
2023-11-23 00:28:20,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2160780.0, ans=0.125
2023-11-23 00:28:45,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2160913.3333333335, ans=0.125
2023-11-23 00:28:52,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2160913.3333333335, ans=0.125
2023-11-23 00:28:57,818 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324150
2023-11-23 00:29:18,298 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 7.992e+01 8.677e+01 9.342e+01 1.172e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 00:29:18,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2161046.6666666665, ans=0.2
2023-11-23 00:29:22,206 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11550, loss[loss=0.07072, simple_loss=0.0911, pruned_loss=0.01465, audio_tagging_loss=0.01051, over 14340.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09402, pruned_loss=0.01468, audio_tagging_loss=0.009127, over 3060422.93 frames. ], batch size: 53, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:29:51,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2161246.6666666665, ans=0.05
2023-11-23 00:30:02,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2161313.3333333335, ans=0.0
2023-11-23 00:30:03,369 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:30:04,628 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324200
2023-11-23 00:30:08,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2161313.3333333335, ans=0.0
2023-11-23 00:30:16,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2161380.0, ans=10.0
2023-11-23 00:30:22,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2161380.0, ans=0.125
2023-11-23 00:30:29,734 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11600, loss[loss=0.06759, simple_loss=0.09012, pruned_loss=0.01385, audio_tagging_loss=0.008683, over 16367.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.0944, pruned_loss=0.01471, audio_tagging_loss=0.0091, over 3057598.18 frames. ], batch size: 61, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:30:31,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2161446.6666666665, ans=0.0
2023-11-23 00:30:42,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2161446.6666666665, ans=0.0
2023-11-23 00:31:10,698 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324250
2023-11-23 00:31:21,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2161713.3333333335, ans=0.125
2023-11-23 00:31:25,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2161713.3333333335, ans=0.0
2023-11-23 00:31:32,553 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.566e+01 8.394e+01 8.884e+01 9.610e+01 1.253e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 00:31:36,385 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11650, loss[loss=0.05492, simple_loss=0.06767, pruned_loss=0.01031, audio_tagging_loss=0.01078, over 15303.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.0936, pruned_loss=0.0147, audio_tagging_loss=0.009193, over 3048815.64 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:31:40,655 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.70 vs. limit=15.0
2023-11-23 00:31:42,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.70 vs. limit=22.5
2023-11-23 00:32:00,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2161913.3333333335, ans=0.0
2023-11-23 00:32:05,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2161913.3333333335, ans=0.0
2023-11-23 00:32:11,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2161913.3333333335, ans=0.1
2023-11-23 00:32:17,555 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324300
2023-11-23 00:32:31,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2162046.6666666665, ans=0.125
2023-11-23 00:32:33,645 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.06 vs. limit=22.5
2023-11-23 00:32:37,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2162046.6666666665, ans=0.0
2023-11-23 00:32:40,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2162113.3333333335, ans=0.125
2023-11-23 00:32:42,006 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11700, loss[loss=0.07234, simple_loss=0.09477, pruned_loss=0.016, audio_tagging_loss=0.008963, over 15143.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09262, pruned_loss=0.01458, audio_tagging_loss=0.009309, over 3049023.09 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:32:54,843 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:33:14,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2162246.6666666665, ans=0.2
2023-11-23 00:33:21,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.73 vs. limit=10.0
2023-11-23 00:33:23,482 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324350
2023-11-23 00:33:33,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff2.min_abs, batch_count=2162380.0, ans=0.1
2023-11-23 00:33:43,253 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.270e+01 8.836e+01 9.259e+01 1.366e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 00:33:46,982 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11750, loss[loss=0.08143, simple_loss=0.1162, pruned_loss=0.01574, audio_tagging_loss=0.007574, over 15184.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09313, pruned_loss=0.01455, audio_tagging_loss=0.009215, over 3044181.55 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:33:48,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2162446.6666666665, ans=0.0
2023-11-23 00:34:11,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2162513.3333333335, ans=0.125
2023-11-23 00:34:14,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2162580.0, ans=0.125
2023-11-23 00:34:15,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2162580.0, ans=10.0
2023-11-23 00:34:18,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2162580.0, ans=0.0
2023-11-23 00:34:25,276 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.42 vs. limit=22.5
2023-11-23 00:34:28,639 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324400
2023-11-23 00:34:54,180 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11800, loss[loss=0.08357, simple_loss=0.1127, pruned_loss=0.01762, audio_tagging_loss=0.009592, over 15680.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09407, pruned_loss=0.01488, audio_tagging_loss=0.009211, over 3042565.16 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:34:54,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2162780.0, ans=0.0
2023-11-23 00:35:16,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2162846.6666666665, ans=0.0
2023-11-23 00:35:23,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2162913.3333333335, ans=0.95
2023-11-23 00:35:34,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324450
2023-11-23 00:35:47,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2163046.6666666665, ans=0.125
2023-11-23 00:35:57,196 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.225e+01 9.023e+01 9.525e+01 1.272e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 00:35:57,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2163046.6666666665, ans=0.125
2023-11-23 00:35:59,724 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11850, loss[loss=0.06419, simple_loss=0.08413, pruned_loss=0.01228, audio_tagging_loss=0.009841, over 13669.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09281, pruned_loss=0.01481, audio_tagging_loss=0.009293, over 3046672.73 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:36:21,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2163180.0, ans=0.2
2023-11-23 00:36:29,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2163246.6666666665, ans=0.125
2023-11-23 00:36:31,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.12 vs. limit=5.0
2023-11-23 00:36:32,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2163246.6666666665, ans=0.125
2023-11-23 00:36:40,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324500
2023-11-23 00:36:46,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2163313.3333333335, ans=0.125
2023-11-23 00:36:59,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2163380.0, ans=0.125
2023-11-23 00:37:04,534 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11900, loss[loss=0.07274, simple_loss=0.1088, pruned_loss=0.01121, audio_tagging_loss=0.007122, over 16372.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09265, pruned_loss=0.01457, audio_tagging_loss=0.009355, over 3046350.13 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:37:37,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2163580.0, ans=0.0
2023-11-23 00:37:45,415 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324550
2023-11-23 00:38:01,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2163713.3333333335, ans=0.0
2023-11-23 00:38:08,411 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.374e+01 8.930e+01 9.704e+01 2.155e+02, threshold=1.786e+02, percent-clipped=1.0
2023-11-23 00:38:10,888 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 11950, loss[loss=0.08635, simple_loss=0.1122, pruned_loss=0.02013, audio_tagging_loss=0.01012, over 15382.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09273, pruned_loss=0.01467, audio_tagging_loss=0.009496, over 3041218.70 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:38:12,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.54 vs. limit=15.0
2023-11-23 00:38:17,454 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:38:46,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2163913.3333333335, ans=0.1
2023-11-23 00:38:46,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.98 vs. limit=10.0
2023-11-23 00:38:49,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324600
2023-11-23 00:38:55,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2163980.0, ans=22.5
2023-11-23 00:38:55,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2163980.0, ans=0.125
2023-11-23 00:39:14,383 INFO [train_asr.py:1221] (2/4) Epoch 27, batch 12000, loss[loss=0.06632, simple_loss=0.08705, pruned_loss=0.01153, audio_tagging_loss=0.01126, over 15967.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09292, pruned_loss=0.01466, audio_tagging_loss=0.009659, over 3036765.16 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:39:14,384 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 00:39:33,247 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.6009, 3.0097, 3.1462, 2.7286], device='cuda:2')
2023-11-23 00:39:47,354 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.2827, 2.9409, 3.2014, 3.0944, 3.6926, 3.7701, 3.2372, 3.2380],
       device='cuda:2')
2023-11-23 00:39:52,223 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4794, 3.7605, 2.6663, 3.7800], device='cuda:2')
2023-11-23 00:39:57,338 INFO [train_asr.py:1253] (2/4) Epoch 27, validation: loss=0.05869, simple_loss=0.05138, pruned_loss=0.005099, audio_tagging_loss=0.0279, over 4681554.00 frames. 
2023-11-23 00:39:57,339 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 00:40:14,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2164180.0, ans=0.0
2023-11-23 00:40:15,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2164180.0, ans=0.0
2023-11-23 00:40:17,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.08 vs. limit=10.0
2023-11-23 00:40:19,588 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.72 vs. limit=15.0
2023-11-23 00:41:02,699 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 0, loss[loss=0.07888, simple_loss=0.09164, pruned_loss=0.01388, audio_tagging_loss=0.01918, over 15176.00 frames. ], tot_loss[loss=0.07888, simple_loss=0.09164, pruned_loss=0.01388, audio_tagging_loss=0.01918, over 15176.00 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:41:02,700 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 00:41:39,765 INFO [train_asr.py:1253] (2/4) Epoch 28, validation: loss=0.0583, simple_loss=0.05139, pruned_loss=0.005161, audio_tagging_loss=0.02744, over 4681554.00 frames. 
2023-11-23 00:41:39,766 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 00:41:47,380 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324650
2023-11-23 00:42:09,901 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.350e+01 8.610e+01 9.593e+01 1.048e+02 1.343e+02, threshold=1.919e+02, percent-clipped=0.0
2023-11-23 00:42:31,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2164546.6666666665, ans=0.1
2023-11-23 00:42:34,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2164546.6666666665, ans=0.125
2023-11-23 00:42:43,119 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 50, loss[loss=0.08758, simple_loss=0.1073, pruned_loss=0.01699, audio_tagging_loss=0.01695, over 14953.00 frames. ], tot_loss[loss=0.08023, simple_loss=0.09633, pruned_loss=0.01443, audio_tagging_loss=0.01764, over 684267.84 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:42:50,555 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324700
2023-11-23 00:43:11,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2164746.6666666665, ans=0.125
2023-11-23 00:43:28,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2164813.3333333335, ans=0.125
2023-11-23 00:43:33,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.74 vs. limit=15.0
2023-11-23 00:43:46,719 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 100, loss[loss=0.07895, simple_loss=0.08892, pruned_loss=0.01718, audio_tagging_loss=0.01731, over 15092.00 frames. ], tot_loss[loss=0.0789, simple_loss=0.09458, pruned_loss=0.01451, audio_tagging_loss=0.01711, over 1209940.89 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:43:48,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2164946.6666666665, ans=0.04949747468305833
2023-11-23 00:43:54,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324750
2023-11-23 00:44:10,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2165013.3333333335, ans=0.0
2023-11-23 00:44:17,222 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.407e+01 9.043e+01 9.747e+01 1.056e+02 1.211e+02, threshold=1.949e+02, percent-clipped=0.0
2023-11-23 00:44:17,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2165080.0, ans=0.125
2023-11-23 00:44:37,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2165213.3333333335, ans=0.0
2023-11-23 00:44:46,492 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:44:49,967 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 150, loss[loss=0.07004, simple_loss=0.0959, pruned_loss=0.01262, audio_tagging_loss=0.009476, over 14583.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09429, pruned_loss=0.01423, audio_tagging_loss=0.01522, over 1613979.15 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:44:58,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324800
2023-11-23 00:45:33,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2165480.0, ans=0.125
2023-11-23 00:45:35,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2165480.0, ans=0.125
2023-11-23 00:45:51,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2165546.6666666665, ans=0.125
2023-11-23 00:45:55,027 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 200, loss[loss=0.07422, simple_loss=0.1042, pruned_loss=0.01387, audio_tagging_loss=0.008261, over 15597.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09354, pruned_loss=0.01421, audio_tagging_loss=0.01341, over 1937996.73 frames. ], batch size: 61, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:45:58,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2165613.3333333335, ans=0.0
2023-11-23 00:46:02,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324850
2023-11-23 00:46:16,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2165680.0, ans=0.125
2023-11-23 00:46:25,064 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.466e+01 9.149e+01 9.933e+01 1.935e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 00:46:47,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2165880.0, ans=0.125
2023-11-23 00:46:58,571 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 250, loss[loss=0.07466, simple_loss=0.08844, pruned_loss=0.018, audio_tagging_loss=0.01244, over 13865.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09441, pruned_loss=0.01452, audio_tagging_loss=0.01206, over 2182449.20 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:47:03,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2165946.6666666665, ans=0.2
2023-11-23 00:47:06,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324900
2023-11-23 00:47:15,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2166013.3333333335, ans=0.0
2023-11-23 00:47:17,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.40 vs. limit=15.0
2023-11-23 00:47:26,998 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.63 vs. limit=15.0
2023-11-23 00:47:31,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2166080.0, ans=0.125
2023-11-23 00:47:31,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2166080.0, ans=0.1
2023-11-23 00:47:56,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2166213.3333333335, ans=0.2
2023-11-23 00:48:03,087 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 300, loss[loss=0.05402, simple_loss=0.06431, pruned_loss=0.008974, audio_tagging_loss=0.01289, over 14818.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09468, pruned_loss=0.0148, audio_tagging_loss=0.01128, over 2368767.88 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:48:03,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2166280.0, ans=0.0
2023-11-23 00:48:11,778 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 324950
2023-11-23 00:48:34,180 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.439e+01 8.323e+01 9.001e+01 1.002e+02 1.826e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 00:48:58,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2166546.6666666665, ans=0.025
2023-11-23 00:49:09,305 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 350, loss[loss=0.06375, simple_loss=0.08821, pruned_loss=0.01045, audio_tagging_loss=0.009188, over 15256.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09403, pruned_loss=0.01459, audio_tagging_loss=0.01069, over 2518456.26 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 00:49:16,959 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325000
2023-11-23 00:49:17,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=2166613.3333333335, ans=15.0
2023-11-23 00:49:37,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2166746.6666666665, ans=0.125
2023-11-23 00:49:42,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2166746.6666666665, ans=0.0
2023-11-23 00:49:52,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2166813.3333333335, ans=0.125
2023-11-23 00:49:59,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2166880.0, ans=10.0
2023-11-23 00:50:12,992 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 400, loss[loss=0.07204, simple_loss=0.0927, pruned_loss=0.01637, audio_tagging_loss=0.009322, over 15216.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09219, pruned_loss=0.01434, audio_tagging_loss=0.01036, over 2633986.84 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:50:16,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.48 vs. limit=15.0
2023-11-23 00:50:20,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325050
2023-11-23 00:50:24,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2167013.3333333335, ans=0.125
2023-11-23 00:50:32,114 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-23 00:50:38,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2167080.0, ans=0.125
2023-11-23 00:50:38,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2167080.0, ans=0.125
2023-11-23 00:50:45,768 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.175e+01 8.785e+01 9.638e+01 1.202e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 00:51:00,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2167146.6666666665, ans=0.1
2023-11-23 00:51:03,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2167213.3333333335, ans=0.125
2023-11-23 00:51:17,312 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 450, loss[loss=0.05977, simple_loss=0.08281, pruned_loss=0.01026, audio_tagging_loss=0.008108, over 14425.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09243, pruned_loss=0.0143, audio_tagging_loss=0.01008, over 2725714.46 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:51:23,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2167280.0, ans=0.2
2023-11-23 00:51:23,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2167280.0, ans=0.1
2023-11-23 00:51:25,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325100
2023-11-23 00:51:26,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2167280.0, ans=0.2
2023-11-23 00:51:49,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2167413.3333333335, ans=0.2
2023-11-23 00:52:01,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2167480.0, ans=0.0
2023-11-23 00:52:02,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2167480.0, ans=0.09899494936611666
2023-11-23 00:52:05,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.62 vs. limit=15.0
2023-11-23 00:52:23,259 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 500, loss[loss=0.06516, simple_loss=0.08044, pruned_loss=0.01651, audio_tagging_loss=0.008431, over 14839.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.0921, pruned_loss=0.01416, audio_tagging_loss=0.009896, over 2798984.54 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:52:30,627 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325150
2023-11-23 00:52:54,606 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.145e+01 8.686e+01 9.449e+01 1.268e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 00:53:09,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2167813.3333333335, ans=0.2
2023-11-23 00:53:27,165 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 550, loss[loss=0.09673, simple_loss=0.1261, pruned_loss=0.02578, audio_tagging_loss=0.007873, over 16466.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09193, pruned_loss=0.0142, audio_tagging_loss=0.009795, over 2854942.60 frames. ], batch size: 60, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:53:34,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325200
2023-11-23 00:53:42,961 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:53:44,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2168013.3333333335, ans=0.2
2023-11-23 00:53:47,130 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.14 vs. limit=6.0
2023-11-23 00:54:29,145 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.35 vs. limit=12.0
2023-11-23 00:54:32,738 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 600, loss[loss=0.07473, simple_loss=0.1005, pruned_loss=0.01679, audio_tagging_loss=0.0077, over 15605.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09216, pruned_loss=0.01423, audio_tagging_loss=0.009681, over 2898436.94 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:54:38,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2168280.0, ans=0.0
2023-11-23 00:54:40,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325250
2023-11-23 00:54:51,671 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.38 vs. limit=15.0
2023-11-23 00:55:01,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2168413.3333333335, ans=0.125
2023-11-23 00:55:04,339 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.119e+01 8.698e+01 9.405e+01 1.451e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-23 00:55:13,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=15.0
2023-11-23 00:55:29,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2168546.6666666665, ans=0.125
2023-11-23 00:55:36,609 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 650, loss[loss=0.06096, simple_loss=0.07075, pruned_loss=0.0129, audio_tagging_loss=0.01269, over 14788.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09196, pruned_loss=0.01418, audio_tagging_loss=0.009616, over 2931589.91 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:55:44,107 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325300
2023-11-23 00:56:05,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2168746.6666666665, ans=0.125
2023-11-23 00:56:08,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.45 vs. limit=15.0
2023-11-23 00:56:37,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2168880.0, ans=0.0
2023-11-23 00:56:40,462 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 700, loss[loss=0.06343, simple_loss=0.08757, pruned_loss=0.009879, audio_tagging_loss=0.009761, over 15098.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.0911, pruned_loss=0.01406, audio_tagging_loss=0.009677, over 2957718.55 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:56:47,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2168946.6666666665, ans=0.125
2023-11-23 00:56:47,986 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325350
2023-11-23 00:57:00,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.77 vs. limit=15.0
2023-11-23 00:57:03,677 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:57:13,020 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.373e+01 9.027e+01 9.909e+01 1.159e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 00:57:22,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2169146.6666666665, ans=0.0
2023-11-23 00:57:27,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2169146.6666666665, ans=0.04949747468305833
2023-11-23 00:57:43,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2169280.0, ans=0.025
2023-11-23 00:57:44,763 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 750, loss[loss=0.05764, simple_loss=0.07568, pruned_loss=0.01036, audio_tagging_loss=0.009428, over 15411.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09147, pruned_loss=0.01411, audio_tagging_loss=0.009601, over 2982090.36 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:57:46,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2169280.0, ans=0.2
2023-11-23 00:57:52,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325400
2023-11-23 00:58:17,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.16 vs. limit=22.5
2023-11-23 00:58:49,715 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 800, loss[loss=0.07015, simple_loss=0.09554, pruned_loss=0.01273, audio_tagging_loss=0.009655, over 15788.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09257, pruned_loss=0.0143, audio_tagging_loss=0.009466, over 2993535.00 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:58:57,592 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325450
2023-11-23 00:59:00,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2169613.3333333335, ans=0.07
2023-11-23 00:59:01,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2169680.0, ans=0.5
2023-11-23 00:59:08,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2169680.0, ans=0.04949747468305833
2023-11-23 00:59:11,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.51 vs. limit=15.0
2023-11-23 00:59:21,169 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.947e+01 8.109e+01 8.929e+01 9.816e+01 1.280e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 00:59:29,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2169813.3333333335, ans=0.125
2023-11-23 00:59:31,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.94 vs. limit=15.0
2023-11-23 00:59:53,404 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:59:54,377 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 850, loss[loss=0.06007, simple_loss=0.07651, pruned_loss=0.01168, audio_tagging_loss=0.01014, over 15386.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09173, pruned_loss=0.01411, audio_tagging_loss=0.009495, over 3006642.15 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:00:01,694 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325500
2023-11-23 01:00:06,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2170013.3333333335, ans=0.0
2023-11-23 01:00:24,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2170080.0, ans=0.0
2023-11-23 01:00:35,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2170146.6666666665, ans=0.125
2023-11-23 01:00:35,651 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.39 vs. limit=15.0
2023-11-23 01:00:39,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.59 vs. limit=15.0
2023-11-23 01:00:47,016 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:00:57,815 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 900, loss[loss=0.07537, simple_loss=0.1099, pruned_loss=0.01411, audio_tagging_loss=0.006332, over 16121.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09283, pruned_loss=0.01424, audio_tagging_loss=0.009543, over 3015230.52 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:01:06,215 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325550
2023-11-23 01:01:20,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2170346.6666666665, ans=0.125
2023-11-23 01:01:22,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2170346.6666666665, ans=0.125
2023-11-23 01:01:32,253 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.700e+01 8.072e+01 8.678e+01 9.441e+01 1.146e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 01:01:51,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2170546.6666666665, ans=0.0
2023-11-23 01:01:54,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=2170546.6666666665, ans=8.0
2023-11-23 01:01:56,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2170546.6666666665, ans=0.1
2023-11-23 01:02:01,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2170613.3333333335, ans=0.125
2023-11-23 01:02:02,730 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 950, loss[loss=0.06835, simple_loss=0.09315, pruned_loss=0.01071, audio_tagging_loss=0.01107, over 15177.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09319, pruned_loss=0.01431, audio_tagging_loss=0.009429, over 3021785.48 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:02:08,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2170613.3333333335, ans=0.125
2023-11-23 01:02:09,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2170613.3333333335, ans=0.125
2023-11-23 01:02:11,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325600
2023-11-23 01:02:29,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2170746.6666666665, ans=0.07
2023-11-23 01:02:47,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2170813.3333333335, ans=0.125
2023-11-23 01:02:50,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2170813.3333333335, ans=0.0
2023-11-23 01:02:50,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2170813.3333333335, ans=0.0
2023-11-23 01:02:58,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.00 vs. limit=15.0
2023-11-23 01:03:08,277 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1000, loss[loss=0.05602, simple_loss=0.06933, pruned_loss=0.009961, audio_tagging_loss=0.0114, over 14247.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09205, pruned_loss=0.01419, audio_tagging_loss=0.009372, over 3022698.43 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:03:16,388 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325650
2023-11-23 01:03:25,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2171013.3333333335, ans=0.125
2023-11-23 01:03:31,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2171013.3333333335, ans=0.125
2023-11-23 01:03:34,966 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:03:35,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2171080.0, ans=0.125
2023-11-23 01:03:41,020 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.239e+01 8.945e+01 9.781e+01 1.255e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 01:03:52,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2171146.6666666665, ans=0.1
2023-11-23 01:04:07,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2171213.3333333335, ans=0.125
2023-11-23 01:04:11,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2171280.0, ans=0.125
2023-11-23 01:04:11,924 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1050, loss[loss=0.07523, simple_loss=0.1047, pruned_loss=0.01642, audio_tagging_loss=0.006471, over 15223.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09138, pruned_loss=0.01398, audio_tagging_loss=0.009259, over 3022588.60 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:04:19,847 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325700
2023-11-23 01:04:42,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2171413.3333333335, ans=0.0
2023-11-23 01:04:55,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2171480.0, ans=0.1
2023-11-23 01:04:59,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2171480.0, ans=0.125
2023-11-23 01:05:09,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2171546.6666666665, ans=0.0
2023-11-23 01:05:15,394 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1100, loss[loss=0.07271, simple_loss=0.09933, pruned_loss=0.01416, audio_tagging_loss=0.008888, over 14496.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09081, pruned_loss=0.01394, audio_tagging_loss=0.009258, over 3020538.27 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:05:19,834 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:05:23,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325750
2023-11-23 01:05:37,856 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.03 vs. limit=22.5
2023-11-23 01:05:48,720 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.331e+01 9.113e+01 9.844e+01 1.667e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-23 01:05:52,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2171813.3333333335, ans=0.0
2023-11-23 01:05:57,176 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:06:03,232 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:06:12,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2171880.0, ans=0.125
2023-11-23 01:06:20,014 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1150, loss[loss=0.05637, simple_loss=0.06778, pruned_loss=0.01299, audio_tagging_loss=0.009493, over 14755.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09067, pruned_loss=0.01421, audio_tagging_loss=0.009291, over 3019487.53 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:06:27,729 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325800
2023-11-23 01:06:29,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2171946.6666666665, ans=0.2
2023-11-23 01:06:52,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2172080.0, ans=0.0
2023-11-23 01:06:54,496 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.21 vs. limit=22.5
2023-11-23 01:07:24,593 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1200, loss[loss=0.07748, simple_loss=0.1023, pruned_loss=0.01542, audio_tagging_loss=0.01092, over 15318.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09095, pruned_loss=0.01428, audio_tagging_loss=0.009273, over 3025873.04 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:07:31,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0
2023-11-23 01:07:32,251 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325850
2023-11-23 01:07:47,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2172346.6666666665, ans=10.0
2023-11-23 01:07:57,976 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.204e+01 8.430e+01 8.952e+01 9.650e+01 2.402e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 01:08:26,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2172546.6666666665, ans=0.1
2023-11-23 01:08:29,009 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1250, loss[loss=0.04329, simple_loss=0.05349, pruned_loss=0.007808, audio_tagging_loss=0.008737, over 15078.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09098, pruned_loss=0.01419, audio_tagging_loss=0.009171, over 3031793.04 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:08:37,158 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325900
2023-11-23 01:08:48,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2172680.0, ans=0.2
2023-11-23 01:08:52,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2172680.0, ans=0.125
2023-11-23 01:08:54,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2172746.6666666665, ans=0.125
2023-11-23 01:08:58,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2172746.6666666665, ans=0.2
2023-11-23 01:09:34,083 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1300, loss[loss=0.0578, simple_loss=0.07229, pruned_loss=0.01247, audio_tagging_loss=0.009189, over 14538.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09083, pruned_loss=0.01409, audio_tagging_loss=0.009086, over 3026356.56 frames. ], batch size: 55, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:09:39,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.38 vs. limit=22.5
2023-11-23 01:09:41,723 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 325950
2023-11-23 01:10:04,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2173080.0, ans=0.125
2023-11-23 01:10:08,885 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.443e+01 8.136e+01 8.697e+01 9.484e+01 1.452e+02, threshold=1.739e+02, percent-clipped=1.0
2023-11-23 01:10:10,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2173080.0, ans=0.1
2023-11-23 01:10:38,134 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1350, loss[loss=0.05474, simple_loss=0.06877, pruned_loss=0.01073, audio_tagging_loss=0.009622, over 15585.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09096, pruned_loss=0.01408, audio_tagging_loss=0.009202, over 3034832.53 frames. ], batch size: 61, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:10:38,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2173280.0, ans=0.2
2023-11-23 01:10:45,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326000
2023-11-23 01:10:54,950 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.61 vs. limit=22.5
2023-11-23 01:11:00,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2173346.6666666665, ans=0.1
2023-11-23 01:11:02,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2173346.6666666665, ans=0.1
2023-11-23 01:11:06,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2173413.3333333335, ans=0.125
2023-11-23 01:11:07,901 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.47 vs. limit=15.0
2023-11-23 01:11:13,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2173413.3333333335, ans=0.125
2023-11-23 01:11:25,632 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:11:36,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.33 vs. limit=15.0
2023-11-23 01:11:42,780 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1400, loss[loss=0.05667, simple_loss=0.06369, pruned_loss=0.01279, audio_tagging_loss=0.01204, over 13873.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09096, pruned_loss=0.01379, audio_tagging_loss=0.009216, over 3040533.34 frames. ], batch size: 55, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:11:50,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326050
2023-11-23 01:12:04,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.26 vs. limit=15.0
2023-11-23 01:12:17,026 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.070e+01 8.559e+01 9.321e+01 1.215e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-23 01:12:22,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2173813.3333333335, ans=0.0
2023-11-23 01:12:32,663 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.15 vs. limit=15.0
2023-11-23 01:12:37,415 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.00 vs. limit=12.0
2023-11-23 01:12:47,049 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1450, loss[loss=0.07904, simple_loss=0.1053, pruned_loss=0.01499, audio_tagging_loss=0.01137, over 14728.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09168, pruned_loss=0.0141, audio_tagging_loss=0.00922, over 3046357.75 frames. ], batch size: 53, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:12:54,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326100
2023-11-23 01:13:04,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2174013.3333333335, ans=0.07
2023-11-23 01:13:07,483 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.04 vs. limit=6.0
2023-11-23 01:13:15,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.86 vs. limit=15.0
2023-11-23 01:13:22,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2174080.0, ans=0.1
2023-11-23 01:13:25,220 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.33 vs. limit=15.0
2023-11-23 01:13:26,166 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:13:50,379 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1500, loss[loss=0.06812, simple_loss=0.08874, pruned_loss=0.01205, audio_tagging_loss=0.0117, over 15052.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09202, pruned_loss=0.01417, audio_tagging_loss=0.009292, over 3042377.43 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:13:57,958 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326150
2023-11-23 01:14:16,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2174413.3333333335, ans=0.0
2023-11-23 01:14:26,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.046e+01 8.933e+01 9.681e+01 1.186e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 01:14:37,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.50 vs. limit=15.0
2023-11-23 01:14:39,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2174480.0, ans=0.0
2023-11-23 01:14:55,304 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1550, loss[loss=0.06092, simple_loss=0.07671, pruned_loss=0.008878, audio_tagging_loss=0.01369, over 14771.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09217, pruned_loss=0.01426, audio_tagging_loss=0.00943, over 3041948.98 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:15:02,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.08 vs. limit=15.0
2023-11-23 01:15:04,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326200
2023-11-23 01:15:12,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2174680.0, ans=0.125
2023-11-23 01:15:17,477 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:15:31,587 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.73 vs. limit=10.0
2023-11-23 01:15:49,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2174880.0, ans=0.125
2023-11-23 01:16:02,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2174946.6666666665, ans=0.05
2023-11-23 01:16:03,425 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1600, loss[loss=0.05993, simple_loss=0.08801, pruned_loss=0.007341, audio_tagging_loss=0.008586, over 14998.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09293, pruned_loss=0.01428, audio_tagging_loss=0.009459, over 3043896.12 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:16:03,768 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:16:11,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326250
2023-11-23 01:16:15,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2175013.3333333335, ans=0.125
2023-11-23 01:16:16,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2175013.3333333335, ans=0.1
2023-11-23 01:16:21,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2175013.3333333335, ans=0.1
2023-11-23 01:16:22,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2175013.3333333335, ans=0.07
2023-11-23 01:16:31,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2175080.0, ans=0.125
2023-11-23 01:16:37,599 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.240e+01 8.854e+01 9.452e+01 1.276e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 01:16:50,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2175146.6666666665, ans=0.125
2023-11-23 01:16:55,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2175213.3333333335, ans=0.2
2023-11-23 01:17:07,126 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1650, loss[loss=0.06552, simple_loss=0.08778, pruned_loss=0.01044, audio_tagging_loss=0.01118, over 15559.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09223, pruned_loss=0.01415, audio_tagging_loss=0.009531, over 3042372.93 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:17:12,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=2175280.0, ans=15.0
2023-11-23 01:17:14,770 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326300
2023-11-23 01:17:16,394 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.51 vs. limit=15.0
2023-11-23 01:17:39,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2175413.3333333335, ans=0.125
2023-11-23 01:17:42,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2175413.3333333335, ans=0.05
2023-11-23 01:17:55,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2175480.0, ans=0.04949747468305833
2023-11-23 01:18:08,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.94 vs. limit=15.0
2023-11-23 01:18:11,452 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1700, loss[loss=0.05846, simple_loss=0.07208, pruned_loss=0.0135, audio_tagging_loss=0.008921, over 15736.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09302, pruned_loss=0.0144, audio_tagging_loss=0.009516, over 3043564.14 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:18:14,135 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:18:19,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326350
2023-11-23 01:18:23,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2175680.0, ans=0.0
2023-11-23 01:18:39,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2175746.6666666665, ans=0.125
2023-11-23 01:18:43,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2175746.6666666665, ans=0.2
2023-11-23 01:18:43,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.02 vs. limit=15.0
2023-11-23 01:18:47,668 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.185e+01 8.813e+01 9.746e+01 1.184e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 01:18:56,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2175813.3333333335, ans=0.0
2023-11-23 01:19:01,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2175880.0, ans=0.0
2023-11-23 01:19:15,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.99 vs. limit=15.0
2023-11-23 01:19:16,147 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1750, loss[loss=0.07634, simple_loss=0.1099, pruned_loss=0.01374, audio_tagging_loss=0.007631, over 15568.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09279, pruned_loss=0.01429, audio_tagging_loss=0.00949, over 3053408.37 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:19:23,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326400
2023-11-23 01:19:23,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2175946.6666666665, ans=0.025
2023-11-23 01:19:54,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.06 vs. limit=10.0
2023-11-23 01:20:20,578 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1800, loss[loss=0.06838, simple_loss=0.09175, pruned_loss=0.01278, audio_tagging_loss=0.009729, over 15567.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09341, pruned_loss=0.0144, audio_tagging_loss=0.00933, over 3056486.84 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:20:22,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2176280.0, ans=0.95
2023-11-23 01:20:27,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2176280.0, ans=0.0
2023-11-23 01:20:28,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326450
2023-11-23 01:20:32,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2176346.6666666665, ans=0.0
2023-11-23 01:20:57,610 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 8.178e+01 8.671e+01 9.411e+01 1.178e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-23 01:21:04,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2176480.0, ans=0.125
2023-11-23 01:21:14,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2176546.6666666665, ans=10.0
2023-11-23 01:21:17,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2176546.6666666665, ans=0.1
2023-11-23 01:21:20,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2176546.6666666665, ans=0.125
2023-11-23 01:21:25,126 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1850, loss[loss=0.07834, simple_loss=0.1059, pruned_loss=0.01555, audio_tagging_loss=0.009821, over 16294.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09308, pruned_loss=0.01437, audio_tagging_loss=0.009258, over 3052624.16 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:21:33,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326500
2023-11-23 01:21:43,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2176680.0, ans=0.1
2023-11-23 01:22:02,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2176746.6666666665, ans=0.125
2023-11-23 01:22:03,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2176813.3333333335, ans=0.125
2023-11-23 01:22:31,054 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1900, loss[loss=0.08189, simple_loss=0.1038, pruned_loss=0.01995, audio_tagging_loss=0.01004, over 16018.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09206, pruned_loss=0.01426, audio_tagging_loss=0.009221, over 3052265.55 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:22:39,631 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326550
2023-11-23 01:22:42,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2176946.6666666665, ans=0.95
2023-11-23 01:22:59,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2177080.0, ans=0.1
2023-11-23 01:23:04,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2177080.0, ans=0.2
2023-11-23 01:23:06,400 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.801e+01 8.128e+01 8.928e+01 9.740e+01 2.315e+02, threshold=1.786e+02, percent-clipped=1.0
2023-11-23 01:23:09,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2177146.6666666665, ans=0.0
2023-11-23 01:23:24,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2177213.3333333335, ans=0.05
2023-11-23 01:23:36,013 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 1950, loss[loss=0.0455, simple_loss=0.05828, pruned_loss=0.004653, audio_tagging_loss=0.0117, over 16582.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09099, pruned_loss=0.01407, audio_tagging_loss=0.009198, over 3049032.14 frames. ], batch size: 66, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:23:43,559 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326600
2023-11-23 01:23:43,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2177280.0, ans=0.2
2023-11-23 01:23:47,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2177346.6666666665, ans=0.0
2023-11-23 01:24:05,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2177413.3333333335, ans=0.0
2023-11-23 01:24:11,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2177413.3333333335, ans=0.125
2023-11-23 01:24:22,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2177480.0, ans=0.0
2023-11-23 01:24:37,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2177546.6666666665, ans=0.09899494936611666
2023-11-23 01:24:38,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2177546.6666666665, ans=0.0
2023-11-23 01:24:40,691 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2000, loss[loss=0.08422, simple_loss=0.1151, pruned_loss=0.01954, audio_tagging_loss=0.007121, over 15277.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09029, pruned_loss=0.0138, audio_tagging_loss=0.009294, over 3046848.98 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:24:48,126 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326650
2023-11-23 01:24:57,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2177680.0, ans=0.125
2023-11-23 01:25:17,649 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.041e+01 8.683e+01 9.382e+01 1.367e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 01:25:20,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2177813.3333333335, ans=0.125
2023-11-23 01:25:31,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2177880.0, ans=0.125
2023-11-23 01:25:35,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2177880.0, ans=0.125
2023-11-23 01:25:45,372 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2050, loss[loss=0.07591, simple_loss=0.09554, pruned_loss=0.0146, audio_tagging_loss=0.01353, over 15196.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.0908, pruned_loss=0.01401, audio_tagging_loss=0.009271, over 3035795.08 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:25:53,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326700
2023-11-23 01:26:06,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2178013.3333333335, ans=0.2
2023-11-23 01:26:09,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2178013.3333333335, ans=0.2
2023-11-23 01:26:17,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2178080.0, ans=0.125
2023-11-23 01:26:24,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2178146.6666666665, ans=0.125
2023-11-23 01:26:26,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2178146.6666666665, ans=0.95
2023-11-23 01:26:44,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2178213.3333333335, ans=0.0
2023-11-23 01:26:49,583 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2100, loss[loss=0.07741, simple_loss=0.1019, pruned_loss=0.01799, audio_tagging_loss=0.008452, over 15253.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09191, pruned_loss=0.0142, audio_tagging_loss=0.009118, over 3046091.78 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:26:57,533 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326750
2023-11-23 01:27:25,011 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.865e+01 8.249e+01 8.824e+01 9.661e+01 1.236e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 01:27:48,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.80 vs. limit=22.5
2023-11-23 01:27:53,263 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2150, loss[loss=0.06566, simple_loss=0.07867, pruned_loss=0.01581, audio_tagging_loss=0.01052, over 15096.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.0928, pruned_loss=0.01425, audio_tagging_loss=0.009081, over 3041860.00 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:27:57,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2178613.3333333335, ans=0.125
2023-11-23 01:28:00,734 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326800
2023-11-23 01:28:03,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2178613.3333333335, ans=0.125
2023-11-23 01:28:23,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2178746.6666666665, ans=0.125
2023-11-23 01:28:25,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2178746.6666666665, ans=0.125
2023-11-23 01:28:33,812 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:28:37,131 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:28:49,424 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:28:58,350 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2200, loss[loss=0.06394, simple_loss=0.09007, pruned_loss=0.01046, audio_tagging_loss=0.008444, over 15046.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09339, pruned_loss=0.0144, audio_tagging_loss=0.009171, over 3047843.11 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:29:03,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2178946.6666666665, ans=0.2
2023-11-23 01:29:06,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2178946.6666666665, ans=0.0
2023-11-23 01:29:06,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326850
2023-11-23 01:29:12,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2179013.3333333335, ans=0.04949747468305833
2023-11-23 01:29:18,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2179013.3333333335, ans=0.125
2023-11-23 01:29:24,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2179080.0, ans=0.125
2023-11-23 01:29:31,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.06 vs. limit=22.5
2023-11-23 01:29:34,173 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.469e+01 8.071e+01 8.694e+01 9.748e+01 1.270e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-23 01:29:39,632 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.20 vs. limit=15.0
2023-11-23 01:29:55,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2179213.3333333335, ans=0.0
2023-11-23 01:30:02,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2179280.0, ans=0.0
2023-11-23 01:30:03,040 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2250, loss[loss=0.08202, simple_loss=0.1083, pruned_loss=0.01903, audio_tagging_loss=0.00882, over 15313.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09393, pruned_loss=0.01468, audio_tagging_loss=0.009171, over 3047996.40 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:30:06,137 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.58 vs. limit=15.0
2023-11-23 01:30:10,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326900
2023-11-23 01:30:15,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2179346.6666666665, ans=0.125
2023-11-23 01:30:24,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2179346.6666666665, ans=0.2
2023-11-23 01:30:36,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2179413.3333333335, ans=0.125
2023-11-23 01:30:37,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2179413.3333333335, ans=0.125
2023-11-23 01:30:43,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2179480.0, ans=0.0
2023-11-23 01:30:55,984 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.46 vs. limit=10.0
2023-11-23 01:31:07,426 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2300, loss[loss=0.04977, simple_loss=0.05226, pruned_loss=0.01167, audio_tagging_loss=0.01197, over 13842.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09287, pruned_loss=0.0145, audio_tagging_loss=0.009281, over 3040074.79 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:31:14,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 326950
2023-11-23 01:31:28,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2179680.0, ans=0.1
2023-11-23 01:31:30,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2179680.0, ans=0.025
2023-11-23 01:31:44,828 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.250e+01 8.860e+01 9.394e+01 1.286e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 01:32:05,588 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:32:05,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2179880.0, ans=0.2
2023-11-23 01:32:10,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2179946.6666666665, ans=0.1
2023-11-23 01:32:12,379 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2350, loss[loss=0.07255, simple_loss=0.1011, pruned_loss=0.0122, audio_tagging_loss=0.009791, over 17104.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09289, pruned_loss=0.0144, audio_tagging_loss=0.00935, over 3039394.08 frames. ], batch size: 62, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:32:14,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2179946.6666666665, ans=22.5
2023-11-23 01:32:19,805 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327000
2023-11-23 01:32:33,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2180013.3333333335, ans=0.2
2023-11-23 01:32:37,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2180080.0, ans=0.0
2023-11-23 01:32:53,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.13 vs. limit=15.0
2023-11-23 01:33:00,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2180146.6666666665, ans=0.125
2023-11-23 01:33:17,545 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2400, loss[loss=0.07263, simple_loss=0.0952, pruned_loss=0.01466, audio_tagging_loss=0.01036, over 16041.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09281, pruned_loss=0.01433, audio_tagging_loss=0.009381, over 3037654.75 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:33:22,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2180280.0, ans=0.125
2023-11-23 01:33:25,007 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327050
2023-11-23 01:33:26,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2180280.0, ans=0.2
2023-11-23 01:33:32,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2180346.6666666665, ans=0.1
2023-11-23 01:33:46,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2180413.3333333335, ans=0.1
2023-11-23 01:33:53,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2180413.3333333335, ans=0.2
2023-11-23 01:33:54,368 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.524e+01 9.128e+01 9.621e+01 1.327e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 01:34:04,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2180480.0, ans=0.2
2023-11-23 01:34:14,601 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.15 vs. limit=15.0
2023-11-23 01:34:21,276 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2450, loss[loss=0.09013, simple_loss=0.1223, pruned_loss=0.02058, audio_tagging_loss=0.00843, over 15488.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09301, pruned_loss=0.01424, audio_tagging_loss=0.00944, over 3042321.29 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:34:28,750 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327100
2023-11-23 01:34:30,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2180613.3333333335, ans=0.125
2023-11-23 01:34:54,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2180746.6666666665, ans=0.125
2023-11-23 01:34:55,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2180746.6666666665, ans=0.125
2023-11-23 01:34:57,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2180746.6666666665, ans=0.0
2023-11-23 01:35:00,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2180813.3333333335, ans=0.125
2023-11-23 01:35:03,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2180813.3333333335, ans=0.125
2023-11-23 01:35:03,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2180813.3333333335, ans=0.07
2023-11-23 01:35:07,901 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:35:09,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2180813.3333333335, ans=0.1
2023-11-23 01:35:11,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2180880.0, ans=0.1
2023-11-23 01:35:17,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2180880.0, ans=0.125
2023-11-23 01:35:23,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2180880.0, ans=0.125
2023-11-23 01:35:25,319 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2500, loss[loss=0.05334, simple_loss=0.06955, pruned_loss=0.006791, audio_tagging_loss=0.01177, over 14642.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09343, pruned_loss=0.01442, audio_tagging_loss=0.009432, over 3043453.16 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:35:25,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2180946.6666666665, ans=0.1
2023-11-23 01:35:33,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327150
2023-11-23 01:36:02,410 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.294e+01 8.910e+01 9.572e+01 1.200e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 01:36:08,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2181146.6666666665, ans=0.2
2023-11-23 01:36:23,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2181213.3333333335, ans=0.04949747468305833
2023-11-23 01:36:30,890 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2550, loss[loss=0.07126, simple_loss=0.09282, pruned_loss=0.01543, audio_tagging_loss=0.009426, over 16440.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09264, pruned_loss=0.0143, audio_tagging_loss=0.009334, over 3045133.59 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:36:38,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327200
2023-11-23 01:36:54,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2181413.3333333335, ans=0.0
2023-11-23 01:37:20,917 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.49 vs. limit=15.0
2023-11-23 01:37:26,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2181546.6666666665, ans=0.125
2023-11-23 01:37:35,129 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2600, loss[loss=0.06803, simple_loss=0.08714, pruned_loss=0.01618, audio_tagging_loss=0.008286, over 14517.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09253, pruned_loss=0.01422, audio_tagging_loss=0.00916, over 3035933.51 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:37:42,634 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327250
2023-11-23 01:37:50,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2181680.0, ans=0.2
2023-11-23 01:37:53,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2181680.0, ans=0.125
2023-11-23 01:38:13,508 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.227e+01 8.835e+01 9.553e+01 1.278e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 01:38:39,230 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2650, loss[loss=0.06778, simple_loss=0.09137, pruned_loss=0.01452, audio_tagging_loss=0.007569, over 15829.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09279, pruned_loss=0.01436, audio_tagging_loss=0.009121, over 3034288.87 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:38:46,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.66 vs. limit=15.0
2023-11-23 01:38:47,246 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327300
2023-11-23 01:38:57,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2182013.3333333335, ans=0.2
2023-11-23 01:39:10,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2182080.0, ans=0.0
2023-11-23 01:39:21,126 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0
2023-11-23 01:39:23,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2182146.6666666665, ans=0.0
2023-11-23 01:39:37,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2182213.3333333335, ans=0.125
2023-11-23 01:39:39,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2182213.3333333335, ans=0.07
2023-11-23 01:39:44,316 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2700, loss[loss=0.0667, simple_loss=0.09011, pruned_loss=0.01343, audio_tagging_loss=0.008217, over 15185.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09241, pruned_loss=0.0143, audio_tagging_loss=0.009073, over 3037668.21 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:39:47,958 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.73 vs. limit=22.5
2023-11-23 01:39:52,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327350
2023-11-23 01:39:55,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2182280.0, ans=0.2
2023-11-23 01:40:08,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.09 vs. limit=22.5
2023-11-23 01:40:19,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2182413.3333333335, ans=0.125
2023-11-23 01:40:21,082 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.147e+01 8.099e+01 8.608e+01 9.208e+01 1.380e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-23 01:40:31,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2182480.0, ans=0.2
2023-11-23 01:40:35,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2182546.6666666665, ans=0.125
2023-11-23 01:40:48,627 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2750, loss[loss=0.07293, simple_loss=0.09358, pruned_loss=0.01659, audio_tagging_loss=0.009547, over 14011.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09185, pruned_loss=0.01413, audio_tagging_loss=0.009042, over 3038936.08 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:40:56,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327400
2023-11-23 01:41:21,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2182746.6666666665, ans=0.0
2023-11-23 01:41:30,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2182813.3333333335, ans=15.0
2023-11-23 01:41:44,319 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:41:52,831 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2800, loss[loss=0.04165, simple_loss=0.04774, pruned_loss=0.00656, audio_tagging_loss=0.01122, over 15686.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09195, pruned_loss=0.01405, audio_tagging_loss=0.009084, over 3042628.86 frames. ], batch size: 62, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:42:00,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327450
2023-11-23 01:42:00,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2182946.6666666665, ans=0.125
2023-11-23 01:42:05,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2183013.3333333335, ans=0.1
2023-11-23 01:42:31,837 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.504e+01 8.188e+01 8.710e+01 9.337e+01 1.091e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-23 01:42:38,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2183146.6666666665, ans=0.1
2023-11-23 01:42:56,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2183280.0, ans=0.125
2023-11-23 01:42:58,031 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2850, loss[loss=0.07243, simple_loss=0.08919, pruned_loss=0.01659, audio_tagging_loss=0.01125, over 14229.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.0921, pruned_loss=0.01413, audio_tagging_loss=0.009123, over 3047280.70 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:43:06,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327500
2023-11-23 01:43:38,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.64 vs. limit=15.0
2023-11-23 01:43:54,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2183546.6666666665, ans=0.0
2023-11-23 01:44:01,865 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2900, loss[loss=0.05708, simple_loss=0.07867, pruned_loss=0.007628, audio_tagging_loss=0.01012, over 15219.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.0914, pruned_loss=0.01411, audio_tagging_loss=0.009148, over 3045847.96 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:44:09,334 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327550
2023-11-23 01:44:12,153 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.88 vs. limit=22.5
2023-11-23 01:44:26,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2183746.6666666665, ans=0.1
2023-11-23 01:44:26,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2183746.6666666665, ans=0.125
2023-11-23 01:44:41,223 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.459e+01 9.104e+01 9.833e+01 1.241e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-23 01:44:47,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2183813.3333333335, ans=0.125
2023-11-23 01:44:51,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2183813.3333333335, ans=0.125
2023-11-23 01:44:54,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2183880.0, ans=0.125
2023-11-23 01:44:57,345 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:45:05,854 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 2950, loss[loss=0.06715, simple_loss=0.081, pruned_loss=0.01388, audio_tagging_loss=0.01277, over 14777.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09238, pruned_loss=0.01438, audio_tagging_loss=0.009129, over 3049760.07 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:45:13,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327600
2023-11-23 01:45:18,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.54 vs. limit=15.0
2023-11-23 01:45:43,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2184080.0, ans=0.125
2023-11-23 01:45:43,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2184080.0, ans=0.125
2023-11-23 01:45:46,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.93 vs. limit=10.0
2023-11-23 01:46:01,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2184213.3333333335, ans=0.125
2023-11-23 01:46:05,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2184213.3333333335, ans=0.125
2023-11-23 01:46:09,986 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3000, loss[loss=0.08088, simple_loss=0.112, pruned_loss=0.01638, audio_tagging_loss=0.008502, over 15523.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09261, pruned_loss=0.01441, audio_tagging_loss=0.009147, over 3056185.00 frames. ], batch size: 61, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:46:09,986 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 01:46:53,260 INFO [train_asr.py:1253] (2/4) Epoch 28, validation: loss=0.05807, simple_loss=0.05122, pruned_loss=0.005026, audio_tagging_loss=0.02744, over 4681554.00 frames. 
2023-11-23 01:46:53,261 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 01:47:00,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327650
2023-11-23 01:47:32,176 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.143e+01 8.821e+01 9.621e+01 1.257e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 01:47:36,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2184480.0, ans=0.2
2023-11-23 01:47:44,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2184546.6666666665, ans=0.125
2023-11-23 01:47:47,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2184546.6666666665, ans=0.0
2023-11-23 01:47:48,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2184546.6666666665, ans=0.125
2023-11-23 01:47:48,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2184546.6666666665, ans=0.125
2023-11-23 01:47:56,824 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3050, loss[loss=0.08741, simple_loss=0.1144, pruned_loss=0.02018, audio_tagging_loss=0.01003, over 16012.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09247, pruned_loss=0.01436, audio_tagging_loss=0.009242, over 3051746.94 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:47:57,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2184613.3333333335, ans=0.2
2023-11-23 01:48:04,747 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327700
2023-11-23 01:48:06,093 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:48:08,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2184680.0, ans=0.0
2023-11-23 01:48:19,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2184680.0, ans=0.125
2023-11-23 01:48:23,030 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.14 vs. limit=12.0
2023-11-23 01:48:25,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2184746.6666666665, ans=0.0
2023-11-23 01:48:35,712 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:48:56,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2184880.0, ans=0.125
2023-11-23 01:48:58,969 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0
2023-11-23 01:49:01,421 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3100, loss[loss=0.05693, simple_loss=0.06595, pruned_loss=0.01419, audio_tagging_loss=0.009767, over 15767.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09204, pruned_loss=0.0143, audio_tagging_loss=0.009414, over 3044951.62 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:49:09,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327750
2023-11-23 01:49:20,414 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.12 vs. limit=15.0
2023-11-23 01:49:33,774 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.95 vs. limit=15.0
2023-11-23 01:49:39,091 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.214e+01 8.129e+01 8.806e+01 9.628e+01 1.604e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 01:49:55,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2185213.3333333335, ans=0.035
2023-11-23 01:50:05,057 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.30 vs. limit=22.5
2023-11-23 01:50:05,768 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3150, loss[loss=0.08666, simple_loss=0.1201, pruned_loss=0.01771, audio_tagging_loss=0.008897, over 16281.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.0914, pruned_loss=0.01418, audio_tagging_loss=0.009448, over 3039668.37 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:50:11,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2185280.0, ans=0.125
2023-11-23 01:50:13,380 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327800
2023-11-23 01:50:16,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2185280.0, ans=0.125
2023-11-23 01:50:30,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2185413.3333333335, ans=0.0
2023-11-23 01:51:09,978 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3200, loss[loss=0.08811, simple_loss=0.1271, pruned_loss=0.01769, audio_tagging_loss=0.006877, over 15831.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09273, pruned_loss=0.01427, audio_tagging_loss=0.009405, over 3042152.64 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:51:17,674 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327850
2023-11-23 01:51:23,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2185680.0, ans=0.1
2023-11-23 01:51:23,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2185680.0, ans=0.09899494936611666
2023-11-23 01:51:26,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2185680.0, ans=0.0
2023-11-23 01:51:43,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.24 vs. limit=15.0
2023-11-23 01:51:46,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2185746.6666666665, ans=0.125
2023-11-23 01:51:49,568 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.239e+01 8.167e+01 8.814e+01 9.617e+01 1.193e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 01:51:52,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten.whitening_limit, batch_count=2185813.3333333335, ans=15.0
2023-11-23 01:51:58,526 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:52:14,663 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3250, loss[loss=0.05513, simple_loss=0.06869, pruned_loss=0.008539, audio_tagging_loss=0.01225, over 15807.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.0918, pruned_loss=0.0142, audio_tagging_loss=0.00961, over 3033268.32 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:52:23,230 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327900
2023-11-23 01:52:23,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2185946.6666666665, ans=0.0
2023-11-23 01:52:25,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2185946.6666666665, ans=0.07
2023-11-23 01:52:39,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2186080.0, ans=0.125
2023-11-23 01:53:01,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2186146.6666666665, ans=0.0
2023-11-23 01:53:14,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2186213.3333333335, ans=0.95
2023-11-23 01:53:18,585 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3300, loss[loss=0.08877, simple_loss=0.1253, pruned_loss=0.01878, audio_tagging_loss=0.007328, over 15919.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09259, pruned_loss=0.01431, audio_tagging_loss=0.00962, over 3041767.82 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:53:26,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 327950
2023-11-23 01:53:31,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2186346.6666666665, ans=0.2
2023-11-23 01:53:57,272 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.779e+01 8.400e+01 9.018e+01 9.669e+01 1.285e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 01:54:11,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2186546.6666666665, ans=0.125
2023-11-23 01:54:18,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2186546.6666666665, ans=0.125
2023-11-23 01:54:18,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2186546.6666666665, ans=0.1
2023-11-23 01:54:22,722 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3350, loss[loss=0.08357, simple_loss=0.1099, pruned_loss=0.02113, audio_tagging_loss=0.007483, over 15732.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.09442, pruned_loss=0.01478, audio_tagging_loss=0.009491, over 3045788.79 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:54:30,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328000
2023-11-23 01:54:30,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2186613.3333333335, ans=0.125
2023-11-23 01:54:49,756 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.58 vs. limit=22.5
2023-11-23 01:55:04,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2186813.3333333335, ans=0.125
2023-11-23 01:55:19,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2186880.0, ans=0.125
2023-11-23 01:55:22,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2186880.0, ans=0.125
2023-11-23 01:55:30,826 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3400, loss[loss=0.09152, simple_loss=0.1243, pruned_loss=0.0234, audio_tagging_loss=0.005964, over 15349.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09458, pruned_loss=0.01487, audio_tagging_loss=0.009389, over 3047780.03 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:55:39,409 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328050
2023-11-23 01:55:44,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2187013.3333333335, ans=0.0
2023-11-23 01:55:46,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2187013.3333333335, ans=0.2
2023-11-23 01:56:06,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2187080.0, ans=0.125
2023-11-23 01:56:09,472 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.501e+01 8.298e+01 8.922e+01 9.490e+01 1.312e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 01:56:35,852 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3450, loss[loss=0.07532, simple_loss=0.09953, pruned_loss=0.01471, audio_tagging_loss=0.01085, over 15534.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09413, pruned_loss=0.01479, audio_tagging_loss=0.009243, over 3052078.53 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:56:39,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2187280.0, ans=0.2
2023-11-23 01:56:43,028 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328100
2023-11-23 01:57:33,446 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.24 vs. limit=15.0
2023-11-23 01:57:39,747 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3500, loss[loss=0.07297, simple_loss=0.1046, pruned_loss=0.01141, audio_tagging_loss=0.009254, over 15491.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09408, pruned_loss=0.01461, audio_tagging_loss=0.009123, over 3056435.33 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:57:42,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2187613.3333333335, ans=0.125
2023-11-23 01:57:47,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328150
2023-11-23 01:58:06,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2187746.6666666665, ans=0.125
2023-11-23 01:58:13,435 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:58:18,276 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.297e+01 8.906e+01 9.538e+01 1.458e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 01:58:43,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.65 vs. limit=15.0
2023-11-23 01:58:44,558 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3550, loss[loss=0.07081, simple_loss=0.09349, pruned_loss=0.01547, audio_tagging_loss=0.008598, over 15358.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09383, pruned_loss=0.0145, audio_tagging_loss=0.009195, over 3056206.34 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:58:53,218 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328200
2023-11-23 01:59:21,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2188080.0, ans=0.0
2023-11-23 01:59:33,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=15.0
2023-11-23 01:59:35,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2188213.3333333335, ans=0.0
2023-11-23 01:59:45,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2188213.3333333335, ans=0.125
2023-11-23 01:59:50,105 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3600, loss[loss=0.06612, simple_loss=0.08965, pruned_loss=0.0151, audio_tagging_loss=0.006195, over 15205.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09384, pruned_loss=0.01459, audio_tagging_loss=0.009105, over 3048682.61 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:59:55,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2188280.0, ans=0.0
2023-11-23 01:59:57,455 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328250
2023-11-23 01:59:57,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2188280.0, ans=0.1
2023-11-23 01:59:58,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2188280.0, ans=0.125
2023-11-23 02:00:02,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2188346.6666666665, ans=0.2
2023-11-23 02:00:24,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2188413.3333333335, ans=0.125
2023-11-23 02:00:27,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2188480.0, ans=0.125
2023-11-23 02:00:28,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.210e+01 8.927e+01 9.741e+01 1.268e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 02:00:46,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2188546.6666666665, ans=0.125
2023-11-23 02:00:50,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2188546.6666666665, ans=0.0
2023-11-23 02:00:53,677 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3650, loss[loss=0.06489, simple_loss=0.07597, pruned_loss=0.01496, audio_tagging_loss=0.01194, over 14191.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09449, pruned_loss=0.01473, audio_tagging_loss=0.009073, over 3042164.15 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:01:00,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328300
2023-11-23 02:01:16,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=2188680.0, ans=22.5
2023-11-23 02:01:30,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2188746.6666666665, ans=0.1
2023-11-23 02:01:32,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2188813.3333333335, ans=0.2
2023-11-23 02:01:42,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2188813.3333333335, ans=0.5
2023-11-23 02:01:57,282 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3700, loss[loss=0.06232, simple_loss=0.08226, pruned_loss=0.009858, audio_tagging_loss=0.01133, over 15466.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.0938, pruned_loss=0.01467, audio_tagging_loss=0.009057, over 3049278.46 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:02:05,389 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328350
2023-11-23 02:02:19,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2189013.3333333335, ans=0.0
2023-11-23 02:02:37,558 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.324e+01 8.915e+01 9.653e+01 1.223e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 02:02:41,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2189146.6666666665, ans=0.0
2023-11-23 02:02:50,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.21 vs. limit=6.0
2023-11-23 02:03:03,362 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3750, loss[loss=0.08795, simple_loss=0.1226, pruned_loss=0.01967, audio_tagging_loss=0.006983, over 15614.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09383, pruned_loss=0.01478, audio_tagging_loss=0.009058, over 3049076.53 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:03:04,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2189280.0, ans=0.2
2023-11-23 02:03:05,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.55 vs. limit=22.5
2023-11-23 02:03:10,806 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328400
2023-11-23 02:03:16,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2189346.6666666665, ans=0.0
2023-11-23 02:03:30,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2189413.3333333335, ans=0.125
2023-11-23 02:03:38,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2189413.3333333335, ans=0.035
2023-11-23 02:03:40,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2189480.0, ans=0.07
2023-11-23 02:03:48,560 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:03:55,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2189546.6666666665, ans=0.1
2023-11-23 02:04:06,822 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3800, loss[loss=0.05613, simple_loss=0.07099, pruned_loss=0.01093, audio_tagging_loss=0.009709, over 16001.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09367, pruned_loss=0.01474, audio_tagging_loss=0.009078, over 3052209.78 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:04:08,804 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.22 vs. limit=15.0
2023-11-23 02:04:12,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2189613.3333333335, ans=0.125
2023-11-23 02:04:14,402 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328450
2023-11-23 02:04:15,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2189613.3333333335, ans=0.125
2023-11-23 02:04:47,088 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.455e+01 8.996e+01 9.813e+01 1.168e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 02:04:57,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2189880.0, ans=0.125
2023-11-23 02:05:01,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-23 02:05:10,388 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3850, loss[loss=0.05805, simple_loss=0.07986, pruned_loss=0.00832, audio_tagging_loss=0.009805, over 15087.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09414, pruned_loss=0.01463, audio_tagging_loss=0.009033, over 3056379.04 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:05:19,151 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328500
2023-11-23 02:06:15,814 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3900, loss[loss=0.0601, simple_loss=0.0826, pruned_loss=0.008801, audio_tagging_loss=0.009996, over 14710.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09321, pruned_loss=0.01439, audio_tagging_loss=0.009159, over 3057998.55 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:06:17,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2190280.0, ans=0.025
2023-11-23 02:06:23,376 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328550
2023-11-23 02:06:24,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2190280.0, ans=0.0
2023-11-23 02:06:38,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2190346.6666666665, ans=0.125
2023-11-23 02:06:45,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.73 vs. limit=12.0
2023-11-23 02:06:47,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2190413.3333333335, ans=0.0
2023-11-23 02:06:48,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-23 02:06:53,555 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.052e+01 8.828e+01 9.660e+01 1.313e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 02:07:18,494 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 3950, loss[loss=0.06498, simple_loss=0.08657, pruned_loss=0.007037, audio_tagging_loss=0.01466, over 14851.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09413, pruned_loss=0.01443, audio_tagging_loss=0.009173, over 3050288.01 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:07:25,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328600
2023-11-23 02:07:26,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2190613.3333333335, ans=0.07
2023-11-23 02:07:31,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2190680.0, ans=0.125
2023-11-23 02:07:49,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2190746.6666666665, ans=0.04949747468305833
2023-11-23 02:07:52,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2190746.6666666665, ans=0.0
2023-11-23 02:08:00,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2190813.3333333335, ans=0.125
2023-11-23 02:08:07,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2190813.3333333335, ans=0.125
2023-11-23 02:08:11,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2190880.0, ans=0.0
2023-11-23 02:08:22,771 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4000, loss[loss=0.07899, simple_loss=0.1053, pruned_loss=0.01643, audio_tagging_loss=0.009917, over 15642.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09405, pruned_loss=0.01435, audio_tagging_loss=0.009254, over 3050450.68 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:08:30,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328650
2023-11-23 02:08:35,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2191013.3333333335, ans=0.125
2023-11-23 02:08:48,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2191080.0, ans=0.95
2023-11-23 02:08:58,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.41 vs. limit=5.0
2023-11-23 02:09:03,448 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.297e+01 8.975e+01 9.665e+01 1.273e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 02:09:07,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2191146.6666666665, ans=0.125
2023-11-23 02:09:28,247 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4050, loss[loss=0.08023, simple_loss=0.1095, pruned_loss=0.01888, audio_tagging_loss=0.006592, over 15293.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09462, pruned_loss=0.01448, audio_tagging_loss=0.009252, over 3045176.21 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:09:32,513 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:09:32,865 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:09:36,282 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328700
2023-11-23 02:09:47,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2191346.6666666665, ans=0.125
2023-11-23 02:09:49,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2191346.6666666665, ans=0.125
2023-11-23 02:09:56,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2191413.3333333335, ans=0.0
2023-11-23 02:09:56,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.76 vs. limit=15.0
2023-11-23 02:10:32,635 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4100, loss[loss=0.07733, simple_loss=0.104, pruned_loss=0.0155, audio_tagging_loss=0.009819, over 15090.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09472, pruned_loss=0.01456, audio_tagging_loss=0.009242, over 3045051.25 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:10:40,067 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328750
2023-11-23 02:10:59,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2191746.6666666665, ans=0.07
2023-11-23 02:10:59,452 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.95 vs. limit=22.5
2023-11-23 02:11:04,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2191746.6666666665, ans=15.0
2023-11-23 02:11:05,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2191746.6666666665, ans=0.125
2023-11-23 02:11:10,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2191813.3333333335, ans=0.5
2023-11-23 02:11:13,906 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.198e+01 8.662e+01 9.320e+01 1.292e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-23 02:11:20,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2191813.3333333335, ans=0.07
2023-11-23 02:11:28,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2191880.0, ans=0.125
2023-11-23 02:11:29,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2191880.0, ans=0.125
2023-11-23 02:11:33,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2191880.0, ans=0.125
2023-11-23 02:11:36,020 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4150, loss[loss=0.07747, simple_loss=0.1022, pruned_loss=0.01771, audio_tagging_loss=0.008692, over 14135.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09432, pruned_loss=0.01449, audio_tagging_loss=0.009145, over 3040496.31 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:11:43,596 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328800
2023-11-23 02:11:46,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2191946.6666666665, ans=0.0
2023-11-23 02:11:48,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2192013.3333333335, ans=0.125
2023-11-23 02:11:49,825 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.47 vs. limit=12.0
2023-11-23 02:11:50,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2192013.3333333335, ans=0.0
2023-11-23 02:11:51,202 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.22 vs. limit=22.5
2023-11-23 02:12:10,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2192080.0, ans=0.125
2023-11-23 02:12:11,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2192080.0, ans=0.125
2023-11-23 02:12:14,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2192146.6666666665, ans=15.0
2023-11-23 02:12:15,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.87 vs. limit=15.0
2023-11-23 02:12:16,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2192146.6666666665, ans=15.0
2023-11-23 02:12:22,720 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:12:25,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2192146.6666666665, ans=0.1
2023-11-23 02:12:27,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2192213.3333333335, ans=0.125
2023-11-23 02:12:40,185 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4200, loss[loss=0.06317, simple_loss=0.08362, pruned_loss=0.01193, audio_tagging_loss=0.009433, over 15595.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09431, pruned_loss=0.01452, audio_tagging_loss=0.009082, over 3049207.03 frames. ], batch size: 61, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:12:48,739 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328850
2023-11-23 02:12:55,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2192346.6666666665, ans=0.125
2023-11-23 02:13:00,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2192346.6666666665, ans=0.125
2023-11-23 02:13:10,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2192413.3333333335, ans=0.0
2023-11-23 02:13:19,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2192480.0, ans=0.05
2023-11-23 02:13:20,632 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.473e+01 9.086e+01 9.924e+01 1.385e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 02:13:37,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.10 vs. limit=22.5
2023-11-23 02:13:41,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2192546.6666666665, ans=0.125
2023-11-23 02:13:44,947 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4250, loss[loss=0.06548, simple_loss=0.07784, pruned_loss=0.01319, audio_tagging_loss=0.01337, over 15365.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09421, pruned_loss=0.01448, audio_tagging_loss=0.00906, over 3050196.51 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:13:52,410 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328900
2023-11-23 02:14:04,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.80 vs. limit=15.0
2023-11-23 02:14:31,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2192813.3333333335, ans=0.0
2023-11-23 02:14:33,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2192813.3333333335, ans=0.0
2023-11-23 02:14:38,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2192880.0, ans=0.07
2023-11-23 02:14:41,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2192880.0, ans=0.1
2023-11-23 02:14:46,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2192880.0, ans=0.125
2023-11-23 02:14:49,000 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4300, loss[loss=0.06683, simple_loss=0.08987, pruned_loss=0.01026, audio_tagging_loss=0.01164, over 14557.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09391, pruned_loss=0.01438, audio_tagging_loss=0.00908, over 3047975.61 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:14:55,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2192946.6666666665, ans=0.125
2023-11-23 02:14:56,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 328950
2023-11-23 02:15:04,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2193013.3333333335, ans=0.125
2023-11-23 02:15:20,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2193080.0, ans=0.04949747468305833
2023-11-23 02:15:28,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2193146.6666666665, ans=0.125
2023-11-23 02:15:30,846 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.649e+01 8.452e+01 8.978e+01 9.927e+01 1.263e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 02:15:53,695 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4350, loss[loss=0.08355, simple_loss=0.1058, pruned_loss=0.02259, audio_tagging_loss=0.008039, over 14805.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09354, pruned_loss=0.01445, audio_tagging_loss=0.009054, over 3046085.49 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:15:56,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2193280.0, ans=0.125
2023-11-23 02:16:02,567 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329000
2023-11-23 02:16:02,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2193280.0, ans=0.125
2023-11-23 02:16:26,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2193413.3333333335, ans=0.1
2023-11-23 02:16:58,966 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4400, loss[loss=0.06233, simple_loss=0.0826, pruned_loss=0.01027, audio_tagging_loss=0.01076, over 16741.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09349, pruned_loss=0.01445, audio_tagging_loss=0.008878, over 3046927.93 frames. ], batch size: 63, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:17:00,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=15.0
2023-11-23 02:17:05,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2193613.3333333335, ans=10.0
2023-11-23 02:17:06,000 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.022e-02
2023-11-23 02:17:06,994 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329050
2023-11-23 02:17:18,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2193680.0, ans=0.2
2023-11-23 02:17:25,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2193746.6666666665, ans=0.07
2023-11-23 02:17:40,400 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.171e+01 8.943e+01 9.862e+01 1.169e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 02:17:54,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.18 vs. limit=15.0
2023-11-23 02:17:54,793 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.17 vs. limit=15.0
2023-11-23 02:18:03,080 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4450, loss[loss=0.07802, simple_loss=0.104, pruned_loss=0.01449, audio_tagging_loss=0.01155, over 16220.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09366, pruned_loss=0.01435, audio_tagging_loss=0.008937, over 3050078.00 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:18:10,627 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329100
2023-11-23 02:18:14,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2194013.3333333335, ans=0.125
2023-11-23 02:19:07,658 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4500, loss[loss=0.0561, simple_loss=0.07423, pruned_loss=0.009624, audio_tagging_loss=0.009365, over 14249.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09356, pruned_loss=0.0143, audio_tagging_loss=0.008903, over 3049948.55 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:19:12,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2194280.0, ans=0.0
2023-11-23 02:19:16,368 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329150
2023-11-23 02:19:26,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2194346.6666666665, ans=0.125
2023-11-23 02:19:30,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2194346.6666666665, ans=0.0
2023-11-23 02:19:40,014 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0
2023-11-23 02:19:43,715 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=15.0
2023-11-23 02:19:45,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2194480.0, ans=0.05
2023-11-23 02:19:48,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.124e+01 8.364e+01 9.070e+01 9.690e+01 1.146e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-23 02:20:02,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2194546.6666666665, ans=0.05
2023-11-23 02:20:06,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2194546.6666666665, ans=0.1
2023-11-23 02:20:09,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2194546.6666666665, ans=0.2
2023-11-23 02:20:09,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2194546.6666666665, ans=0.1
2023-11-23 02:20:13,041 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4550, loss[loss=0.07289, simple_loss=0.08495, pruned_loss=0.01798, audio_tagging_loss=0.01244, over 15198.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.0931, pruned_loss=0.01427, audio_tagging_loss=0.009076, over 3053541.80 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:20:18,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.11 vs. limit=15.0
2023-11-23 02:20:20,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329200
2023-11-23 02:20:37,732 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.05 vs. limit=15.0
2023-11-23 02:21:01,336 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.96 vs. limit=22.5
2023-11-23 02:21:01,780 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:21:04,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2194880.0, ans=0.1
2023-11-23 02:21:11,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2194880.0, ans=0.125
2023-11-23 02:21:16,483 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4600, loss[loss=0.05109, simple_loss=0.06487, pruned_loss=0.0115, audio_tagging_loss=0.007161, over 15309.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.092, pruned_loss=0.01397, audio_tagging_loss=0.009172, over 3053918.05 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:21:18,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2194946.6666666665, ans=0.125
2023-11-23 02:21:22,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2194946.6666666665, ans=0.125
2023-11-23 02:21:24,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329250
2023-11-23 02:21:24,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2194946.6666666665, ans=0.025
2023-11-23 02:21:39,624 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.84 vs. limit=10.0
2023-11-23 02:21:58,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.216e+01 8.878e+01 9.478e+01 1.349e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 02:22:02,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2195146.6666666665, ans=0.07
2023-11-23 02:22:11,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2195213.3333333335, ans=0.125
2023-11-23 02:22:21,458 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4650, loss[loss=0.07955, simple_loss=0.1034, pruned_loss=0.01938, audio_tagging_loss=0.008458, over 16341.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09174, pruned_loss=0.01402, audio_tagging_loss=0.009291, over 3049485.44 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:22:21,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2195280.0, ans=0.0
2023-11-23 02:22:29,314 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329300
2023-11-23 02:22:39,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2195346.6666666665, ans=0.0
2023-11-23 02:23:01,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2195480.0, ans=0.125
2023-11-23 02:23:27,287 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4700, loss[loss=0.0806, simple_loss=0.1134, pruned_loss=0.01523, audio_tagging_loss=0.008679, over 15717.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.093, pruned_loss=0.01449, audio_tagging_loss=0.009386, over 3058120.89 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:23:34,802 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329350
2023-11-23 02:23:42,651 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.02 vs. limit=6.0
2023-11-23 02:23:56,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2195746.6666666665, ans=0.0
2023-11-23 02:23:58,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2195746.6666666665, ans=0.1
2023-11-23 02:24:06,156 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-23 02:24:08,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.09 vs. limit=22.5
2023-11-23 02:24:09,207 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.291e+01 9.016e+01 9.707e+01 1.177e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 02:24:31,718 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4750, loss[loss=0.08383, simple_loss=0.1139, pruned_loss=0.01736, audio_tagging_loss=0.009508, over 14742.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09281, pruned_loss=0.01448, audio_tagging_loss=0.009542, over 3048199.17 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:24:34,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2195946.6666666665, ans=0.95
2023-11-23 02:24:35,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.58 vs. limit=15.0
2023-11-23 02:24:39,005 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329400
2023-11-23 02:24:45,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2196013.3333333335, ans=0.0
2023-11-23 02:24:58,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2196080.0, ans=0.0
2023-11-23 02:25:05,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2196080.0, ans=0.05
2023-11-23 02:25:31,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2196213.3333333335, ans=0.2
2023-11-23 02:25:37,108 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4800, loss[loss=0.06573, simple_loss=0.08467, pruned_loss=0.01501, audio_tagging_loss=0.008386, over 14471.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09175, pruned_loss=0.01433, audio_tagging_loss=0.00961, over 3046078.22 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:25:44,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329450
2023-11-23 02:26:02,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2196413.3333333335, ans=0.2
2023-11-23 02:26:11,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2196413.3333333335, ans=0.125
2023-11-23 02:26:12,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2196413.3333333335, ans=0.0
2023-11-23 02:26:19,706 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.219e+01 8.831e+01 9.548e+01 1.229e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 02:26:35,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2196546.6666666665, ans=0.125
2023-11-23 02:26:43,270 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4850, loss[loss=0.09047, simple_loss=0.1209, pruned_loss=0.02208, audio_tagging_loss=0.00796, over 14881.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.091, pruned_loss=0.01407, audio_tagging_loss=0.009696, over 3044433.84 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:26:50,749 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329500
2023-11-23 02:26:56,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2196680.0, ans=0.05
2023-11-23 02:26:59,539 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.81 vs. limit=15.0
2023-11-23 02:27:01,742 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:27:16,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2196746.6666666665, ans=0.2
2023-11-23 02:27:18,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2196746.6666666665, ans=0.125
2023-11-23 02:27:20,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0
2023-11-23 02:27:31,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2196813.3333333335, ans=0.125
2023-11-23 02:27:42,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2196880.0, ans=0.1
2023-11-23 02:27:47,941 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4900, loss[loss=0.07863, simple_loss=0.1139, pruned_loss=0.0141, audio_tagging_loss=0.007564, over 14707.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09169, pruned_loss=0.01422, audio_tagging_loss=0.009638, over 3040932.98 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:27:49,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2196946.6666666665, ans=0.0
2023-11-23 02:27:49,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2196946.6666666665, ans=0.125
2023-11-23 02:27:55,521 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329550
2023-11-23 02:28:31,615 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.170e+01 8.747e+01 9.254e+01 1.120e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 02:28:39,293 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:28:40,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2197213.3333333335, ans=0.1
2023-11-23 02:28:52,710 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 4950, loss[loss=0.06568, simple_loss=0.0797, pruned_loss=0.01461, audio_tagging_loss=0.01122, over 15425.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09262, pruned_loss=0.01442, audio_tagging_loss=0.009369, over 3039572.64 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:29:00,739 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329600
2023-11-23 02:29:01,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2197280.0, ans=0.1
2023-11-23 02:29:01,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.86 vs. limit=22.5
2023-11-23 02:29:14,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2197346.6666666665, ans=0.2
2023-11-23 02:29:18,142 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0
2023-11-23 02:29:19,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2197413.3333333335, ans=0.05
2023-11-23 02:29:35,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2197480.0, ans=0.125
2023-11-23 02:29:59,488 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5000, loss[loss=0.06468, simple_loss=0.08576, pruned_loss=0.01243, audio_tagging_loss=0.009367, over 13679.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09204, pruned_loss=0.01427, audio_tagging_loss=0.009339, over 3036123.34 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:30:07,543 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329650
2023-11-23 02:30:13,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2197680.0, ans=0.125
2023-11-23 02:30:21,341 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:30:39,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.54 vs. limit=6.0
2023-11-23 02:30:41,659 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.477e+01 9.069e+01 9.699e+01 1.131e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-23 02:30:49,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2197813.3333333335, ans=0.09899494936611666
2023-11-23 02:30:59,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2197880.0, ans=0.125
2023-11-23 02:31:04,431 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5050, loss[loss=0.06795, simple_loss=0.09004, pruned_loss=0.01649, audio_tagging_loss=0.00645, over 14132.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09239, pruned_loss=0.01427, audio_tagging_loss=0.009242, over 3040166.22 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:31:10,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.06 vs. limit=22.5
2023-11-23 02:31:11,895 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329700
2023-11-23 02:31:27,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2198013.3333333335, ans=0.125
2023-11-23 02:31:28,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2198080.0, ans=0.125
2023-11-23 02:31:39,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.61 vs. limit=15.0
2023-11-23 02:31:54,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2198146.6666666665, ans=0.125
2023-11-23 02:31:55,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2198213.3333333335, ans=0.0
2023-11-23 02:32:08,929 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5100, loss[loss=0.05435, simple_loss=0.07374, pruned_loss=0.008552, audio_tagging_loss=0.008931, over 15043.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09145, pruned_loss=0.01415, audio_tagging_loss=0.009283, over 3040769.08 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:32:16,313 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329750
2023-11-23 02:32:37,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2198413.3333333335, ans=0.0
2023-11-23 02:32:39,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.01 vs. limit=22.5
2023-11-23 02:32:40,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2198413.3333333335, ans=0.09899494936611666
2023-11-23 02:32:51,745 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.231e+01 8.134e+01 8.796e+01 9.528e+01 1.100e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 02:33:07,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.00 vs. limit=15.0
2023-11-23 02:33:11,252 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.67 vs. limit=15.0
2023-11-23 02:33:13,799 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5150, loss[loss=0.06377, simple_loss=0.085, pruned_loss=0.01212, audio_tagging_loss=0.009155, over 15073.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09117, pruned_loss=0.01414, audio_tagging_loss=0.009329, over 3043499.49 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:33:14,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2198613.3333333335, ans=0.2
2023-11-23 02:33:23,176 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329800
2023-11-23 02:33:27,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2198680.0, ans=0.1
2023-11-23 02:33:46,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2198746.6666666665, ans=0.1
2023-11-23 02:34:02,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2198813.3333333335, ans=0.125
2023-11-23 02:34:20,427 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5200, loss[loss=0.05157, simple_loss=0.06066, pruned_loss=0.00995, audio_tagging_loss=0.01129, over 14751.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09275, pruned_loss=0.01448, audio_tagging_loss=0.009285, over 3036872.11 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:34:27,867 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329850
2023-11-23 02:34:40,787 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:34:41,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2199013.3333333335, ans=0.125
2023-11-23 02:34:45,859 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:34:46,137 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.33 vs. limit=15.0
2023-11-23 02:34:52,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2199080.0, ans=0.125
2023-11-23 02:35:05,352 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.387e+01 9.079e+01 9.668e+01 1.776e+02, threshold=1.816e+02, percent-clipped=1.0
2023-11-23 02:35:25,317 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5250, loss[loss=0.06346, simple_loss=0.07915, pruned_loss=0.01355, audio_tagging_loss=0.01033, over 15194.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.0929, pruned_loss=0.01451, audio_tagging_loss=0.009246, over 3037473.90 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:35:32,867 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329900
2023-11-23 02:35:35,956 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0
2023-11-23 02:35:38,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.91 vs. limit=15.0
2023-11-23 02:36:01,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2199413.3333333335, ans=0.0
2023-11-23 02:36:28,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2199546.6666666665, ans=0.2
2023-11-23 02:36:30,304 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5300, loss[loss=0.08898, simple_loss=0.112, pruned_loss=0.02338, audio_tagging_loss=0.0096, over 15981.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09335, pruned_loss=0.01441, audio_tagging_loss=0.009169, over 3045368.85 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:36:39,678 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 329950
2023-11-23 02:36:45,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2199680.0, ans=0.0
2023-11-23 02:36:49,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2199680.0, ans=0.125
2023-11-23 02:37:14,803 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.403e+01 8.848e+01 9.438e+01 1.127e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 02:37:37,493 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5350, loss[loss=0.0611, simple_loss=0.07903, pruned_loss=0.01247, audio_tagging_loss=0.009117, over 14405.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09347, pruned_loss=0.01435, audio_tagging_loss=0.009128, over 3040576.01 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:37:37,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2199946.6666666665, ans=0.125
2023-11-23 02:37:44,893 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330000
2023-11-23 02:37:45,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2199946.6666666665, ans=0.2
2023-11-23 02:37:59,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2200013.3333333335, ans=0.0
2023-11-23 02:38:24,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2200146.6666666665, ans=0.0
2023-11-23 02:38:25,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2200146.6666666665, ans=0.2
2023-11-23 02:38:27,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2200146.6666666665, ans=0.07
2023-11-23 02:38:30,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2200213.3333333335, ans=0.125
2023-11-23 02:38:42,227 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5400, loss[loss=0.08359, simple_loss=0.1053, pruned_loss=0.02241, audio_tagging_loss=0.008512, over 15342.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09253, pruned_loss=0.01411, audio_tagging_loss=0.009274, over 3037500.15 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:38:49,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330050
2023-11-23 02:38:57,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2200346.6666666665, ans=0.0
2023-11-23 02:38:57,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2200346.6666666665, ans=0.1
2023-11-23 02:39:16,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2200413.3333333335, ans=0.125
2023-11-23 02:39:19,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2200413.3333333335, ans=0.0
2023-11-23 02:39:27,025 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.068e+01 8.843e+01 9.502e+01 1.253e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 02:39:27,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2200480.0, ans=0.0
2023-11-23 02:39:47,562 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5450, loss[loss=0.06416, simple_loss=0.085, pruned_loss=0.01403, audio_tagging_loss=0.007637, over 14996.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09224, pruned_loss=0.01422, audio_tagging_loss=0.009278, over 3038955.44 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:39:49,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2200613.3333333335, ans=0.0
2023-11-23 02:39:56,126 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330100
2023-11-23 02:39:56,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2200613.3333333335, ans=0.125
2023-11-23 02:40:02,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2200680.0, ans=0.125
2023-11-23 02:40:18,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2200746.6666666665, ans=0.07
2023-11-23 02:40:48,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2200880.0, ans=0.125
2023-11-23 02:40:53,880 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5500, loss[loss=0.07516, simple_loss=0.1016, pruned_loss=0.01537, audio_tagging_loss=0.008989, over 14910.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09318, pruned_loss=0.01449, audio_tagging_loss=0.009249, over 3040683.83 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:40:54,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2200946.6666666665, ans=0.125
2023-11-23 02:41:01,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330150
2023-11-23 02:41:13,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2201013.3333333335, ans=0.125
2023-11-23 02:41:25,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2201080.0, ans=0.125
2023-11-23 02:41:36,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2201146.6666666665, ans=0.0
2023-11-23 02:41:37,661 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.324e+01 8.941e+01 9.835e+01 1.232e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 02:41:40,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2201146.6666666665, ans=0.2
2023-11-23 02:41:43,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2201146.6666666665, ans=0.0
2023-11-23 02:41:44,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2201213.3333333335, ans=0.125
2023-11-23 02:41:50,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2201213.3333333335, ans=0.1
2023-11-23 02:41:51,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2201213.3333333335, ans=15.0
2023-11-23 02:41:56,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten.whitening_limit, batch_count=2201213.3333333335, ans=22.5
2023-11-23 02:41:57,981 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5550, loss[loss=0.06942, simple_loss=0.09705, pruned_loss=0.01115, audio_tagging_loss=0.009746, over 15615.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09237, pruned_loss=0.01434, audio_tagging_loss=0.009366, over 3044081.25 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:42:05,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330200
2023-11-23 02:42:09,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2201346.6666666665, ans=0.125
2023-11-23 02:42:21,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2201346.6666666665, ans=0.1
2023-11-23 02:42:22,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2201413.3333333335, ans=0.125
2023-11-23 02:42:25,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.98 vs. limit=15.0
2023-11-23 02:42:35,983 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:42:36,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2201480.0, ans=10.0
2023-11-23 02:43:02,323 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5600, loss[loss=0.0796, simple_loss=0.09729, pruned_loss=0.02131, audio_tagging_loss=0.00965, over 14969.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09259, pruned_loss=0.01441, audio_tagging_loss=0.009466, over 3037491.99 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:43:10,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330250
2023-11-23 02:43:13,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2201613.3333333335, ans=0.0
2023-11-23 02:43:16,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.24 vs. limit=15.0
2023-11-23 02:43:18,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2201680.0, ans=0.125
2023-11-23 02:43:26,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2201680.0, ans=0.1
2023-11-23 02:43:31,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2201746.6666666665, ans=0.125
2023-11-23 02:43:31,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.26 vs. limit=15.0
2023-11-23 02:43:34,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.95 vs. limit=10.0
2023-11-23 02:43:47,709 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.200e+01 8.820e+01 9.406e+01 1.283e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 02:43:49,092 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:43:51,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2201813.3333333335, ans=0.1
2023-11-23 02:44:00,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2201880.0, ans=0.125
2023-11-23 02:44:08,218 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5650, loss[loss=0.07019, simple_loss=0.09659, pruned_loss=0.01084, audio_tagging_loss=0.01105, over 16130.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09267, pruned_loss=0.01431, audio_tagging_loss=0.009557, over 3043512.15 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:44:13,902 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.58 vs. limit=15.0
2023-11-23 02:44:15,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330300
2023-11-23 02:44:18,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2201946.6666666665, ans=0.125
2023-11-23 02:44:41,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2202080.0, ans=0.125
2023-11-23 02:44:58,589 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:45:03,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2202213.3333333335, ans=0.125
2023-11-23 02:45:11,945 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5700, loss[loss=0.05644, simple_loss=0.07964, pruned_loss=0.007687, audio_tagging_loss=0.008932, over 15060.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09219, pruned_loss=0.01408, audio_tagging_loss=0.009529, over 3041213.06 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:45:20,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330350
2023-11-23 02:45:34,323 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.23 vs. limit=15.0
2023-11-23 02:45:55,723 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.84 vs. limit=10.0
2023-11-23 02:45:57,494 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.129e+01 8.046e+01 8.620e+01 9.394e+01 1.276e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-23 02:46:16,715 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5750, loss[loss=0.07517, simple_loss=0.09895, pruned_loss=0.01569, audio_tagging_loss=0.01001, over 16097.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09188, pruned_loss=0.01397, audio_tagging_loss=0.009397, over 3041832.89 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:46:18,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2202613.3333333335, ans=0.2
2023-11-23 02:46:24,313 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330400
2023-11-23 02:46:24,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2202613.3333333335, ans=0.125
2023-11-23 02:46:55,954 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.74 vs. limit=10.0
2023-11-23 02:47:06,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2202813.3333333335, ans=0.0
2023-11-23 02:47:06,486 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0
2023-11-23 02:47:10,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.11 vs. limit=15.0
2023-11-23 02:47:22,032 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5800, loss[loss=0.06863, simple_loss=0.08553, pruned_loss=0.01457, audio_tagging_loss=0.0113, over 15923.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09269, pruned_loss=0.01414, audio_tagging_loss=0.009287, over 3046997.62 frames. ], batch size: 62, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:47:29,584 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330450
2023-11-23 02:47:51,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2203080.0, ans=0.125
2023-11-23 02:47:53,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2203080.0, ans=0.0
2023-11-23 02:48:06,942 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.335e+01 8.418e+01 9.019e+01 9.630e+01 1.229e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 02:48:10,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2203146.6666666665, ans=0.2
2023-11-23 02:48:24,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2203213.3333333335, ans=0.1
2023-11-23 02:48:25,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2203280.0, ans=0.1
2023-11-23 02:48:26,304 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5850, loss[loss=0.07485, simple_loss=0.1005, pruned_loss=0.01597, audio_tagging_loss=0.008642, over 15665.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09219, pruned_loss=0.01395, audio_tagging_loss=0.009219, over 3053093.27 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:48:33,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330500
2023-11-23 02:48:46,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2203346.6666666665, ans=0.125
2023-11-23 02:48:53,125 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:48:53,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.35 vs. limit=22.5
2023-11-23 02:48:54,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2203413.3333333335, ans=0.0
2023-11-23 02:49:04,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2203480.0, ans=0.025
2023-11-23 02:49:24,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2203546.6666666665, ans=0.2
2023-11-23 02:49:30,673 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5900, loss[loss=0.06912, simple_loss=0.099, pruned_loss=0.01262, audio_tagging_loss=0.006995, over 16460.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09288, pruned_loss=0.01408, audio_tagging_loss=0.009128, over 3060846.94 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:49:30,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2203613.3333333335, ans=0.125
2023-11-23 02:49:38,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330550
2023-11-23 02:50:15,281 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.273e+01 9.005e+01 9.559e+01 1.170e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 02:50:35,421 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 5950, loss[loss=0.08022, simple_loss=0.1088, pruned_loss=0.0176, audio_tagging_loss=0.008206, over 15524.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09243, pruned_loss=0.01394, audio_tagging_loss=0.009048, over 3066186.18 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:50:35,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2203946.6666666665, ans=0.05
2023-11-23 02:50:39,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2203946.6666666665, ans=0.125
2023-11-23 02:50:43,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330600
2023-11-23 02:50:44,073 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.02 vs. limit=15.0
2023-11-23 02:50:53,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2204013.3333333335, ans=0.125
2023-11-23 02:51:02,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2204080.0, ans=0.0
2023-11-23 02:51:06,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2204080.0, ans=0.0
2023-11-23 02:51:14,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2204146.6666666665, ans=0.125
2023-11-23 02:51:14,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2204146.6666666665, ans=0.2
2023-11-23 02:51:27,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2204213.3333333335, ans=0.125
2023-11-23 02:51:40,710 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6000, loss[loss=0.07333, simple_loss=0.1005, pruned_loss=0.0147, audio_tagging_loss=0.008377, over 17194.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09343, pruned_loss=0.0141, audio_tagging_loss=0.009023, over 3061484.37 frames. ], batch size: 64, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:51:40,711 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 02:52:24,767 INFO [train_asr.py:1253] (2/4) Epoch 28, validation: loss=0.05863, simple_loss=0.05128, pruned_loss=0.0051, audio_tagging_loss=0.02789, over 4681554.00 frames. 
2023-11-23 02:52:24,768 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 02:52:28,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2204280.0, ans=0.125
2023-11-23 02:52:33,033 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330650
2023-11-23 02:52:44,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2204346.6666666665, ans=0.125
2023-11-23 02:52:53,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.92 vs. limit=15.0
2023-11-23 02:52:55,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2204413.3333333335, ans=0.125
2023-11-23 02:53:10,010 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.180e+01 8.677e+01 9.605e+01 1.280e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 02:53:11,340 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:53:15,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2204546.6666666665, ans=0.025
2023-11-23 02:53:22,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2204546.6666666665, ans=0.0
2023-11-23 02:53:30,848 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6050, loss[loss=0.09428, simple_loss=0.131, pruned_loss=0.02148, audio_tagging_loss=0.007298, over 15259.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09364, pruned_loss=0.01423, audio_tagging_loss=0.009028, over 3059612.17 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:53:32,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2204613.3333333335, ans=0.125
2023-11-23 02:53:33,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.55 vs. limit=15.0
2023-11-23 02:53:38,454 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330700
2023-11-23 02:53:50,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2204680.0, ans=0.0
2023-11-23 02:54:13,446 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.87 vs. limit=22.5
2023-11-23 02:54:26,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2204880.0, ans=0.07
2023-11-23 02:54:29,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2204880.0, ans=0.125
2023-11-23 02:54:35,066 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6100, loss[loss=0.09559, simple_loss=0.1277, pruned_loss=0.02211, audio_tagging_loss=0.009614, over 15890.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09381, pruned_loss=0.01444, audio_tagging_loss=0.009046, over 3052239.70 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:54:42,612 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330750
2023-11-23 02:54:49,339 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.17 vs. limit=15.0
2023-11-23 02:54:50,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2205013.3333333335, ans=0.1
2023-11-23 02:55:05,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.93 vs. limit=22.5
2023-11-23 02:55:15,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2205146.6666666665, ans=0.0
2023-11-23 02:55:21,019 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 7.997e+01 8.605e+01 9.181e+01 1.192e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-23 02:55:22,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2205146.6666666665, ans=0.1
2023-11-23 02:55:29,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2205213.3333333335, ans=0.125
2023-11-23 02:55:36,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2205213.3333333335, ans=0.125
2023-11-23 02:55:39,755 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6150, loss[loss=0.08232, simple_loss=0.1057, pruned_loss=0.01956, audio_tagging_loss=0.009898, over 14971.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09399, pruned_loss=0.01456, audio_tagging_loss=0.009028, over 3045235.79 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:55:48,254 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330800
2023-11-23 02:55:55,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2205346.6666666665, ans=0.2
2023-11-23 02:56:35,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2205546.6666666665, ans=0.2
2023-11-23 02:56:45,213 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6200, loss[loss=0.09579, simple_loss=0.1298, pruned_loss=0.02311, audio_tagging_loss=0.007761, over 15815.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09385, pruned_loss=0.01456, audio_tagging_loss=0.009151, over 3044559.28 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:56:49,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2205613.3333333335, ans=0.05
2023-11-23 02:56:53,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330850
2023-11-23 02:57:13,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2205746.6666666665, ans=0.0
2023-11-23 02:57:28,390 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.73 vs. limit=22.5
2023-11-23 02:57:30,718 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.609e+01 8.249e+01 8.951e+01 9.781e+01 1.178e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 02:57:45,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2205880.0, ans=0.5
2023-11-23 02:57:48,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2205880.0, ans=0.2
2023-11-23 02:57:50,231 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6250, loss[loss=0.05105, simple_loss=0.06738, pruned_loss=0.007216, audio_tagging_loss=0.01014, over 14915.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09334, pruned_loss=0.01451, audio_tagging_loss=0.009243, over 3044352.08 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:57:54,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2205946.6666666665, ans=0.1
2023-11-23 02:57:57,663 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330900
2023-11-23 02:57:59,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2205946.6666666665, ans=0.125
2023-11-23 02:58:05,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2206013.3333333335, ans=0.2
2023-11-23 02:58:10,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2206013.3333333335, ans=0.09899494936611666
2023-11-23 02:58:24,729 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.72 vs. limit=10.0
2023-11-23 02:58:38,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2206146.6666666665, ans=0.0
2023-11-23 02:58:40,978 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:58:53,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2206280.0, ans=0.2
2023-11-23 02:58:54,285 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6300, loss[loss=0.08207, simple_loss=0.1082, pruned_loss=0.01765, audio_tagging_loss=0.01031, over 16526.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09416, pruned_loss=0.01452, audio_tagging_loss=0.00926, over 3050824.31 frames. ], batch size: 62, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:58:56,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2206280.0, ans=0.125
2023-11-23 02:59:01,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 330950
2023-11-23 02:59:02,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2206280.0, ans=0.125
2023-11-23 02:59:03,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2206280.0, ans=0.125
2023-11-23 02:59:16,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2206346.6666666665, ans=0.5
2023-11-23 02:59:39,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2206480.0, ans=0.2
2023-11-23 02:59:40,240 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.187e+01 8.713e+01 9.379e+01 1.255e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-23 02:59:59,643 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6350, loss[loss=0.05727, simple_loss=0.07595, pruned_loss=0.009325, audio_tagging_loss=0.009975, over 14533.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09278, pruned_loss=0.01421, audio_tagging_loss=0.009352, over 3046988.86 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:00:08,862 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331000
2023-11-23 03:00:21,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2206680.0, ans=0.125
2023-11-23 03:00:25,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2206746.6666666665, ans=0.2
2023-11-23 03:00:39,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.99 vs. limit=6.0
2023-11-23 03:00:40,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=2206813.3333333335, ans=10.0
2023-11-23 03:00:51,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2206880.0, ans=0.125
2023-11-23 03:00:52,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2206880.0, ans=0.95
2023-11-23 03:00:58,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2206880.0, ans=0.125
2023-11-23 03:01:05,833 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6400, loss[loss=0.06987, simple_loss=0.09196, pruned_loss=0.01606, audio_tagging_loss=0.007829, over 14445.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09237, pruned_loss=0.01411, audio_tagging_loss=0.009448, over 3048824.36 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:01:13,430 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331050
2023-11-23 03:01:15,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2206946.6666666665, ans=0.1
2023-11-23 03:01:18,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2207013.3333333335, ans=0.05
2023-11-23 03:01:36,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2207080.0, ans=0.125
2023-11-23 03:01:52,582 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.045e+01 8.704e+01 9.425e+01 1.351e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-23 03:01:59,577 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.73 vs. limit=15.0
2023-11-23 03:02:04,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.13 vs. limit=22.5
2023-11-23 03:02:09,961 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6450, loss[loss=0.08102, simple_loss=0.1076, pruned_loss=0.01581, audio_tagging_loss=0.01141, over 14162.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09247, pruned_loss=0.01409, audio_tagging_loss=0.009475, over 3050825.87 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:02:17,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331100
2023-11-23 03:02:21,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2207346.6666666665, ans=0.1
2023-11-23 03:02:21,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2207346.6666666665, ans=0.0
2023-11-23 03:02:33,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2207346.6666666665, ans=0.0
2023-11-23 03:02:34,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2207346.6666666665, ans=0.125
2023-11-23 03:03:13,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2207613.3333333335, ans=0.0
2023-11-23 03:03:15,315 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6500, loss[loss=0.08693, simple_loss=0.1238, pruned_loss=0.01809, audio_tagging_loss=0.006928, over 15896.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09235, pruned_loss=0.01411, audio_tagging_loss=0.009508, over 3052617.75 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:03:23,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331150
2023-11-23 03:03:47,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2207746.6666666665, ans=0.0
2023-11-23 03:03:58,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2207813.3333333335, ans=0.125
2023-11-23 03:04:00,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2207813.3333333335, ans=0.125
2023-11-23 03:04:01,372 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 8.162e+01 8.865e+01 9.552e+01 1.149e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 03:04:03,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.55 vs. limit=15.0
2023-11-23 03:04:21,225 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6550, loss[loss=0.05817, simple_loss=0.08187, pruned_loss=0.009877, audio_tagging_loss=0.00736, over 15439.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09204, pruned_loss=0.0141, audio_tagging_loss=0.009322, over 3053510.82 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:04:28,948 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331200
2023-11-23 03:04:45,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2208080.0, ans=0.125
2023-11-23 03:05:14,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2208213.3333333335, ans=0.2
2023-11-23 03:05:24,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2208280.0, ans=0.5
2023-11-23 03:05:25,327 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6600, loss[loss=0.06766, simple_loss=0.08049, pruned_loss=0.01969, audio_tagging_loss=0.007727, over 14554.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09371, pruned_loss=0.01441, audio_tagging_loss=0.009109, over 3052803.65 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:05:31,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2208280.0, ans=0.125
2023-11-23 03:05:32,760 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331250
2023-11-23 03:05:35,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2208280.0, ans=0.2
2023-11-23 03:05:37,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2208346.6666666665, ans=0.1
2023-11-23 03:05:47,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2208346.6666666665, ans=0.0
2023-11-23 03:05:47,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2208346.6666666665, ans=0.2
2023-11-23 03:05:51,378 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.74 vs. limit=15.0
2023-11-23 03:06:01,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2208413.3333333335, ans=0.125
2023-11-23 03:06:12,265 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.179e+01 8.913e+01 9.440e+01 1.641e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 03:06:29,555 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6650, loss[loss=0.065, simple_loss=0.08325, pruned_loss=0.01376, audio_tagging_loss=0.009614, over 14677.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09383, pruned_loss=0.01444, audio_tagging_loss=0.009127, over 3050184.16 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:06:37,768 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331300
2023-11-23 03:06:46,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2208680.0, ans=0.125
2023-11-23 03:06:56,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2208746.6666666665, ans=0.05
2023-11-23 03:06:58,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2208746.6666666665, ans=0.125
2023-11-23 03:07:07,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2208813.3333333335, ans=0.125
2023-11-23 03:07:07,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2208813.3333333335, ans=0.0
2023-11-23 03:07:34,606 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6700, loss[loss=0.0777, simple_loss=0.1062, pruned_loss=0.01748, audio_tagging_loss=0.007096, over 14676.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09353, pruned_loss=0.01441, audio_tagging_loss=0.00915, over 3046970.40 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:07:43,172 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331350
2023-11-23 03:08:11,568 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:08:22,300 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.251e+01 8.844e+01 9.540e+01 1.359e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 03:08:25,532 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.47 vs. limit=22.5
2023-11-23 03:08:27,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2209213.3333333335, ans=0.0
2023-11-23 03:08:40,175 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6750, loss[loss=0.0522, simple_loss=0.05717, pruned_loss=0.01104, audio_tagging_loss=0.01257, over 14593.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09325, pruned_loss=0.01444, audio_tagging_loss=0.009093, over 3040454.51 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:08:42,973 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:08:47,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331400
2023-11-23 03:09:18,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2209480.0, ans=0.1
2023-11-23 03:09:22,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2209480.0, ans=0.0
2023-11-23 03:09:26,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2209480.0, ans=0.125
2023-11-23 03:09:39,054 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:09:45,062 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6800, loss[loss=0.07538, simple_loss=0.1058, pruned_loss=0.01369, audio_tagging_loss=0.008802, over 14620.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.0931, pruned_loss=0.0143, audio_tagging_loss=0.009132, over 3041278.41 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:09:53,297 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331450
2023-11-23 03:10:00,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2209680.0, ans=0.125
2023-11-23 03:10:02,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.58 vs. limit=15.0
2023-11-23 03:10:10,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2209746.6666666665, ans=0.0
2023-11-23 03:10:13,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2209746.6666666665, ans=0.2
2023-11-23 03:10:16,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2209746.6666666665, ans=0.0
2023-11-23 03:10:25,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.59 vs. limit=15.0
2023-11-23 03:10:31,991 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.723e+01 8.090e+01 8.741e+01 9.523e+01 1.182e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 03:10:43,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2209880.0, ans=0.125
2023-11-23 03:10:45,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2209880.0, ans=0.125
2023-11-23 03:10:46,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2209880.0, ans=0.0
2023-11-23 03:10:51,114 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6850, loss[loss=0.07317, simple_loss=0.1024, pruned_loss=0.01429, audio_tagging_loss=0.007661, over 16162.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09267, pruned_loss=0.01418, audio_tagging_loss=0.009074, over 3041494.65 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:10:58,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331500
2023-11-23 03:11:18,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2210080.0, ans=0.125
2023-11-23 03:11:19,546 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.80 vs. limit=10.0
2023-11-23 03:11:31,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2210146.6666666665, ans=0.0
2023-11-23 03:11:37,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2210146.6666666665, ans=0.2
2023-11-23 03:11:49,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2210213.3333333335, ans=0.0
2023-11-23 03:11:56,032 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6900, loss[loss=0.06512, simple_loss=0.08062, pruned_loss=0.01267, audio_tagging_loss=0.01214, over 14320.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09243, pruned_loss=0.01415, audio_tagging_loss=0.00908, over 3039172.87 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:12:03,650 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331550
2023-11-23 03:12:14,104 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.26 vs. limit=15.0
2023-11-23 03:12:41,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2210480.0, ans=0.1
2023-11-23 03:12:43,535 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.094e+01 8.690e+01 9.402e+01 1.184e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-23 03:12:44,879 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:12:54,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2210546.6666666665, ans=0.125
2023-11-23 03:13:00,463 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 6950, loss[loss=0.05622, simple_loss=0.06421, pruned_loss=0.013, audio_tagging_loss=0.01112, over 15487.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09145, pruned_loss=0.0138, audio_tagging_loss=0.009115, over 3043631.71 frames. ], batch size: 62, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:13:03,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2210613.3333333335, ans=0.0
2023-11-23 03:13:04,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2210613.3333333335, ans=0.125
2023-11-23 03:13:07,980 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331600
2023-11-23 03:13:22,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2210680.0, ans=0.1
2023-11-23 03:13:39,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2210813.3333333335, ans=0.2
2023-11-23 03:13:46,881 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.82 vs. limit=15.0
2023-11-23 03:13:49,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2210813.3333333335, ans=22.5
2023-11-23 03:14:06,096 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7000, loss[loss=0.07034, simple_loss=0.09329, pruned_loss=0.01282, audio_tagging_loss=0.01087, over 14788.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09194, pruned_loss=0.01407, audio_tagging_loss=0.009127, over 3044949.82 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:14:06,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2210946.6666666665, ans=0.125
2023-11-23 03:14:14,256 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331650
2023-11-23 03:14:44,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2211146.6666666665, ans=0.125
2023-11-23 03:14:53,597 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.328e+01 8.294e+01 8.963e+01 9.673e+01 1.557e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 03:15:01,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2211213.3333333335, ans=0.2
2023-11-23 03:15:08,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2211213.3333333335, ans=0.2
2023-11-23 03:15:10,762 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7050, loss[loss=0.0667, simple_loss=0.09573, pruned_loss=0.01034, audio_tagging_loss=0.008495, over 15268.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09139, pruned_loss=0.01401, audio_tagging_loss=0.009117, over 3047424.27 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:15:18,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331700
2023-11-23 03:15:28,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2211346.6666666665, ans=0.0
2023-11-23 03:15:28,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2211346.6666666665, ans=0.125
2023-11-23 03:16:14,847 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7100, loss[loss=0.07843, simple_loss=0.1042, pruned_loss=0.01547, audio_tagging_loss=0.01086, over 16535.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09148, pruned_loss=0.01412, audio_tagging_loss=0.009167, over 3037905.64 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:16:18,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2211613.3333333335, ans=0.07
2023-11-23 03:16:22,285 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331750
2023-11-23 03:16:22,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2211613.3333333335, ans=0.2
2023-11-23 03:16:41,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2211746.6666666665, ans=0.2
2023-11-23 03:17:02,517 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.317e+01 8.441e+01 9.062e+01 9.663e+01 1.297e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 03:17:07,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2211880.0, ans=0.1
2023-11-23 03:17:08,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2211880.0, ans=0.0
2023-11-23 03:17:12,920 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.82 vs. limit=12.0
2023-11-23 03:17:15,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2211880.0, ans=0.0
2023-11-23 03:17:19,049 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7150, loss[loss=0.04991, simple_loss=0.06451, pruned_loss=0.005533, audio_tagging_loss=0.01213, over 14442.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09205, pruned_loss=0.01421, audio_tagging_loss=0.009397, over 3043289.28 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:17:26,871 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331800
2023-11-23 03:17:38,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2212013.3333333335, ans=0.0
2023-11-23 03:17:50,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2212080.0, ans=0.125
2023-11-23 03:18:22,457 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7200, loss[loss=0.08386, simple_loss=0.1148, pruned_loss=0.01796, audio_tagging_loss=0.008513, over 15878.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09269, pruned_loss=0.01427, audio_tagging_loss=0.009396, over 3041504.81 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:18:26,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2212280.0, ans=0.1
2023-11-23 03:18:26,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.33 vs. limit=15.0
2023-11-23 03:18:29,883 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331850
2023-11-23 03:18:56,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2212413.3333333335, ans=0.1
2023-11-23 03:19:02,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2212480.0, ans=0.125
2023-11-23 03:19:08,959 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.113e+01 8.957e+01 9.713e+01 1.213e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 03:19:23,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2212613.3333333335, ans=0.0
2023-11-23 03:19:24,864 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7250, loss[loss=0.0735, simple_loss=0.09177, pruned_loss=0.01744, audio_tagging_loss=0.01017, over 14349.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09281, pruned_loss=0.01417, audio_tagging_loss=0.009433, over 3040063.04 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:19:32,946 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331900
2023-11-23 03:19:33,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2212613.3333333335, ans=0.1
2023-11-23 03:20:02,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2212813.3333333335, ans=0.125
2023-11-23 03:20:07,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2212813.3333333335, ans=0.0
2023-11-23 03:20:29,111 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7300, loss[loss=0.08035, simple_loss=0.1124, pruned_loss=0.0157, audio_tagging_loss=0.008444, over 16081.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09224, pruned_loss=0.01416, audio_tagging_loss=0.009423, over 3044468.48 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:20:36,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 331950
2023-11-23 03:20:37,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2212946.6666666665, ans=0.09899494936611666
2023-11-23 03:20:39,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2212946.6666666665, ans=0.2
2023-11-23 03:20:44,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2213013.3333333335, ans=0.125
2023-11-23 03:20:51,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2213013.3333333335, ans=0.0
2023-11-23 03:20:57,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2213080.0, ans=0.1
2023-11-23 03:21:13,567 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.12 vs. limit=15.0
2023-11-23 03:21:15,975 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.147e+01 8.723e+01 9.402e+01 1.286e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 03:21:18,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2213213.3333333335, ans=0.0
2023-11-23 03:21:24,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.32 vs. limit=15.0
2023-11-23 03:21:33,052 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7350, loss[loss=0.04504, simple_loss=0.06659, pruned_loss=0.005002, audio_tagging_loss=0.006742, over 14627.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09225, pruned_loss=0.01415, audio_tagging_loss=0.009256, over 3044097.06 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:21:40,387 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332000
2023-11-23 03:21:51,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2213346.6666666665, ans=0.0
2023-11-23 03:21:54,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2213346.6666666665, ans=0.0
2023-11-23 03:21:55,453 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0
2023-11-23 03:22:39,812 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7400, loss[loss=0.06339, simple_loss=0.07637, pruned_loss=0.01321, audio_tagging_loss=0.01199, over 14916.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09291, pruned_loss=0.0142, audio_tagging_loss=0.009112, over 3045996.28 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:22:47,286 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332050
2023-11-23 03:22:47,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2213613.3333333335, ans=0.125
2023-11-23 03:22:51,759 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:23:00,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2213680.0, ans=0.1
2023-11-23 03:23:06,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2213746.6666666665, ans=0.04949747468305833
2023-11-23 03:23:26,923 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-23 03:23:28,618 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.451e+01 8.955e+01 9.673e+01 1.193e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 03:23:44,631 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7450, loss[loss=0.08453, simple_loss=0.1154, pruned_loss=0.01644, audio_tagging_loss=0.01038, over 14508.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09256, pruned_loss=0.01416, audio_tagging_loss=0.009068, over 3051695.43 frames. ], batch size: 53, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:23:53,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332100
2023-11-23 03:23:58,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2214013.3333333335, ans=0.0
2023-11-23 03:24:11,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2214080.0, ans=0.125
2023-11-23 03:24:12,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2214080.0, ans=0.0
2023-11-23 03:24:21,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2214146.6666666665, ans=0.125
2023-11-23 03:24:48,988 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7500, loss[loss=0.0683, simple_loss=0.09683, pruned_loss=0.01255, audio_tagging_loss=0.007336, over 15772.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09281, pruned_loss=0.01416, audio_tagging_loss=0.008973, over 3051616.97 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:24:56,659 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332150
2023-11-23 03:25:28,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2214480.0, ans=0.125
2023-11-23 03:25:34,981 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.40 vs. limit=10.0
2023-11-23 03:25:37,020 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:25:37,859 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.184e+01 8.938e+01 9.763e+01 1.268e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 03:25:43,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.44 vs. limit=22.5
2023-11-23 03:25:52,739 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7550, loss[loss=0.0829, simple_loss=0.1241, pruned_loss=0.01507, audio_tagging_loss=0.00577, over 16592.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09294, pruned_loss=0.01433, audio_tagging_loss=0.008895, over 3047302.93 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:25:56,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2214613.3333333335, ans=0.125
2023-11-23 03:26:00,064 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332200
2023-11-23 03:26:55,314 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.53 vs. limit=10.0
2023-11-23 03:26:55,767 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7600, loss[loss=0.06399, simple_loss=0.08149, pruned_loss=0.01428, audio_tagging_loss=0.008968, over 15946.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.0914, pruned_loss=0.01422, audio_tagging_loss=0.009115, over 3046833.34 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:27:03,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2214946.6666666665, ans=0.2
2023-11-23 03:27:04,401 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332250
2023-11-23 03:27:05,122 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.52 vs. limit=15.0
2023-11-23 03:27:13,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2215013.3333333335, ans=0.0
2023-11-23 03:27:21,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2215080.0, ans=0.125
2023-11-23 03:27:26,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2215080.0, ans=0.125
2023-11-23 03:27:29,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.97 vs. limit=15.0
2023-11-23 03:27:29,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.31 vs. limit=15.0
2023-11-23 03:27:31,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2215080.0, ans=0.125
2023-11-23 03:27:32,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2215080.0, ans=0.125
2023-11-23 03:27:41,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2215146.6666666665, ans=0.125
2023-11-23 03:27:44,364 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.025e+01 8.532e+01 9.208e+01 1.132e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-23 03:28:01,619 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7650, loss[loss=0.06329, simple_loss=0.08047, pruned_loss=0.01533, audio_tagging_loss=0.007727, over 14497.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09201, pruned_loss=0.01425, audio_tagging_loss=0.00908, over 3045874.14 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:28:02,130 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.76 vs. limit=12.0
2023-11-23 03:28:09,048 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332300
2023-11-23 03:28:42,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2215480.0, ans=0.2
2023-11-23 03:28:47,473 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.96 vs. limit=22.5
2023-11-23 03:28:59,709 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.70 vs. limit=15.0
2023-11-23 03:29:05,104 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7700, loss[loss=0.04117, simple_loss=0.04037, pruned_loss=0.00856, audio_tagging_loss=0.01242, over 13746.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.0917, pruned_loss=0.01415, audio_tagging_loss=0.009128, over 3044401.20 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:29:05,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.71 vs. limit=15.0
2023-11-23 03:29:05,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0
2023-11-23 03:29:12,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332350
2023-11-23 03:29:12,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2215613.3333333335, ans=0.125
2023-11-23 03:29:12,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2215613.3333333335, ans=0.125
2023-11-23 03:29:13,087 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.21 vs. limit=15.0
2023-11-23 03:29:19,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.70 vs. limit=15.0
2023-11-23 03:29:23,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2215680.0, ans=0.125
2023-11-23 03:29:35,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2215746.6666666665, ans=0.125
2023-11-23 03:29:35,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2215746.6666666665, ans=0.1
2023-11-23 03:29:36,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.39 vs. limit=15.0
2023-11-23 03:29:55,232 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.060e+01 8.592e+01 9.630e+01 1.129e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-23 03:30:08,623 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7750, loss[loss=0.07576, simple_loss=0.09649, pruned_loss=0.01746, audio_tagging_loss=0.01005, over 15207.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09135, pruned_loss=0.01413, audio_tagging_loss=0.009245, over 3042723.04 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:30:09,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.36 vs. limit=15.0
2023-11-23 03:30:16,640 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332400
2023-11-23 03:30:24,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2216013.3333333335, ans=0.125
2023-11-23 03:30:51,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2216146.6666666665, ans=0.125
2023-11-23 03:30:55,275 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:31:14,408 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7800, loss[loss=0.07629, simple_loss=0.08923, pruned_loss=0.01823, audio_tagging_loss=0.01344, over 14083.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09317, pruned_loss=0.01442, audio_tagging_loss=0.009164, over 3049647.35 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:31:20,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2216280.0, ans=0.1
2023-11-23 03:31:21,750 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332450
2023-11-23 03:31:44,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2216413.3333333335, ans=0.125
2023-11-23 03:32:00,485 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.83 vs. limit=15.0
2023-11-23 03:32:00,754 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.06 vs. limit=15.0
2023-11-23 03:32:04,587 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.076e+01 8.334e+01 8.987e+01 9.736e+01 1.242e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 03:32:18,087 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7850, loss[loss=0.07965, simple_loss=0.1145, pruned_loss=0.01515, audio_tagging_loss=0.007265, over 14991.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09226, pruned_loss=0.01415, audio_tagging_loss=0.009298, over 3055558.30 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:32:20,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2216613.3333333335, ans=0.5
2023-11-23 03:32:25,690 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332500
2023-11-23 03:32:26,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.08 vs. limit=15.0
2023-11-23 03:32:55,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.77 vs. limit=12.0
2023-11-23 03:33:00,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2216813.3333333335, ans=0.125
2023-11-23 03:33:08,911 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.06 vs. limit=6.0
2023-11-23 03:33:11,326 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.26 vs. limit=6.0
2023-11-23 03:33:19,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2216880.0, ans=0.125
2023-11-23 03:33:19,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2216880.0, ans=0.125
2023-11-23 03:33:21,803 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7900, loss[loss=0.07646, simple_loss=0.09915, pruned_loss=0.01553, audio_tagging_loss=0.01136, over 15581.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09215, pruned_loss=0.01412, audio_tagging_loss=0.009451, over 3054451.68 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:33:29,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332550
2023-11-23 03:33:50,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2217080.0, ans=0.025
2023-11-23 03:33:50,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0
2023-11-23 03:33:53,821 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:34:10,855 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.71 vs. limit=15.0
2023-11-23 03:34:11,386 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.317e+01 8.969e+01 9.609e+01 1.150e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 03:34:21,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.46 vs. limit=15.0
2023-11-23 03:34:25,263 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.34 vs. limit=5.0
2023-11-23 03:34:26,056 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 7950, loss[loss=0.08167, simple_loss=0.1205, pruned_loss=0.01538, audio_tagging_loss=0.006055, over 14954.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09227, pruned_loss=0.01418, audio_tagging_loss=0.009451, over 3057809.56 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:34:33,830 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332600
2023-11-23 03:34:40,824 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:34:41,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2217346.6666666665, ans=0.0
2023-11-23 03:34:59,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2217413.3333333335, ans=0.125
2023-11-23 03:34:59,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2217413.3333333335, ans=0.125
2023-11-23 03:35:00,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2217413.3333333335, ans=0.125
2023-11-23 03:35:08,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2217480.0, ans=0.125
2023-11-23 03:35:20,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2217546.6666666665, ans=0.125
2023-11-23 03:35:20,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2217546.6666666665, ans=0.1
2023-11-23 03:35:30,610 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8000, loss[loss=0.08297, simple_loss=0.1088, pruned_loss=0.01927, audio_tagging_loss=0.009306, over 16230.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09185, pruned_loss=0.0142, audio_tagging_loss=0.009508, over 3054303.20 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:35:37,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332650
2023-11-23 03:35:59,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2217746.6666666665, ans=0.125
2023-11-23 03:36:06,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2217746.6666666665, ans=0.125
2023-11-23 03:36:08,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2217813.3333333335, ans=0.0
2023-11-23 03:36:20,151 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.081e+01 8.720e+01 9.413e+01 1.910e+02, threshold=1.744e+02, percent-clipped=1.0
2023-11-23 03:36:28,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2217880.0, ans=0.2
2023-11-23 03:36:33,475 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8050, loss[loss=0.07448, simple_loss=0.09816, pruned_loss=0.0168, audio_tagging_loss=0.008597, over 14794.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09134, pruned_loss=0.01401, audio_tagging_loss=0.009553, over 3048931.77 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:36:40,713 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332700
2023-11-23 03:37:06,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2218080.0, ans=0.1
2023-11-23 03:37:31,002 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=15.0
2023-11-23 03:37:37,082 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8100, loss[loss=0.06705, simple_loss=0.08507, pruned_loss=0.01395, audio_tagging_loss=0.01057, over 15178.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09262, pruned_loss=0.01428, audio_tagging_loss=0.0094, over 3047672.17 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:37:45,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332750
2023-11-23 03:38:09,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2218413.3333333335, ans=0.125
2023-11-23 03:38:10,587 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.03 vs. limit=15.0
2023-11-23 03:38:16,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.26 vs. limit=15.0
2023-11-23 03:38:27,686 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.342e+01 8.867e+01 9.602e+01 1.203e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 03:38:31,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2218546.6666666665, ans=0.0
2023-11-23 03:38:40,751 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8150, loss[loss=0.06149, simple_loss=0.0777, pruned_loss=0.01027, audio_tagging_loss=0.01238, over 14756.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.0934, pruned_loss=0.01432, audio_tagging_loss=0.00913, over 3057385.19 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:38:48,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332800
2023-11-23 03:39:00,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2218680.0, ans=0.125
2023-11-23 03:39:13,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2218746.6666666665, ans=0.125
2023-11-23 03:39:19,745 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.13 vs. limit=22.5
2023-11-23 03:39:30,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2218813.3333333335, ans=0.0
2023-11-23 03:39:45,651 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8200, loss[loss=0.09275, simple_loss=0.1316, pruned_loss=0.0179, audio_tagging_loss=0.009071, over 16903.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.094, pruned_loss=0.01435, audio_tagging_loss=0.009053, over 3058937.47 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:39:45,682 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:39:53,176 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332850
2023-11-23 03:40:11,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.21 vs. limit=15.0
2023-11-23 03:40:24,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.32 vs. limit=15.0
2023-11-23 03:40:27,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.87 vs. limit=15.0
2023-11-23 03:40:37,030 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.334e+01 8.200e+01 8.804e+01 9.476e+01 1.277e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 03:40:49,905 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8250, loss[loss=0.04719, simple_loss=0.05202, pruned_loss=0.007281, audio_tagging_loss=0.0139, over 14550.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09441, pruned_loss=0.01446, audio_tagging_loss=0.009064, over 3062185.49 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:40:58,509 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332900
2023-11-23 03:40:59,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2219280.0, ans=0.2
2023-11-23 03:41:54,267 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8300, loss[loss=0.07061, simple_loss=0.08693, pruned_loss=0.01838, audio_tagging_loss=0.008763, over 14864.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09413, pruned_loss=0.01452, audio_tagging_loss=0.009066, over 3052176.88 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:41:55,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2219613.3333333335, ans=0.2
2023-11-23 03:41:56,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2219613.3333333335, ans=0.04949747468305833
2023-11-23 03:41:56,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2219613.3333333335, ans=0.0
2023-11-23 03:41:59,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2219613.3333333335, ans=0.0
2023-11-23 03:42:01,588 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 332950
2023-11-23 03:42:01,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2219613.3333333335, ans=0.1
2023-11-23 03:42:11,028 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:42:19,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2219746.6666666665, ans=0.0
2023-11-23 03:42:44,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2219880.0, ans=0.0
2023-11-23 03:42:45,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2219880.0, ans=0.125
2023-11-23 03:42:46,801 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.213e+01 8.531e+01 9.088e+01 9.532e+01 2.273e+02, threshold=1.818e+02, percent-clipped=2.0
2023-11-23 03:42:54,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2219880.0, ans=0.1
2023-11-23 03:42:54,481 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:42:57,834 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8350, loss[loss=0.08096, simple_loss=0.1109, pruned_loss=0.01908, audio_tagging_loss=0.006413, over 15108.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.0939, pruned_loss=0.01439, audio_tagging_loss=0.009061, over 3053514.07 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 8.0
2023-11-23 03:42:59,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2219946.6666666665, ans=0.1
2023-11-23 03:43:05,976 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333000
2023-11-23 03:43:29,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2220080.0, ans=0.0
2023-11-23 03:43:31,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2220080.0, ans=0.125
2023-11-23 03:43:35,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2220080.0, ans=0.125
2023-11-23 03:43:43,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2220146.6666666665, ans=0.05
2023-11-23 03:44:03,100 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8400, loss[loss=0.08786, simple_loss=0.1159, pruned_loss=0.02182, audio_tagging_loss=0.008093, over 15245.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09349, pruned_loss=0.01431, audio_tagging_loss=0.009039, over 3050912.30 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:44:10,399 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333050
2023-11-23 03:44:55,758 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 7.977e+01 8.728e+01 9.553e+01 1.214e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-23 03:44:58,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2220546.6666666665, ans=0.05
2023-11-23 03:45:07,880 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8450, loss[loss=0.06614, simple_loss=0.08986, pruned_loss=0.01136, audio_tagging_loss=0.009853, over 17108.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09318, pruned_loss=0.01408, audio_tagging_loss=0.009033, over 3057315.34 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:45:15,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333100
2023-11-23 03:45:22,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.11 vs. limit=15.0
2023-11-23 03:45:55,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2220813.3333333335, ans=0.0
2023-11-23 03:46:12,219 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8500, loss[loss=0.07906, simple_loss=0.09985, pruned_loss=0.01758, audio_tagging_loss=0.01155, over 14917.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09342, pruned_loss=0.01421, audio_tagging_loss=0.009058, over 3063600.28 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:46:19,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333150
2023-11-23 03:46:19,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2220946.6666666665, ans=0.1
2023-11-23 03:46:20,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-23 03:46:21,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2220946.6666666665, ans=15.0
2023-11-23 03:46:27,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2221013.3333333335, ans=0.0
2023-11-23 03:47:03,977 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.153e+01 8.814e+01 9.449e+01 1.976e+02, threshold=1.763e+02, percent-clipped=1.0
2023-11-23 03:47:07,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2221213.3333333335, ans=0.1
2023-11-23 03:47:15,519 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8550, loss[loss=0.07783, simple_loss=0.1016, pruned_loss=0.01698, audio_tagging_loss=0.01004, over 15502.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09276, pruned_loss=0.01421, audio_tagging_loss=0.009072, over 3060837.42 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:47:21,818 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=15.0
2023-11-23 03:47:23,680 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333200
2023-11-23 03:47:26,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2221280.0, ans=0.125
2023-11-23 03:47:27,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.64 vs. limit=6.0
2023-11-23 03:47:31,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2221346.6666666665, ans=0.07
2023-11-23 03:47:32,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2221346.6666666665, ans=0.125
2023-11-23 03:47:43,719 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.71 vs. limit=10.0
2023-11-23 03:47:49,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2221413.3333333335, ans=0.125
2023-11-23 03:48:18,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2221546.6666666665, ans=0.2
2023-11-23 03:48:19,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.39 vs. limit=12.0
2023-11-23 03:48:20,339 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8600, loss[loss=0.07048, simple_loss=0.09139, pruned_loss=0.01556, audio_tagging_loss=0.009225, over 14956.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09298, pruned_loss=0.01446, audio_tagging_loss=0.009122, over 3058682.42 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:48:21,781 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:48:27,571 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333250
2023-11-23 03:48:30,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2221613.3333333335, ans=0.1
2023-11-23 03:48:42,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.85 vs. limit=10.0
2023-11-23 03:48:56,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2221813.3333333335, ans=0.125
2023-11-23 03:49:07,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2221813.3333333335, ans=0.0
2023-11-23 03:49:11,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2221880.0, ans=0.0
2023-11-23 03:49:12,187 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.558e+01 9.096e+01 9.715e+01 2.705e+02, threshold=1.819e+02, percent-clipped=1.0
2023-11-23 03:49:18,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2221880.0, ans=22.5
2023-11-23 03:49:23,121 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8650, loss[loss=0.06614, simple_loss=0.08482, pruned_loss=0.01429, audio_tagging_loss=0.009439, over 14607.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09322, pruned_loss=0.01444, audio_tagging_loss=0.009161, over 3056476.37 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:49:23,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2221946.6666666665, ans=0.125
2023-11-23 03:49:30,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333300
2023-11-23 03:49:42,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.16 vs. limit=22.5
2023-11-23 03:50:05,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2222146.6666666665, ans=0.1
2023-11-23 03:50:22,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2222213.3333333335, ans=0.125
2023-11-23 03:50:25,860 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8700, loss[loss=0.08558, simple_loss=0.1096, pruned_loss=0.01809, audio_tagging_loss=0.0127, over 15850.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.0941, pruned_loss=0.01446, audio_tagging_loss=0.00921, over 3061682.81 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:50:27,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2222280.0, ans=0.1
2023-11-23 03:50:29,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2222280.0, ans=0.1
2023-11-23 03:50:34,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333350
2023-11-23 03:50:39,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2222346.6666666665, ans=0.125
2023-11-23 03:50:50,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2222413.3333333335, ans=0.125
2023-11-23 03:50:59,604 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.65 vs. limit=15.0
2023-11-23 03:51:06,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2222480.0, ans=0.0
2023-11-23 03:51:17,202 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.257e+01 8.537e+01 9.054e+01 9.922e+01 1.181e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 03:51:27,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2222546.6666666665, ans=0.1
2023-11-23 03:51:30,106 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8750, loss[loss=0.06419, simple_loss=0.08432, pruned_loss=0.01332, audio_tagging_loss=0.008702, over 15027.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.0941, pruned_loss=0.0145, audio_tagging_loss=0.009366, over 3058901.25 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:51:37,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333400
2023-11-23 03:52:09,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2222813.3333333335, ans=0.125
2023-11-23 03:52:33,793 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8800, loss[loss=0.07523, simple_loss=0.09232, pruned_loss=0.01676, audio_tagging_loss=0.01231, over 15224.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09445, pruned_loss=0.01465, audio_tagging_loss=0.009378, over 3052437.03 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:52:35,876 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.18 vs. limit=15.0
2023-11-23 03:52:37,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.42 vs. limit=22.5
2023-11-23 03:52:41,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333450
2023-11-23 03:52:49,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2223013.3333333335, ans=0.125
2023-11-23 03:53:14,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2223146.6666666665, ans=0.2
2023-11-23 03:53:20,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2223146.6666666665, ans=0.1
2023-11-23 03:53:20,546 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.93 vs. limit=22.5
2023-11-23 03:53:25,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2223213.3333333335, ans=0.07
2023-11-23 03:53:26,105 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.173e+01 8.235e+01 8.890e+01 9.571e+01 2.057e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-23 03:53:26,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2223213.3333333335, ans=0.125
2023-11-23 03:53:37,219 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8850, loss[loss=0.1045, simple_loss=0.1432, pruned_loss=0.02671, audio_tagging_loss=0.006218, over 15385.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09482, pruned_loss=0.01471, audio_tagging_loss=0.009315, over 3052480.99 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:53:45,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333500
2023-11-23 03:53:48,744 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:54:27,920 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.18 vs. limit=15.0
2023-11-23 03:54:38,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2223546.6666666665, ans=0.125
2023-11-23 03:54:40,980 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8900, loss[loss=0.07982, simple_loss=0.09667, pruned_loss=0.01671, audio_tagging_loss=0.01478, over 15923.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09577, pruned_loss=0.01502, audio_tagging_loss=0.009157, over 3051566.20 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:54:49,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2223613.3333333335, ans=0.0
2023-11-23 03:54:50,370 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333550
2023-11-23 03:54:57,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2223680.0, ans=0.0
2023-11-23 03:54:59,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.05 vs. limit=12.0
2023-11-23 03:55:06,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2223746.6666666665, ans=0.125
2023-11-23 03:55:34,576 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.471e+01 9.014e+01 9.931e+01 1.268e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 03:55:38,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2223880.0, ans=0.0
2023-11-23 03:55:38,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2223880.0, ans=0.0
2023-11-23 03:55:46,437 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 8950, loss[loss=0.06982, simple_loss=0.09693, pruned_loss=0.01248, audio_tagging_loss=0.008872, over 14974.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09512, pruned_loss=0.01489, audio_tagging_loss=0.009058, over 3046577.31 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:55:53,792 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333600
2023-11-23 03:56:14,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2224080.0, ans=0.125
2023-11-23 03:56:50,586 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9000, loss[loss=0.06397, simple_loss=0.08053, pruned_loss=0.01436, audio_tagging_loss=0.009335, over 15393.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09414, pruned_loss=0.01462, audio_tagging_loss=0.008921, over 3043963.32 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:56:50,587 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 03:57:33,769 INFO [train_asr.py:1253] (2/4) Epoch 28, validation: loss=0.05919, simple_loss=0.05113, pruned_loss=0.004978, audio_tagging_loss=0.02865, over 4681554.00 frames. 
2023-11-23 03:57:33,769 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 03:57:34,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.07 vs. limit=15.0
2023-11-23 03:57:41,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333650
2023-11-23 03:57:43,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2224280.0, ans=0.035
2023-11-23 03:58:13,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2224480.0, ans=0.1
2023-11-23 03:58:24,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2224546.6666666665, ans=0.2
2023-11-23 03:58:28,164 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.337e+01 8.839e+01 9.721e+01 1.226e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-23 03:58:37,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2224613.3333333335, ans=0.0
2023-11-23 03:58:37,980 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9050, loss[loss=0.06101, simple_loss=0.07888, pruned_loss=0.01139, audio_tagging_loss=0.01017, over 15132.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09329, pruned_loss=0.01451, audio_tagging_loss=0.008995, over 3044648.83 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:58:45,327 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333700
2023-11-23 03:58:52,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2224680.0, ans=0.1
2023-11-23 03:59:17,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2224813.3333333335, ans=0.1
2023-11-23 03:59:28,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2224880.0, ans=0.125
2023-11-23 03:59:29,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2224880.0, ans=0.125
2023-11-23 03:59:32,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2224880.0, ans=0.0
2023-11-23 03:59:41,503 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9100, loss[loss=0.05907, simple_loss=0.07799, pruned_loss=0.008707, audio_tagging_loss=0.01136, over 15701.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09284, pruned_loss=0.01431, audio_tagging_loss=0.008957, over 3050373.86 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:59:42,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2224946.6666666665, ans=0.125
2023-11-23 03:59:49,511 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333750
2023-11-23 04:00:01,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2225013.3333333335, ans=0.2
2023-11-23 04:00:17,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.15 vs. limit=15.0
2023-11-23 04:00:29,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2225146.6666666665, ans=0.2
2023-11-23 04:00:34,986 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.085e+01 8.276e+01 8.928e+01 9.613e+01 1.240e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 04:00:43,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2225213.3333333335, ans=0.04949747468305833
2023-11-23 04:00:46,261 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9150, loss[loss=0.0555, simple_loss=0.0729, pruned_loss=0.01111, audio_tagging_loss=0.007937, over 14155.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09124, pruned_loss=0.01408, audio_tagging_loss=0.008977, over 3044293.02 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:00:54,071 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333800
2023-11-23 04:00:55,950 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0
2023-11-23 04:00:59,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2225346.6666666665, ans=0.125
2023-11-23 04:01:07,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2225346.6666666665, ans=0.0
2023-11-23 04:01:16,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2225413.3333333335, ans=0.1
2023-11-23 04:01:50,726 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9200, loss[loss=0.07576, simple_loss=0.108, pruned_loss=0.01314, audio_tagging_loss=0.008611, over 15952.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09136, pruned_loss=0.01401, audio_tagging_loss=0.008976, over 3045246.74 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:01:58,099 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333850
2023-11-23 04:02:27,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2225746.6666666665, ans=0.125
2023-11-23 04:02:38,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2225813.3333333335, ans=0.0
2023-11-23 04:02:44,367 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.331e+01 8.795e+01 9.315e+01 1.180e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 04:02:53,982 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9250, loss[loss=0.08958, simple_loss=0.1164, pruned_loss=0.02208, audio_tagging_loss=0.009314, over 15692.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09229, pruned_loss=0.01413, audio_tagging_loss=0.008959, over 3044852.72 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:03:01,444 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333900
2023-11-23 04:03:13,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2226013.3333333335, ans=0.0
2023-11-23 04:03:22,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2226080.0, ans=0.125
2023-11-23 04:03:41,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2226146.6666666665, ans=0.125
2023-11-23 04:03:58,069 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9300, loss[loss=0.08572, simple_loss=0.1153, pruned_loss=0.01793, audio_tagging_loss=0.01014, over 15177.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09256, pruned_loss=0.01421, audio_tagging_loss=0.009056, over 3044760.71 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:04:06,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 333950
2023-11-23 04:04:40,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2226480.0, ans=0.0
2023-11-23 04:04:52,645 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.259e+01 8.847e+01 9.648e+01 1.420e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 04:05:02,501 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9350, loss[loss=0.06863, simple_loss=0.09532, pruned_loss=0.01038, audio_tagging_loss=0.01059, over 15105.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.0925, pruned_loss=0.01416, audio_tagging_loss=0.009086, over 3049212.37 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:05:10,457 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334000
2023-11-23 04:05:43,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2226813.3333333335, ans=0.125
2023-11-23 04:05:44,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2226813.3333333335, ans=0.1
2023-11-23 04:05:48,560 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.66 vs. limit=12.0
2023-11-23 04:05:54,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2226880.0, ans=0.125
2023-11-23 04:05:55,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2226880.0, ans=0.0
2023-11-23 04:06:06,792 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9400, loss[loss=0.06746, simple_loss=0.09985, pruned_loss=0.01027, audio_tagging_loss=0.007269, over 15362.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09223, pruned_loss=0.01425, audio_tagging_loss=0.009186, over 3045180.97 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:06:14,279 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334050
2023-11-23 04:06:21,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2227013.3333333335, ans=0.2
2023-11-23 04:06:36,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2227080.0, ans=0.1
2023-11-23 04:06:48,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.07 vs. limit=6.0
2023-11-23 04:06:50,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2227146.6666666665, ans=0.0
2023-11-23 04:07:00,199 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.797e+01 8.264e+01 9.005e+01 9.544e+01 1.394e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 04:07:04,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2227213.3333333335, ans=0.0
2023-11-23 04:07:07,472 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:07:09,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.81 vs. limit=12.0
2023-11-23 04:07:10,612 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9450, loss[loss=0.06465, simple_loss=0.08776, pruned_loss=0.01338, audio_tagging_loss=0.007388, over 16781.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.0922, pruned_loss=0.01437, audio_tagging_loss=0.009305, over 3046997.45 frames. ], batch size: 63, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:07:18,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334100
2023-11-23 04:07:28,180 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.23 vs. limit=15.0
2023-11-23 04:08:08,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2227546.6666666665, ans=0.125
2023-11-23 04:08:15,163 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9500, loss[loss=0.06263, simple_loss=0.07864, pruned_loss=0.01297, audio_tagging_loss=0.01034, over 15453.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09219, pruned_loss=0.01426, audio_tagging_loss=0.009387, over 3044937.20 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:08:15,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2227613.3333333335, ans=0.5
2023-11-23 04:08:22,589 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334150
2023-11-23 04:08:22,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2227613.3333333335, ans=0.125
2023-11-23 04:08:23,016 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.71 vs. limit=10.0
2023-11-23 04:08:43,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2227746.6666666665, ans=0.0
2023-11-23 04:08:55,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2227813.3333333335, ans=0.2
2023-11-23 04:09:08,993 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.198e+01 8.793e+01 9.435e+01 1.623e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 04:09:19,595 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9550, loss[loss=0.07456, simple_loss=0.1004, pruned_loss=0.01614, audio_tagging_loss=0.008225, over 14837.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09132, pruned_loss=0.01407, audio_tagging_loss=0.009486, over 3038404.52 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:09:26,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334200
2023-11-23 04:09:31,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2228013.3333333335, ans=0.0
2023-11-23 04:09:32,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2228013.3333333335, ans=0.125
2023-11-23 04:10:23,895 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9600, loss[loss=0.09427, simple_loss=0.1196, pruned_loss=0.02591, audio_tagging_loss=0.008538, over 15886.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09169, pruned_loss=0.01422, audio_tagging_loss=0.00963, over 3042604.77 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:10:31,895 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334250
2023-11-23 04:10:45,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2228346.6666666665, ans=0.125
2023-11-23 04:10:53,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2228413.3333333335, ans=0.125
2023-11-23 04:11:17,368 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.386e+01 9.005e+01 9.855e+01 1.264e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 04:11:28,317 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9650, loss[loss=0.08302, simple_loss=0.1081, pruned_loss=0.02151, audio_tagging_loss=0.007457, over 15138.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09269, pruned_loss=0.01428, audio_tagging_loss=0.009457, over 3036182.15 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:11:29,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=2228613.3333333335, ans=0.1
2023-11-23 04:11:34,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2228613.3333333335, ans=0.125
2023-11-23 04:11:35,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334300
2023-11-23 04:11:38,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2228613.3333333335, ans=0.1
2023-11-23 04:11:40,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2228680.0, ans=0.2
2023-11-23 04:11:45,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2228680.0, ans=0.125
2023-11-23 04:12:24,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2228880.0, ans=0.1
2023-11-23 04:12:31,996 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9700, loss[loss=0.06489, simple_loss=0.08372, pruned_loss=0.0123, audio_tagging_loss=0.01073, over 14943.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09258, pruned_loss=0.01411, audio_tagging_loss=0.009244, over 3038585.33 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:12:33,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2228946.6666666665, ans=0.125
2023-11-23 04:12:39,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334350
2023-11-23 04:12:58,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2229080.0, ans=0.05
2023-11-23 04:13:11,895 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.26 vs. limit=15.0
2023-11-23 04:13:21,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2229146.6666666665, ans=0.0
2023-11-23 04:13:22,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2229213.3333333335, ans=0.1
2023-11-23 04:13:27,075 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.979e+01 8.051e+01 8.815e+01 9.310e+01 1.256e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 04:13:36,180 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9750, loss[loss=0.07889, simple_loss=0.1006, pruned_loss=0.019, audio_tagging_loss=0.009582, over 15888.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09213, pruned_loss=0.01399, audio_tagging_loss=0.009169, over 3037870.31 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:13:40,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2229280.0, ans=0.2
2023-11-23 04:13:44,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334400
2023-11-23 04:13:53,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=12.0
2023-11-23 04:14:12,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2229413.3333333335, ans=0.05
2023-11-23 04:14:33,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2229546.6666666665, ans=0.125
2023-11-23 04:14:41,480 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9800, loss[loss=0.05993, simple_loss=0.0778, pruned_loss=0.01083, audio_tagging_loss=0.0102, over 14969.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09241, pruned_loss=0.01411, audio_tagging_loss=0.009127, over 3046331.38 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:14:42,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2229613.3333333335, ans=0.1
2023-11-23 04:14:48,869 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334450
2023-11-23 04:14:57,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2229680.0, ans=0.1
2023-11-23 04:15:11,616 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.78 vs. limit=15.0
2023-11-23 04:15:18,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2229813.3333333335, ans=0.125
2023-11-23 04:15:18,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2229813.3333333335, ans=0.125
2023-11-23 04:15:23,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2229813.3333333335, ans=0.1
2023-11-23 04:15:29,649 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.84 vs. limit=15.0
2023-11-23 04:15:36,214 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.857e+01 8.531e+01 9.197e+01 9.757e+01 1.250e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 04:15:37,502 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:15:45,000 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9850, loss[loss=0.07306, simple_loss=0.07872, pruned_loss=0.01971, audio_tagging_loss=0.01399, over 14592.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09349, pruned_loss=0.01432, audio_tagging_loss=0.009054, over 3049868.74 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:15:52,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334500
2023-11-23 04:15:55,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2229946.6666666665, ans=0.125
2023-11-23 04:16:46,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2230213.3333333335, ans=0.1
2023-11-23 04:16:48,606 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9900, loss[loss=0.07275, simple_loss=0.09917, pruned_loss=0.01545, audio_tagging_loss=0.007711, over 14642.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09396, pruned_loss=0.01439, audio_tagging_loss=0.00897, over 3055723.11 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:16:51,637 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.46 vs. limit=6.0
2023-11-23 04:16:57,362 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334550
2023-11-23 04:17:04,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2230346.6666666665, ans=0.125
2023-11-23 04:17:06,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2230346.6666666665, ans=0.125
2023-11-23 04:17:10,046 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.80 vs. limit=15.0
2023-11-23 04:17:14,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.69 vs. limit=15.0
2023-11-23 04:17:32,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2230480.0, ans=0.0
2023-11-23 04:17:44,111 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.266e+01 8.858e+01 9.665e+01 1.141e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 04:17:46,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.74 vs. limit=15.0
2023-11-23 04:17:53,601 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 9950, loss[loss=0.09298, simple_loss=0.1305, pruned_loss=0.01997, audio_tagging_loss=0.007741, over 15780.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09401, pruned_loss=0.01447, audio_tagging_loss=0.008944, over 3055530.74 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:17:54,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.08 vs. limit=22.5
2023-11-23 04:17:55,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2230613.3333333335, ans=0.1
2023-11-23 04:17:55,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2230613.3333333335, ans=0.1
2023-11-23 04:18:00,919 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334600
2023-11-23 04:18:34,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2230813.3333333335, ans=0.125
2023-11-23 04:18:57,016 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10000, loss[loss=0.07676, simple_loss=0.1117, pruned_loss=0.01354, audio_tagging_loss=0.007354, over 15056.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09262, pruned_loss=0.01417, audio_tagging_loss=0.008951, over 3053960.74 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:19:04,406 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334650
2023-11-23 04:19:05,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2230946.6666666665, ans=0.0
2023-11-23 04:19:11,269 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.63 vs. limit=15.0
2023-11-23 04:19:26,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2231080.0, ans=0.125
2023-11-23 04:19:36,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2231146.6666666665, ans=0.2
2023-11-23 04:19:38,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2231146.6666666665, ans=0.125
2023-11-23 04:19:51,966 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.134e+01 8.342e+01 8.814e+01 9.358e+01 1.133e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 04:19:55,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2231213.3333333335, ans=0.09899494936611666
2023-11-23 04:19:57,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2231213.3333333335, ans=0.125
2023-11-23 04:20:00,516 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10050, loss[loss=0.08158, simple_loss=0.1113, pruned_loss=0.01907, audio_tagging_loss=0.006853, over 15189.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09254, pruned_loss=0.01419, audio_tagging_loss=0.009157, over 3048125.33 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:20:02,080 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:20:08,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334700
2023-11-23 04:20:33,707 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:20:52,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2231546.6666666665, ans=0.0
2023-11-23 04:21:06,014 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10100, loss[loss=0.09036, simple_loss=0.1147, pruned_loss=0.02415, audio_tagging_loss=0.00885, over 14800.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09295, pruned_loss=0.01431, audio_tagging_loss=0.009199, over 3045717.58 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:21:11,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.29 vs. limit=12.0
2023-11-23 04:21:13,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334750
2023-11-23 04:21:22,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2231680.0, ans=0.2
2023-11-23 04:21:50,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2231813.3333333335, ans=0.0
2023-11-23 04:21:57,915 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:22:02,741 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.395e+01 8.740e+01 9.651e+01 1.218e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 04:22:10,052 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10150, loss[loss=0.08243, simple_loss=0.1113, pruned_loss=0.01669, audio_tagging_loss=0.01008, over 15881.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09355, pruned_loss=0.01448, audio_tagging_loss=0.00913, over 3055333.94 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:22:11,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2231946.6666666665, ans=0.125
2023-11-23 04:22:17,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334800
2023-11-23 04:22:39,047 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:22:56,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2232146.6666666665, ans=0.125
2023-11-23 04:23:12,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2232280.0, ans=0.125
2023-11-23 04:23:13,269 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10200, loss[loss=0.1022, simple_loss=0.1502, pruned_loss=0.02258, audio_tagging_loss=0.004519, over 17052.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09394, pruned_loss=0.01441, audio_tagging_loss=0.009121, over 3059318.93 frames. ], batch size: 60, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:23:17,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2232280.0, ans=0.1
2023-11-23 04:23:20,604 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334850
2023-11-23 04:23:37,529 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:23:49,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2232413.3333333335, ans=0.035
2023-11-23 04:23:56,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2232480.0, ans=0.2
2023-11-23 04:24:01,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2232480.0, ans=0.2
2023-11-23 04:24:05,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.57 vs. limit=12.0
2023-11-23 04:24:08,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2232546.6666666665, ans=0.09899494936611666
2023-11-23 04:24:09,455 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.292e+01 8.929e+01 9.861e+01 1.255e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 04:24:17,277 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10250, loss[loss=0.04724, simple_loss=0.05259, pruned_loss=0.006986, audio_tagging_loss=0.01395, over 15311.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.0923, pruned_loss=0.01414, audio_tagging_loss=0.009296, over 3058094.76 frames. ], batch size: 61, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:24:25,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334900
2023-11-23 04:24:49,967 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:24:56,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2232813.3333333335, ans=0.125
2023-11-23 04:25:00,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2232813.3333333335, ans=0.2
2023-11-23 04:25:06,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2232813.3333333335, ans=0.1
2023-11-23 04:25:06,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2232813.3333333335, ans=0.125
2023-11-23 04:25:16,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2232880.0, ans=0.015
2023-11-23 04:25:22,427 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10300, loss[loss=0.06201, simple_loss=0.07517, pruned_loss=0.01097, audio_tagging_loss=0.01345, over 16422.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09225, pruned_loss=0.01408, audio_tagging_loss=0.009391, over 3057160.02 frames. ], batch size: 60, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:25:29,803 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 334950
2023-11-23 04:25:42,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.61 vs. limit=15.0
2023-11-23 04:25:55,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2233080.0, ans=0.1
2023-11-23 04:26:06,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2233146.6666666665, ans=0.0
2023-11-23 04:26:10,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2233146.6666666665, ans=0.07
2023-11-23 04:26:17,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.28 vs. limit=22.5
2023-11-23 04:26:19,725 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.243e+01 9.019e+01 9.717e+01 1.166e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 04:26:25,949 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10350, loss[loss=0.07384, simple_loss=0.09693, pruned_loss=0.01601, audio_tagging_loss=0.009367, over 14854.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09251, pruned_loss=0.0142, audio_tagging_loss=0.009444, over 3050066.30 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:26:31,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2233280.0, ans=0.125
2023-11-23 04:26:33,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335000
2023-11-23 04:26:39,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2233346.6666666665, ans=0.0
2023-11-23 04:26:42,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2233346.6666666665, ans=0.125
2023-11-23 04:27:07,410 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.47 vs. limit=15.0
2023-11-23 04:27:09,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2233480.0, ans=0.1
2023-11-23 04:27:13,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2233480.0, ans=0.125
2023-11-23 04:27:30,462 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10400, loss[loss=0.07897, simple_loss=0.1095, pruned_loss=0.01657, audio_tagging_loss=0.007658, over 15770.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09322, pruned_loss=0.01416, audio_tagging_loss=0.009382, over 3054803.06 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:27:31,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2233613.3333333335, ans=0.0
2023-11-23 04:27:39,130 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335050
2023-11-23 04:27:44,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2233680.0, ans=0.1
2023-11-23 04:27:52,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.96 vs. limit=10.0
2023-11-23 04:27:53,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2233680.0, ans=0.125
2023-11-23 04:28:06,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2233746.6666666665, ans=0.125
2023-11-23 04:28:09,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2233813.3333333335, ans=0.125
2023-11-23 04:28:30,419 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.149e+01 8.861e+01 9.460e+01 1.224e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 04:28:35,420 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10450, loss[loss=0.09105, simple_loss=0.1347, pruned_loss=0.01561, audio_tagging_loss=0.008087, over 15154.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09369, pruned_loss=0.01416, audio_tagging_loss=0.009286, over 3055136.32 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:28:35,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2233946.6666666665, ans=0.125
2023-11-23 04:28:36,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.85 vs. limit=15.0
2023-11-23 04:28:38,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2233946.6666666665, ans=0.0
2023-11-23 04:28:43,406 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335100
2023-11-23 04:28:47,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.21 vs. limit=15.0
2023-11-23 04:29:12,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2234146.6666666665, ans=15.0
2023-11-23 04:29:39,635 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10500, loss[loss=0.05143, simple_loss=0.0733, pruned_loss=0.007321, audio_tagging_loss=0.007458, over 15142.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09401, pruned_loss=0.01425, audio_tagging_loss=0.009135, over 3050640.74 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:29:47,114 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335150
2023-11-23 04:29:50,138 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.69 vs. limit=15.0
2023-11-23 04:30:06,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2234413.3333333335, ans=0.0
2023-11-23 04:30:25,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2234480.0, ans=0.1
2023-11-23 04:30:30,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2234546.6666666665, ans=0.0
2023-11-23 04:30:37,990 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.079e+01 8.755e+01 9.423e+01 1.196e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-23 04:30:39,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2234546.6666666665, ans=0.125
2023-11-23 04:30:42,946 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10550, loss[loss=0.07229, simple_loss=0.09761, pruned_loss=0.01399, audio_tagging_loss=0.009497, over 15185.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09423, pruned_loss=0.01452, audio_tagging_loss=0.009104, over 3043874.11 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:30:51,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335200
2023-11-23 04:30:52,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2234613.3333333335, ans=0.125
2023-11-23 04:31:47,909 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10600, loss[loss=0.06488, simple_loss=0.08736, pruned_loss=0.01185, audio_tagging_loss=0.009352, over 15229.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09415, pruned_loss=0.01448, audio_tagging_loss=0.009032, over 3043027.16 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:31:55,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335250
2023-11-23 04:31:58,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2234946.6666666665, ans=0.125
2023-11-23 04:32:23,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.14 vs. limit=15.0
2023-11-23 04:32:33,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.49 vs. limit=12.0
2023-11-23 04:32:40,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2235213.3333333335, ans=0.0
2023-11-23 04:32:46,330 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.415e+01 8.292e+01 8.923e+01 9.577e+01 1.197e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 04:32:50,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2235280.0, ans=0.1
2023-11-23 04:32:51,942 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10650, loss[loss=0.07178, simple_loss=0.0934, pruned_loss=0.01634, audio_tagging_loss=0.00874, over 15444.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09405, pruned_loss=0.01462, audio_tagging_loss=0.009078, over 3040984.83 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:32:56,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=2235280.0, ans=15.0
2023-11-23 04:32:59,220 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335300
2023-11-23 04:33:13,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2235346.6666666665, ans=0.125
2023-11-23 04:33:24,527 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.18 vs. limit=15.0
2023-11-23 04:33:25,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2235413.3333333335, ans=0.2
2023-11-23 04:33:29,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2235480.0, ans=0.125
2023-11-23 04:33:32,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.57 vs. limit=15.0
2023-11-23 04:33:55,240 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10700, loss[loss=0.07207, simple_loss=0.1013, pruned_loss=0.01251, audio_tagging_loss=0.008905, over 15021.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09349, pruned_loss=0.01438, audio_tagging_loss=0.009138, over 3046040.20 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:33:59,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2235613.3333333335, ans=0.125
2023-11-23 04:33:59,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2235613.3333333335, ans=0.125
2023-11-23 04:34:03,123 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335350
2023-11-23 04:34:36,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2235813.3333333335, ans=0.125
2023-11-23 04:34:38,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2235813.3333333335, ans=0.0
2023-11-23 04:34:42,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2235813.3333333335, ans=0.1
2023-11-23 04:34:51,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.37 vs. limit=12.0
2023-11-23 04:34:54,567 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.303e+01 8.893e+01 9.663e+01 2.175e+02, threshold=1.779e+02, percent-clipped=1.0
2023-11-23 04:34:59,982 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10750, loss[loss=0.08799, simple_loss=0.1263, pruned_loss=0.01616, audio_tagging_loss=0.008671, over 16349.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09311, pruned_loss=0.01438, audio_tagging_loss=0.009151, over 3048634.53 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:35:07,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335400
2023-11-23 04:35:10,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2235946.6666666665, ans=0.125
2023-11-23 04:35:22,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2236013.3333333335, ans=0.0
2023-11-23 04:35:30,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2236080.0, ans=0.1
2023-11-23 04:35:42,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=2236146.6666666665, ans=0.2
2023-11-23 04:36:00,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2236213.3333333335, ans=0.0
2023-11-23 04:36:03,428 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10800, loss[loss=0.07162, simple_loss=0.09427, pruned_loss=0.01637, audio_tagging_loss=0.008111, over 15877.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09275, pruned_loss=0.01418, audio_tagging_loss=0.00912, over 3053807.85 frames. ], batch size: 60, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:36:09,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2236280.0, ans=0.0
2023-11-23 04:36:11,598 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335450
2023-11-23 04:36:25,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2236346.6666666665, ans=0.0
2023-11-23 04:36:41,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2236480.0, ans=0.0
2023-11-23 04:36:41,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2236480.0, ans=0.0
2023-11-23 04:36:48,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2236480.0, ans=0.0
2023-11-23 04:36:48,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.50 vs. limit=15.0
2023-11-23 04:36:52,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=13.09 vs. limit=22.5
2023-11-23 04:37:00,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2236546.6666666665, ans=0.125
2023-11-23 04:37:00,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2236546.6666666665, ans=0.0
2023-11-23 04:37:02,343 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.072e+01 8.725e+01 9.352e+01 1.267e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 04:37:07,378 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10850, loss[loss=0.07781, simple_loss=0.1023, pruned_loss=0.02044, audio_tagging_loss=0.006223, over 15685.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09276, pruned_loss=0.01414, audio_tagging_loss=0.009116, over 3053669.78 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:37:15,402 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335500
2023-11-23 04:37:53,760 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.68 vs. limit=15.0
2023-11-23 04:37:56,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2236813.3333333335, ans=0.2
2023-11-23 04:37:58,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-23 04:38:00,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2236880.0, ans=0.125
2023-11-23 04:38:07,670 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:38:11,722 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10900, loss[loss=0.0974, simple_loss=0.1299, pruned_loss=0.02692, audio_tagging_loss=0.005547, over 15387.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09321, pruned_loss=0.01424, audio_tagging_loss=0.00913, over 3060004.96 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:38:11,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2236946.6666666665, ans=0.125
2023-11-23 04:38:19,179 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335550
2023-11-23 04:38:28,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2237013.3333333335, ans=0.125
2023-11-23 04:38:35,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2237080.0, ans=0.1
2023-11-23 04:38:46,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2237080.0, ans=0.0
2023-11-23 04:38:48,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2237146.6666666665, ans=0.125
2023-11-23 04:39:00,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.52 vs. limit=10.0
2023-11-23 04:39:01,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2237213.3333333335, ans=0.125
2023-11-23 04:39:10,059 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.359e+01 8.884e+01 9.760e+01 1.616e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 04:39:14,992 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 10950, loss[loss=0.06418, simple_loss=0.08326, pruned_loss=0.01121, audio_tagging_loss=0.01134, over 15141.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09256, pruned_loss=0.01399, audio_tagging_loss=0.009159, over 3055444.89 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:39:20,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2237280.0, ans=0.5
2023-11-23 04:39:22,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335600
2023-11-23 04:39:35,996 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.06 vs. limit=15.0
2023-11-23 04:39:49,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2237413.3333333335, ans=0.125
2023-11-23 04:39:51,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2237413.3333333335, ans=0.1
2023-11-23 04:40:02,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2237480.0, ans=0.0
2023-11-23 04:40:19,229 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11000, loss[loss=0.0604, simple_loss=0.07946, pruned_loss=0.01081, audio_tagging_loss=0.009856, over 15727.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09309, pruned_loss=0.01408, audio_tagging_loss=0.009172, over 3050407.58 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:40:25,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2237613.3333333335, ans=0.125
2023-11-23 04:40:26,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335650
2023-11-23 04:40:27,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2237613.3333333335, ans=0.125
2023-11-23 04:40:29,651 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:40:48,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2237746.6666666665, ans=0.0
2023-11-23 04:40:54,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2237746.6666666665, ans=0.05
2023-11-23 04:41:14,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2237880.0, ans=0.0
2023-11-23 04:41:16,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2237880.0, ans=0.07
2023-11-23 04:41:19,574 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.268e+01 8.870e+01 9.647e+01 1.227e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 04:41:21,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2237880.0, ans=0.1
2023-11-23 04:41:23,739 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11050, loss[loss=0.0694, simple_loss=0.09243, pruned_loss=0.01303, audio_tagging_loss=0.01015, over 15420.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09213, pruned_loss=0.01409, audio_tagging_loss=0.009243, over 3044644.39 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:41:31,659 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335700
2023-11-23 04:41:34,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2237946.6666666665, ans=0.125
2023-11-23 04:41:41,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2238013.3333333335, ans=0.035
2023-11-23 04:41:51,954 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0
2023-11-23 04:41:55,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2238080.0, ans=0.125
2023-11-23 04:42:18,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2238213.3333333335, ans=0.125
2023-11-23 04:42:27,701 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11100, loss[loss=0.08433, simple_loss=0.1139, pruned_loss=0.0183, audio_tagging_loss=0.009091, over 15923.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09341, pruned_loss=0.0144, audio_tagging_loss=0.009202, over 3039374.58 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:42:35,172 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335750
2023-11-23 04:43:00,047 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.03 vs. limit=15.0
2023-11-23 04:43:27,381 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.249e+01 8.341e+01 8.890e+01 9.757e+01 1.222e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 04:43:31,095 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11150, loss[loss=0.09398, simple_loss=0.1319, pruned_loss=0.01935, audio_tagging_loss=0.008675, over 15956.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09303, pruned_loss=0.01443, audio_tagging_loss=0.00932, over 3046220.70 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:43:39,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335800
2023-11-23 04:43:58,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2238746.6666666665, ans=0.125
2023-11-23 04:44:00,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2238746.6666666665, ans=0.015
2023-11-23 04:44:22,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2238880.0, ans=0.125
2023-11-23 04:44:35,961 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11200, loss[loss=0.07067, simple_loss=0.0931, pruned_loss=0.01577, audio_tagging_loss=0.008344, over 14179.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09306, pruned_loss=0.01449, audio_tagging_loss=0.009434, over 3049037.58 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:44:39,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2238946.6666666665, ans=0.0
2023-11-23 04:44:44,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335850
2023-11-23 04:44:49,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2239013.3333333335, ans=0.125
2023-11-23 04:45:04,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2239080.0, ans=0.0
2023-11-23 04:45:10,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2239080.0, ans=0.125
2023-11-23 04:45:16,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2239146.6666666665, ans=0.09899494936611666
2023-11-23 04:45:17,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2239146.6666666665, ans=0.0
2023-11-23 04:45:21,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.92 vs. limit=15.0
2023-11-23 04:45:31,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2239213.3333333335, ans=0.125
2023-11-23 04:45:34,164 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.81 vs. limit=22.5
2023-11-23 04:45:36,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2239213.3333333335, ans=0.2
2023-11-23 04:45:36,865 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.304e+01 9.043e+01 9.933e+01 1.318e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 04:45:40,647 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11250, loss[loss=0.05755, simple_loss=0.07115, pruned_loss=0.01242, audio_tagging_loss=0.009546, over 15454.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09256, pruned_loss=0.01437, audio_tagging_loss=0.009527, over 3050917.92 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:45:48,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335900
2023-11-23 04:46:12,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2239413.3333333335, ans=0.0
2023-11-23 04:46:36,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2239546.6666666665, ans=0.0
2023-11-23 04:46:38,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2239546.6666666665, ans=0.0
2023-11-23 04:46:42,923 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.87 vs. limit=15.0
2023-11-23 04:46:44,907 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11300, loss[loss=0.08177, simple_loss=0.1127, pruned_loss=0.0175, audio_tagging_loss=0.007913, over 14902.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09193, pruned_loss=0.01412, audio_tagging_loss=0.009386, over 3053001.15 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:46:52,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 335950
2023-11-23 04:47:12,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.66 vs. limit=12.0
2023-11-23 04:47:26,178 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.89 vs. limit=6.0
2023-11-23 04:47:43,899 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.333e+01 8.937e+01 9.779e+01 1.201e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 04:47:45,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2239880.0, ans=0.2
2023-11-23 04:47:47,562 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11350, loss[loss=0.07078, simple_loss=0.09455, pruned_loss=0.01662, audio_tagging_loss=0.006883, over 15807.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09204, pruned_loss=0.01403, audio_tagging_loss=0.009233, over 3050163.71 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:47:54,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2239946.6666666665, ans=0.05
2023-11-23 04:47:54,809 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.92 vs. limit=15.0
2023-11-23 04:47:57,501 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336000
2023-11-23 04:48:10,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2240013.3333333335, ans=0.0
2023-11-23 04:48:15,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2240013.3333333335, ans=0.125
2023-11-23 04:48:23,379 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:48:30,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2240146.6666666665, ans=0.2
2023-11-23 04:48:39,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2240146.6666666665, ans=0.1
2023-11-23 04:48:56,659 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11400, loss[loss=0.08602, simple_loss=0.1179, pruned_loss=0.01706, audio_tagging_loss=0.009986, over 14882.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09158, pruned_loss=0.014, audio_tagging_loss=0.009194, over 3048058.87 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:48:58,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.32 vs. limit=15.0
2023-11-23 04:49:01,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2240280.0, ans=0.2
2023-11-23 04:49:03,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336050
2023-11-23 04:49:22,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2240413.3333333335, ans=0.125
2023-11-23 04:49:27,906 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.90 vs. limit=22.5
2023-11-23 04:49:30,990 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.11 vs. limit=15.0
2023-11-23 04:49:48,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.49 vs. limit=15.0
2023-11-23 04:49:50,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2240546.6666666665, ans=0.1
2023-11-23 04:49:52,220 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.94 vs. limit=15.0
2023-11-23 04:49:56,449 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.358e+01 8.216e+01 8.830e+01 9.675e+01 1.432e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 04:49:56,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2240546.6666666665, ans=0.2
2023-11-23 04:50:00,126 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11450, loss[loss=0.0653, simple_loss=0.08449, pruned_loss=0.0143, audio_tagging_loss=0.008761, over 15472.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09146, pruned_loss=0.01401, audio_tagging_loss=0.009186, over 3056267.27 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:50:00,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2240613.3333333335, ans=0.0
2023-11-23 04:50:07,489 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336100
2023-11-23 04:50:26,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2240746.6666666665, ans=10.0
2023-11-23 04:50:38,533 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:51:02,787 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11500, loss[loss=0.0895, simple_loss=0.1168, pruned_loss=0.02315, audio_tagging_loss=0.007956, over 16373.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09116, pruned_loss=0.01407, audio_tagging_loss=0.009159, over 3050140.95 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:51:06,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2240946.6666666665, ans=0.1
2023-11-23 04:51:08,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2240946.6666666665, ans=0.125
2023-11-23 04:51:11,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336150
2023-11-23 04:51:26,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.03 vs. limit=6.0
2023-11-23 04:51:28,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2241080.0, ans=0.125
2023-11-23 04:51:32,805 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.33 vs. limit=22.5
2023-11-23 04:51:40,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2241146.6666666665, ans=0.1
2023-11-23 04:51:56,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2241213.3333333335, ans=0.0
2023-11-23 04:52:04,948 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.510e+01 8.416e+01 8.974e+01 9.531e+01 1.779e+02, threshold=1.795e+02, percent-clipped=1.0
2023-11-23 04:52:08,721 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11550, loss[loss=0.06393, simple_loss=0.08844, pruned_loss=0.01106, audio_tagging_loss=0.008649, over 14891.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09133, pruned_loss=0.01402, audio_tagging_loss=0.00915, over 3052009.32 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:52:16,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336200
2023-11-23 04:52:19,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2241280.0, ans=0.125
2023-11-23 04:52:30,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2241346.6666666665, ans=0.0
2023-11-23 04:52:33,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2241413.3333333335, ans=0.125
2023-11-23 04:52:35,092 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:52:37,797 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.52 vs. limit=15.0
2023-11-23 04:52:44,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2241413.3333333335, ans=0.125
2023-11-23 04:52:46,496 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:52:47,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2241480.0, ans=0.0
2023-11-23 04:53:11,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2241613.3333333335, ans=0.0
2023-11-23 04:53:11,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2241613.3333333335, ans=0.125
2023-11-23 04:53:12,010 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11600, loss[loss=0.06199, simple_loss=0.07528, pruned_loss=0.01096, audio_tagging_loss=0.01339, over 15839.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.0907, pruned_loss=0.0139, audio_tagging_loss=0.00922, over 3056846.47 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:53:19,409 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336250
2023-11-23 04:53:20,869 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:53:26,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2241680.0, ans=0.0
2023-11-23 04:53:49,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2241746.6666666665, ans=0.125
2023-11-23 04:54:12,062 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.276e+01 8.228e+01 9.009e+01 9.457e+01 1.103e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 04:54:12,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2241880.0, ans=0.2
2023-11-23 04:54:15,862 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11650, loss[loss=0.06635, simple_loss=0.08435, pruned_loss=0.01418, audio_tagging_loss=0.009997, over 14863.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09137, pruned_loss=0.014, audio_tagging_loss=0.009149, over 3050880.30 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:54:20,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2241946.6666666665, ans=0.0
2023-11-23 04:54:23,858 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336300
2023-11-23 04:54:39,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2242013.3333333335, ans=0.0
2023-11-23 04:54:40,735 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.28 vs. limit=22.5
2023-11-23 04:54:45,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2242080.0, ans=0.2
2023-11-23 04:55:01,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2242146.6666666665, ans=0.025
2023-11-23 04:55:21,576 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11700, loss[loss=0.06171, simple_loss=0.08405, pruned_loss=0.01088, audio_tagging_loss=0.008797, over 14924.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09114, pruned_loss=0.01401, audio_tagging_loss=0.009153, over 3047857.79 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:55:29,458 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336350
2023-11-23 04:55:30,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2242280.0, ans=0.2
2023-11-23 04:55:31,198 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.91 vs. limit=10.0
2023-11-23 04:55:40,613 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:55:45,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2242413.3333333335, ans=0.0
2023-11-23 04:55:57,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2242480.0, ans=0.1
2023-11-23 04:56:21,989 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.295e+01 8.847e+01 9.516e+01 1.261e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 04:56:25,697 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11750, loss[loss=0.07236, simple_loss=0.1023, pruned_loss=0.01348, audio_tagging_loss=0.007704, over 15905.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09177, pruned_loss=0.01399, audio_tagging_loss=0.00919, over 3047228.30 frames. ], batch size: 61, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:56:33,237 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336400
2023-11-23 04:56:33,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2242613.3333333335, ans=0.125
2023-11-23 04:57:29,436 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11800, loss[loss=0.07412, simple_loss=0.09246, pruned_loss=0.01647, audio_tagging_loss=0.01142, over 14897.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09232, pruned_loss=0.01415, audio_tagging_loss=0.009183, over 3045288.90 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:57:36,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.56 vs. limit=10.0
2023-11-23 04:57:36,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336450
2023-11-23 04:57:37,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2242946.6666666665, ans=0.2
2023-11-23 04:57:38,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2242946.6666666665, ans=0.1
2023-11-23 04:58:00,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2243080.0, ans=0.0
2023-11-23 04:58:04,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2243080.0, ans=0.125
2023-11-23 04:58:20,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2243213.3333333335, ans=0.2
2023-11-23 04:58:21,368 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.54 vs. limit=12.0
2023-11-23 04:58:24,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.27 vs. limit=15.0
2023-11-23 04:58:25,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2243213.3333333335, ans=0.2
2023-11-23 04:58:26,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.59 vs. limit=6.0
2023-11-23 04:58:31,388 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.315e+01 8.837e+01 9.445e+01 1.101e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 04:58:33,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.15 vs. limit=22.5
2023-11-23 04:58:33,798 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11850, loss[loss=0.09031, simple_loss=0.115, pruned_loss=0.0209, audio_tagging_loss=0.0119, over 15953.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09274, pruned_loss=0.01419, audio_tagging_loss=0.009244, over 3046072.12 frames. ], batch size: 60, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:58:39,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.49 vs. limit=15.0
2023-11-23 04:58:41,609 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336500
2023-11-23 04:58:45,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2243346.6666666665, ans=0.04949747468305833
2023-11-23 04:58:45,817 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-23 04:59:20,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2243480.0, ans=0.07
2023-11-23 04:59:26,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2243546.6666666665, ans=0.125
2023-11-23 04:59:35,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2243546.6666666665, ans=0.5
2023-11-23 04:59:38,846 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11900, loss[loss=0.08303, simple_loss=0.1139, pruned_loss=0.01637, audio_tagging_loss=0.009689, over 15695.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09293, pruned_loss=0.01415, audio_tagging_loss=0.009316, over 3049946.29 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:59:46,185 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336550
2023-11-23 04:59:46,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2243613.3333333335, ans=0.025
2023-11-23 04:59:50,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2243680.0, ans=0.125
2023-11-23 05:00:03,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2243746.6666666665, ans=0.0
2023-11-23 05:00:03,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2243746.6666666665, ans=0.1
2023-11-23 05:00:07,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2243746.6666666665, ans=0.0
2023-11-23 05:00:12,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.13 vs. limit=15.0
2023-11-23 05:00:17,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2243813.3333333335, ans=0.0
2023-11-23 05:00:20,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2243813.3333333335, ans=0.125
2023-11-23 05:00:24,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2243813.3333333335, ans=0.2
2023-11-23 05:00:39,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2243880.0, ans=0.07
2023-11-23 05:00:40,668 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.380e+01 8.877e+01 9.570e+01 1.262e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 05:00:41,899 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 11950, loss[loss=0.08516, simple_loss=0.1094, pruned_loss=0.02189, audio_tagging_loss=0.008549, over 14550.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09221, pruned_loss=0.01415, audio_tagging_loss=0.009426, over 3038634.80 frames. ], batch size: 52, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 05:00:44,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2243946.6666666665, ans=0.125
2023-11-23 05:00:49,030 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336600
2023-11-23 05:01:05,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2244013.3333333335, ans=0.0
2023-11-23 05:01:14,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2244080.0, ans=0.125
2023-11-23 05:01:19,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2244146.6666666665, ans=0.125
2023-11-23 05:01:25,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2244146.6666666665, ans=0.05
2023-11-23 05:01:26,764 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.49 vs. limit=15.0
2023-11-23 05:01:43,492 INFO [train_asr.py:1221] (2/4) Epoch 28, batch 12000, loss[loss=0.05198, simple_loss=0.05748, pruned_loss=0.01211, audio_tagging_loss=0.01112, over 13696.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.0927, pruned_loss=0.01433, audio_tagging_loss=0.009441, over 3038624.78 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 05:01:43,493 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 05:02:21,738 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2516, 4.2521, 4.4718, 4.4411], device='cuda:2')
2023-11-23 05:02:27,089 INFO [train_asr.py:1253] (2/4) Epoch 28, validation: loss=0.05897, simple_loss=0.05124, pruned_loss=0.005128, audio_tagging_loss=0.02822, over 4681554.00 frames. 
2023-11-23 05:02:27,090 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 05:02:34,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336650
2023-11-23 05:02:35,851 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:03:30,821 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 0, loss[loss=0.07567, simple_loss=0.08122, pruned_loss=0.01075, audio_tagging_loss=0.02431, over 14188.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.08122, pruned_loss=0.01075, audio_tagging_loss=0.02431, over 14188.00 frames. ], batch size: 56, lr: 2.37e-03, grad_scale: 32.0
2023-11-23 05:03:30,822 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 05:04:08,421 INFO [train_asr.py:1253] (2/4) Epoch 29, validation: loss=0.05816, simple_loss=0.05122, pruned_loss=0.005095, audio_tagging_loss=0.02745, over 4681554.00 frames. 
2023-11-23 05:04:08,422 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 05:04:18,767 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=15.0
2023-11-23 05:04:28,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2244506.6666666665, ans=0.125
2023-11-23 05:04:35,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.24 vs. limit=15.0
2023-11-23 05:04:40,852 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 8.521e+01 9.209e+01 1.026e+02 2.736e+02, threshold=1.842e+02, percent-clipped=1.0
2023-11-23 05:04:49,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336700
2023-11-23 05:05:02,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2244706.6666666665, ans=0.2
2023-11-23 05:05:12,063 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 50, loss[loss=0.07883, simple_loss=0.09812, pruned_loss=0.01286, audio_tagging_loss=0.0169, over 15653.00 frames. ], tot_loss[loss=0.08043, simple_loss=0.09702, pruned_loss=0.01431, audio_tagging_loss=0.01761, over 695138.14 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:05:20,939 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:05:40,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.62 vs. limit=15.0
2023-11-23 05:05:44,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2244906.6666666665, ans=0.0
2023-11-23 05:05:51,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2244973.3333333335, ans=0.1
2023-11-23 05:05:54,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336750
2023-11-23 05:05:56,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2244973.3333333335, ans=0.0
2023-11-23 05:06:18,182 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 100, loss[loss=0.07195, simple_loss=0.08237, pruned_loss=0.01385, audio_tagging_loss=0.01692, over 15182.00 frames. ], tot_loss[loss=0.07864, simple_loss=0.09434, pruned_loss=0.01418, audio_tagging_loss=0.01729, over 1217460.86 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:06:26,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2245106.6666666665, ans=0.1
2023-11-23 05:06:32,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.38 vs. limit=15.0
2023-11-23 05:06:38,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2245173.3333333335, ans=0.1
2023-11-23 05:06:44,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2245240.0, ans=0.0
2023-11-23 05:06:49,664 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.633e+01 8.985e+01 9.652e+01 1.024e+02 1.304e+02, threshold=1.930e+02, percent-clipped=0.0
2023-11-23 05:06:52,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2245240.0, ans=0.125
2023-11-23 05:06:52,825 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.42 vs. limit=12.0
2023-11-23 05:06:59,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336800
2023-11-23 05:07:04,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2245306.6666666665, ans=0.125
2023-11-23 05:07:22,218 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 150, loss[loss=0.06609, simple_loss=0.07904, pruned_loss=0.01493, audio_tagging_loss=0.01163, over 14723.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09285, pruned_loss=0.01417, audio_tagging_loss=0.01558, over 1631921.40 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:07:28,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.05 vs. limit=15.0
2023-11-23 05:07:30,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2245440.0, ans=0.125
2023-11-23 05:07:53,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2245573.3333333335, ans=0.125
2023-11-23 05:07:59,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2245573.3333333335, ans=0.125
2023-11-23 05:08:04,385 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336850
2023-11-23 05:08:20,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2245706.6666666665, ans=0.0
2023-11-23 05:08:21,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2245706.6666666665, ans=0.125
2023-11-23 05:08:27,341 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 200, loss[loss=0.07013, simple_loss=0.08875, pruned_loss=0.01517, audio_tagging_loss=0.01058, over 14051.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09323, pruned_loss=0.01432, audio_tagging_loss=0.01361, over 1938078.59 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:08:56,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2245906.6666666665, ans=0.0
2023-11-23 05:08:56,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.56 vs. limit=15.0
2023-11-23 05:09:00,515 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.383e+01 9.139e+01 9.827e+01 1.313e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 05:09:08,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336900
2023-11-23 05:09:17,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2246040.0, ans=0.1
2023-11-23 05:09:23,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2246040.0, ans=0.0
2023-11-23 05:09:32,722 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 250, loss[loss=0.05736, simple_loss=0.08088, pruned_loss=0.01063, audio_tagging_loss=0.006289, over 14835.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09268, pruned_loss=0.01403, audio_tagging_loss=0.01215, over 2185057.29 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:09:36,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2246106.6666666665, ans=0.125
2023-11-23 05:10:01,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2246240.0, ans=0.0
2023-11-23 05:10:04,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.37 vs. limit=22.5
2023-11-23 05:10:12,450 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 336950
2023-11-23 05:10:31,483 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:10:36,206 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 300, loss[loss=0.07117, simple_loss=0.0924, pruned_loss=0.01535, audio_tagging_loss=0.009615, over 16284.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.0938, pruned_loss=0.01432, audio_tagging_loss=0.01133, over 2376668.47 frames. ], batch size: 61, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:11:10,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.48 vs. limit=15.0
2023-11-23 05:11:10,504 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.241e+01 8.912e+01 9.854e+01 1.344e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 05:11:16,169 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.74 vs. limit=15.0
2023-11-23 05:11:17,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337000
2023-11-23 05:11:34,853 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.01 vs. limit=15.0
2023-11-23 05:11:35,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2246706.6666666665, ans=0.1
2023-11-23 05:11:40,973 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 350, loss[loss=0.09338, simple_loss=0.1291, pruned_loss=0.0202, audio_tagging_loss=0.008622, over 15073.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09374, pruned_loss=0.01419, audio_tagging_loss=0.0107, over 2529487.42 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:11:45,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2246773.3333333335, ans=0.0
2023-11-23 05:11:48,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2246773.3333333335, ans=0.125
2023-11-23 05:11:54,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2246840.0, ans=0.1
2023-11-23 05:12:13,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2246906.6666666665, ans=0.125
2023-11-23 05:12:13,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2246906.6666666665, ans=0.1
2023-11-23 05:12:22,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337050
2023-11-23 05:12:23,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2246973.3333333335, ans=0.125
2023-11-23 05:12:28,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2246973.3333333335, ans=0.125
2023-11-23 05:12:42,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2247040.0, ans=0.0
2023-11-23 05:12:46,371 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 400, loss[loss=0.07441, simple_loss=0.08969, pruned_loss=0.01986, audio_tagging_loss=0.0097, over 15453.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09404, pruned_loss=0.01449, audio_tagging_loss=0.01024, over 2648080.85 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:12:50,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.15 vs. limit=6.0
2023-11-23 05:13:09,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2247240.0, ans=0.1
2023-11-23 05:13:18,401 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.176e+01 8.720e+01 9.339e+01 1.453e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-23 05:13:26,469 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337100
2023-11-23 05:13:50,276 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 450, loss[loss=0.05554, simple_loss=0.07538, pruned_loss=0.01023, audio_tagging_loss=0.007624, over 14606.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09264, pruned_loss=0.01422, audio_tagging_loss=0.01006, over 2734113.92 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:13:58,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2247440.0, ans=0.2
2023-11-23 05:14:25,871 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.29 vs. limit=12.0
2023-11-23 05:14:31,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337150
2023-11-23 05:14:31,607 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:14:46,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2247706.6666666665, ans=0.125
2023-11-23 05:14:52,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2247773.3333333335, ans=0.125
2023-11-23 05:14:53,605 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 500, loss[loss=0.07082, simple_loss=0.09212, pruned_loss=0.01439, audio_tagging_loss=0.01036, over 15155.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09248, pruned_loss=0.01411, audio_tagging_loss=0.009806, over 2804798.41 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:14:59,092 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.20 vs. limit=15.0
2023-11-23 05:15:04,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2247773.3333333335, ans=0.0
2023-11-23 05:15:06,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2247840.0, ans=0.125
2023-11-23 05:15:21,824 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.27 vs. limit=22.5
2023-11-23 05:15:26,959 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.484e+01 9.126e+01 9.707e+01 1.686e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 05:15:28,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2247906.6666666665, ans=0.0
2023-11-23 05:15:34,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337200
2023-11-23 05:15:38,594 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:15:45,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.40 vs. limit=22.5
2023-11-23 05:15:47,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2248040.0, ans=0.125
2023-11-23 05:15:55,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2248040.0, ans=0.0
2023-11-23 05:15:57,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2248106.6666666665, ans=0.125
2023-11-23 05:15:57,888 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 550, loss[loss=0.06669, simple_loss=0.09158, pruned_loss=0.01291, audio_tagging_loss=0.007986, over 15327.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09301, pruned_loss=0.01432, audio_tagging_loss=0.009658, over 2854332.26 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:16:20,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2248173.3333333335, ans=0.0
2023-11-23 05:16:27,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2248240.0, ans=0.125
2023-11-23 05:16:38,172 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337250
2023-11-23 05:16:53,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2248373.3333333335, ans=0.07
2023-11-23 05:17:01,985 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 600, loss[loss=0.07183, simple_loss=0.0941, pruned_loss=0.01422, audio_tagging_loss=0.01056, over 14004.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09302, pruned_loss=0.01429, audio_tagging_loss=0.009556, over 2904645.28 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:17:11,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2248440.0, ans=0.0
2023-11-23 05:17:34,613 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.545e+01 8.479e+01 8.808e+01 9.540e+01 1.178e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 05:17:42,796 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337300
2023-11-23 05:17:43,027 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:17:47,084 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.10 vs. limit=15.0
2023-11-23 05:18:05,043 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 650, loss[loss=0.07208, simple_loss=0.09179, pruned_loss=0.01596, audio_tagging_loss=0.01022, over 14038.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09246, pruned_loss=0.01426, audio_tagging_loss=0.009536, over 2932334.46 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:18:14,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2248773.3333333335, ans=0.07
2023-11-23 05:18:19,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2248840.0, ans=0.2
2023-11-23 05:18:27,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2248840.0, ans=10.0
2023-11-23 05:18:46,982 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337350
2023-11-23 05:18:50,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2248973.3333333335, ans=0.0
2023-11-23 05:19:09,380 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 700, loss[loss=0.06803, simple_loss=0.09001, pruned_loss=0.01325, audio_tagging_loss=0.009775, over 13215.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09326, pruned_loss=0.01427, audio_tagging_loss=0.009424, over 2962045.63 frames. ], batch size: 51, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:19:44,723 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.145e+01 8.167e+01 8.621e+01 9.555e+01 1.511e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-23 05:19:49,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2249306.6666666665, ans=0.0
2023-11-23 05:19:51,200 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337400
2023-11-23 05:19:57,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2249306.6666666665, ans=0.125
2023-11-23 05:20:01,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2249373.3333333335, ans=0.125
2023-11-23 05:20:15,759 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 750, loss[loss=0.09421, simple_loss=0.1238, pruned_loss=0.02201, audio_tagging_loss=0.01029, over 15479.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09432, pruned_loss=0.01444, audio_tagging_loss=0.009422, over 2988463.04 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:20:24,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2249440.0, ans=0.125
2023-11-23 05:20:52,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2249573.3333333335, ans=0.2
2023-11-23 05:20:53,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=15.0
2023-11-23 05:20:57,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337450
2023-11-23 05:21:01,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2249640.0, ans=0.125
2023-11-23 05:21:09,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2249706.6666666665, ans=0.125
2023-11-23 05:21:16,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2249706.6666666665, ans=0.0
2023-11-23 05:21:19,962 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 800, loss[loss=0.0822, simple_loss=0.1191, pruned_loss=0.01629, audio_tagging_loss=0.006376, over 16239.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09411, pruned_loss=0.01447, audio_tagging_loss=0.009291, over 3002455.38 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:21:26,850 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.78 vs. limit=15.0
2023-11-23 05:21:39,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2249840.0, ans=0.95
2023-11-23 05:21:53,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2249906.6666666665, ans=0.125
2023-11-23 05:21:53,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.30 vs. limit=22.5
2023-11-23 05:21:55,387 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.531e+01 8.991e+01 9.709e+01 1.279e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 05:22:01,738 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337500
2023-11-23 05:22:24,138 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 850, loss[loss=0.07016, simple_loss=0.09063, pruned_loss=0.01313, audio_tagging_loss=0.01172, over 15610.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09369, pruned_loss=0.0144, audio_tagging_loss=0.009343, over 3021388.08 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:22:45,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2250173.3333333335, ans=0.5
2023-11-23 05:22:53,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2250240.0, ans=0.125
2023-11-23 05:23:00,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.10 vs. limit=15.0
2023-11-23 05:23:06,274 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337550
2023-11-23 05:23:30,307 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 900, loss[loss=0.06536, simple_loss=0.0851, pruned_loss=0.01366, audio_tagging_loss=0.009154, over 16480.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09341, pruned_loss=0.01427, audio_tagging_loss=0.009282, over 3030027.20 frames. ], batch size: 63, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:23:41,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2250506.6666666665, ans=0.0
2023-11-23 05:24:03,821 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.195e+01 8.718e+01 9.591e+01 1.345e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-23 05:24:04,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2250573.3333333335, ans=0.125
2023-11-23 05:24:11,524 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337600
2023-11-23 05:24:34,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2250773.3333333335, ans=0.0
2023-11-23 05:24:35,017 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 950, loss[loss=0.05154, simple_loss=0.06314, pruned_loss=0.008835, audio_tagging_loss=0.01113, over 15197.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09432, pruned_loss=0.01458, audio_tagging_loss=0.00918, over 3033648.29 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:24:39,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2250773.3333333335, ans=0.1
2023-11-23 05:24:43,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2250773.3333333335, ans=0.1
2023-11-23 05:24:46,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2250840.0, ans=0.125
2023-11-23 05:25:03,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2250906.6666666665, ans=0.0
2023-11-23 05:25:17,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337650
2023-11-23 05:25:39,742 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1000, loss[loss=0.07278, simple_loss=0.09554, pruned_loss=0.01491, audio_tagging_loss=0.0101, over 15523.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09422, pruned_loss=0.01467, audio_tagging_loss=0.00905, over 3036513.59 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:26:07,693 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:26:15,018 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.282e+01 9.010e+01 9.966e+01 1.225e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 05:26:18,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2251306.6666666665, ans=0.0
2023-11-23 05:26:21,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337700
2023-11-23 05:26:24,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2251306.6666666665, ans=0.05
2023-11-23 05:26:44,932 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1050, loss[loss=0.08358, simple_loss=0.1107, pruned_loss=0.01973, audio_tagging_loss=0.008477, over 14803.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09369, pruned_loss=0.01448, audio_tagging_loss=0.009058, over 3034064.53 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:26:50,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2251440.0, ans=0.125
2023-11-23 05:26:52,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2251440.0, ans=0.0
2023-11-23 05:27:02,767 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.24 vs. limit=15.0
2023-11-23 05:27:10,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2251573.3333333335, ans=0.125
2023-11-23 05:27:26,399 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337750
2023-11-23 05:27:34,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2251640.0, ans=0.125
2023-11-23 05:27:47,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2251706.6666666665, ans=0.0
2023-11-23 05:27:50,133 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1100, loss[loss=0.05598, simple_loss=0.0813, pruned_loss=0.006913, audio_tagging_loss=0.008418, over 14918.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09356, pruned_loss=0.01458, audio_tagging_loss=0.009048, over 3032516.74 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:27:50,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2251773.3333333335, ans=0.125
2023-11-23 05:27:50,856 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.76 vs. limit=22.5
2023-11-23 05:27:51,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2251773.3333333335, ans=0.04949747468305833
2023-11-23 05:27:52,673 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:27:56,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2251773.3333333335, ans=0.0
2023-11-23 05:28:25,461 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.248e+01 8.961e+01 9.560e+01 1.246e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 05:28:31,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337800
2023-11-23 05:28:41,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.28 vs. limit=12.0
2023-11-23 05:28:54,904 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1150, loss[loss=0.07087, simple_loss=0.1008, pruned_loss=0.01502, audio_tagging_loss=0.005482, over 15379.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09339, pruned_loss=0.01451, audio_tagging_loss=0.008968, over 3039370.61 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:29:12,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2252173.3333333335, ans=0.0
2023-11-23 05:29:29,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2252240.0, ans=0.125
2023-11-23 05:29:29,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2252240.0, ans=0.0
2023-11-23 05:29:35,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2252306.6666666665, ans=0.125
2023-11-23 05:29:36,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337850
2023-11-23 05:29:44,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2252306.6666666665, ans=0.0
2023-11-23 05:29:47,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff2.min_abs, batch_count=2252373.3333333335, ans=0.1
2023-11-23 05:29:55,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.94 vs. limit=22.5
2023-11-23 05:30:00,345 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1200, loss[loss=0.06841, simple_loss=0.09171, pruned_loss=0.01407, audio_tagging_loss=0.00848, over 15793.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09291, pruned_loss=0.01448, audio_tagging_loss=0.009, over 3036266.55 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:30:14,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2252506.6666666665, ans=0.07
2023-11-23 05:30:18,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2252506.6666666665, ans=0.125
2023-11-23 05:30:20,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2252506.6666666665, ans=0.125
2023-11-23 05:30:35,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.804e+01 8.344e+01 9.032e+01 9.683e+01 1.496e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 05:30:40,247 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337900
2023-11-23 05:31:04,488 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1250, loss[loss=0.07476, simple_loss=0.1033, pruned_loss=0.01634, audio_tagging_loss=0.006753, over 16132.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09298, pruned_loss=0.01428, audio_tagging_loss=0.009011, over 3034310.79 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:31:07,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2252773.3333333335, ans=0.0
2023-11-23 05:31:38,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2252906.6666666665, ans=0.1
2023-11-23 05:31:45,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 337950
2023-11-23 05:32:07,621 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1300, loss[loss=0.08582, simple_loss=0.1008, pruned_loss=0.02315, audio_tagging_loss=0.01225, over 15753.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09341, pruned_loss=0.01447, audio_tagging_loss=0.008987, over 3035281.54 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:32:31,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2253173.3333333335, ans=0.0
2023-11-23 05:32:44,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.370e+01 8.150e+01 8.858e+01 9.270e+01 1.252e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 05:32:46,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2253306.6666666665, ans=0.05
2023-11-23 05:32:49,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338000
2023-11-23 05:33:13,803 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1350, loss[loss=0.06331, simple_loss=0.08184, pruned_loss=0.01219, audio_tagging_loss=0.0102, over 14497.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09245, pruned_loss=0.01432, audio_tagging_loss=0.009096, over 3031787.91 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:33:14,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2253440.0, ans=0.125
2023-11-23 05:33:21,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2253440.0, ans=0.0
2023-11-23 05:33:39,460 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0
2023-11-23 05:33:41,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2253573.3333333335, ans=0.125
2023-11-23 05:33:53,288 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338050
2023-11-23 05:33:59,449 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:34:17,318 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1400, loss[loss=0.05112, simple_loss=0.06387, pruned_loss=0.008583, audio_tagging_loss=0.0106, over 14904.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09215, pruned_loss=0.01415, audio_tagging_loss=0.009151, over 3035784.89 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:34:30,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2253840.0, ans=0.125
2023-11-23 05:34:53,591 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.459e+01 9.010e+01 9.793e+01 1.707e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 05:34:58,595 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338100
2023-11-23 05:35:13,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2254040.0, ans=0.0
2023-11-23 05:35:14,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2254040.0, ans=0.0
2023-11-23 05:35:21,276 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1450, loss[loss=0.06453, simple_loss=0.08563, pruned_loss=0.01179, audio_tagging_loss=0.009924, over 16133.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09272, pruned_loss=0.01409, audio_tagging_loss=0.009173, over 3043849.76 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:35:23,976 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:35:48,818 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0
2023-11-23 05:35:55,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2254240.0, ans=0.125
2023-11-23 05:36:01,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338150
2023-11-23 05:36:24,831 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1500, loss[loss=0.06432, simple_loss=0.08272, pruned_loss=0.01122, audio_tagging_loss=0.01174, over 14945.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09328, pruned_loss=0.01426, audio_tagging_loss=0.009197, over 3045729.96 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:36:37,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2254506.6666666665, ans=0.04949747468305833
2023-11-23 05:36:55,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.78 vs. limit=6.0
2023-11-23 05:37:00,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.437e+01 9.139e+01 9.780e+01 1.863e+02, threshold=1.828e+02, percent-clipped=1.0
2023-11-23 05:37:05,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338200
2023-11-23 05:37:25,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2254706.6666666665, ans=0.125
2023-11-23 05:37:29,284 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1550, loss[loss=0.05319, simple_loss=0.0638, pruned_loss=0.01104, audio_tagging_loss=0.01025, over 13743.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09278, pruned_loss=0.0142, audio_tagging_loss=0.00931, over 3041200.44 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:37:29,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2254773.3333333335, ans=0.05
2023-11-23 05:37:33,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=15.02 vs. limit=15.0
2023-11-23 05:37:39,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2254773.3333333335, ans=0.0
2023-11-23 05:37:48,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2254840.0, ans=0.07
2023-11-23 05:38:06,349 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.43 vs. limit=15.0
2023-11-23 05:38:08,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2254973.3333333335, ans=0.1
2023-11-23 05:38:10,757 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338250
2023-11-23 05:38:15,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.70 vs. limit=15.0
2023-11-23 05:38:16,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.39 vs. limit=22.5
2023-11-23 05:38:16,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.33 vs. limit=15.0
2023-11-23 05:38:28,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.91 vs. limit=15.0
2023-11-23 05:38:32,705 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1600, loss[loss=0.06018, simple_loss=0.07719, pruned_loss=0.01145, audio_tagging_loss=0.01014, over 15105.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09311, pruned_loss=0.01427, audio_tagging_loss=0.009356, over 3040307.95 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:38:37,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2255106.6666666665, ans=0.125
2023-11-23 05:38:38,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.68 vs. limit=15.0
2023-11-23 05:38:42,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2255106.6666666665, ans=10.0
2023-11-23 05:39:05,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.01 vs. limit=10.0
2023-11-23 05:39:09,183 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 8.320e+01 8.917e+01 9.601e+01 1.213e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 05:39:13,020 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338300
2023-11-23 05:39:35,856 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1650, loss[loss=0.0675, simple_loss=0.08674, pruned_loss=0.0144, audio_tagging_loss=0.009729, over 14955.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09247, pruned_loss=0.01413, audio_tagging_loss=0.009392, over 3036475.77 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:39:37,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2255440.0, ans=0.125
2023-11-23 05:39:47,042 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.39 vs. limit=15.0
2023-11-23 05:39:52,607 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:39:56,416 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.09 vs. limit=6.0
2023-11-23 05:39:58,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2255506.6666666665, ans=0.0
2023-11-23 05:40:06,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2255573.3333333335, ans=0.0
2023-11-23 05:40:15,727 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338350
2023-11-23 05:40:16,476 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=15.0
2023-11-23 05:40:38,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2255773.3333333335, ans=0.2
2023-11-23 05:40:39,289 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1700, loss[loss=0.09205, simple_loss=0.1311, pruned_loss=0.01751, audio_tagging_loss=0.009011, over 16392.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.0934, pruned_loss=0.01427, audio_tagging_loss=0.009395, over 3042584.70 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:40:44,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2255773.3333333335, ans=0.125
2023-11-23 05:40:49,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2255773.3333333335, ans=0.125
2023-11-23 05:40:53,365 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.87 vs. limit=22.5
2023-11-23 05:41:16,148 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.211e+01 9.004e+01 9.824e+01 1.268e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 05:41:19,984 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338400
2023-11-23 05:41:23,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.91 vs. limit=15.0
2023-11-23 05:41:27,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2255973.3333333335, ans=0.125
2023-11-23 05:41:32,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_na.min_abs, batch_count=2256040.0, ans=0.02
2023-11-23 05:41:42,301 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1750, loss[loss=0.06067, simple_loss=0.0852, pruned_loss=0.008038, audio_tagging_loss=0.01003, over 16438.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09322, pruned_loss=0.01418, audio_tagging_loss=0.009419, over 3048628.39 frames. ], batch size: 61, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:41:47,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2256106.6666666665, ans=0.125
2023-11-23 05:41:54,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2256173.3333333335, ans=0.125
2023-11-23 05:41:59,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2256173.3333333335, ans=0.0
2023-11-23 05:42:22,890 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338450
2023-11-23 05:42:35,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2256373.3333333335, ans=0.1
2023-11-23 05:42:45,362 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1800, loss[loss=0.07897, simple_loss=0.1039, pruned_loss=0.01883, audio_tagging_loss=0.008183, over 15087.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.0932, pruned_loss=0.01431, audio_tagging_loss=0.009378, over 3044086.95 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:42:47,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2256440.0, ans=0.0
2023-11-23 05:42:54,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2256440.0, ans=0.2
2023-11-23 05:43:04,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2256506.6666666665, ans=0.0
2023-11-23 05:43:06,159 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.72 vs. limit=10.0
2023-11-23 05:43:18,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2256573.3333333335, ans=0.0
2023-11-23 05:43:21,549 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.187e+01 9.056e+01 9.555e+01 1.230e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 05:43:25,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338500
2023-11-23 05:43:41,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2256706.6666666665, ans=0.125
2023-11-23 05:43:42,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2256706.6666666665, ans=0.125
2023-11-23 05:43:48,538 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1850, loss[loss=0.05375, simple_loss=0.06386, pruned_loss=0.008325, audio_tagging_loss=0.01349, over 16001.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09318, pruned_loss=0.01418, audio_tagging_loss=0.009262, over 3047141.80 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:44:29,138 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338550
2023-11-23 05:44:30,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2256973.3333333335, ans=0.04949747468305833
2023-11-23 05:44:51,025 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1900, loss[loss=0.07203, simple_loss=0.1026, pruned_loss=0.01408, audio_tagging_loss=0.00667, over 15136.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09271, pruned_loss=0.01412, audio_tagging_loss=0.009181, over 3046829.26 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:44:58,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2257106.6666666665, ans=0.0
2023-11-23 05:45:28,227 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.552e+01 8.593e+01 9.216e+01 1.001e+02 1.158e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-23 05:45:29,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2257306.6666666665, ans=0.125
2023-11-23 05:45:32,192 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338600
2023-11-23 05:45:35,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2257306.6666666665, ans=0.0
2023-11-23 05:45:54,438 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 1950, loss[loss=0.06805, simple_loss=0.09599, pruned_loss=0.01188, audio_tagging_loss=0.008181, over 14926.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09273, pruned_loss=0.0141, audio_tagging_loss=0.00915, over 3051623.66 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:46:22,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2257573.3333333335, ans=0.125
2023-11-23 05:46:34,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338650
2023-11-23 05:46:49,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2257706.6666666665, ans=0.125
2023-11-23 05:46:55,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2257706.6666666665, ans=0.125
2023-11-23 05:46:57,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2257773.3333333335, ans=0.1
2023-11-23 05:46:57,986 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2000, loss[loss=0.06579, simple_loss=0.08939, pruned_loss=0.01102, audio_tagging_loss=0.01007, over 14583.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09307, pruned_loss=0.01416, audio_tagging_loss=0.00913, over 3046597.70 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:47:01,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2257773.3333333335, ans=0.0
2023-11-23 05:47:04,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2257773.3333333335, ans=0.125
2023-11-23 05:47:28,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2257906.6666666665, ans=0.2
2023-11-23 05:47:33,483 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.322e+01 9.117e+01 1.012e+02 1.277e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-23 05:47:37,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338700
2023-11-23 05:47:45,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2257973.3333333335, ans=0.1
2023-11-23 05:47:45,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.38 vs. limit=15.0
2023-11-23 05:47:58,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2258040.0, ans=0.07
2023-11-23 05:47:59,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2258106.6666666665, ans=0.0
2023-11-23 05:48:00,700 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2050, loss[loss=0.07494, simple_loss=0.1062, pruned_loss=0.01585, audio_tagging_loss=0.006011, over 15361.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09318, pruned_loss=0.01426, audio_tagging_loss=0.009109, over 3046160.87 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:48:06,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2258106.6666666665, ans=0.1
2023-11-23 05:48:17,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2258173.3333333335, ans=0.0
2023-11-23 05:48:18,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2258173.3333333335, ans=0.2
2023-11-23 05:48:21,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2258173.3333333335, ans=0.1
2023-11-23 05:48:25,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2258240.0, ans=0.0
2023-11-23 05:48:38,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.08 vs. limit=15.0
2023-11-23 05:48:41,299 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338750
2023-11-23 05:48:41,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2258306.6666666665, ans=0.125
2023-11-23 05:48:50,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2258373.3333333335, ans=0.125
2023-11-23 05:48:53,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2258373.3333333335, ans=0.125
2023-11-23 05:48:54,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2258373.3333333335, ans=0.09899494936611666
2023-11-23 05:49:03,152 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2100, loss[loss=0.07446, simple_loss=0.08942, pruned_loss=0.01711, audio_tagging_loss=0.01264, over 15004.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09275, pruned_loss=0.01428, audio_tagging_loss=0.009111, over 3040966.33 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:49:08,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2258440.0, ans=0.125
2023-11-23 05:49:31,411 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.39 vs. limit=12.0
2023-11-23 05:49:36,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2258573.3333333335, ans=0.1
2023-11-23 05:49:39,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2258573.3333333335, ans=0.2
2023-11-23 05:49:40,179 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.396e+01 9.168e+01 1.020e+02 1.328e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 05:49:40,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2258640.0, ans=0.95
2023-11-23 05:49:43,923 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338800
2023-11-23 05:49:44,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2258640.0, ans=0.2
2023-11-23 05:50:02,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2258706.6666666665, ans=0.0
2023-11-23 05:50:07,655 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2150, loss[loss=0.06088, simple_loss=0.08744, pruned_loss=0.01034, audio_tagging_loss=0.006818, over 16101.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09266, pruned_loss=0.01432, audio_tagging_loss=0.009066, over 3043123.02 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:50:36,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2258906.6666666665, ans=0.0
2023-11-23 05:50:45,284 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:50:46,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2258973.3333333335, ans=0.125
2023-11-23 05:50:47,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338850
2023-11-23 05:50:54,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2258973.3333333335, ans=0.0
2023-11-23 05:50:59,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2259040.0, ans=0.125
2023-11-23 05:51:06,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2259040.0, ans=0.2
2023-11-23 05:51:07,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2259040.0, ans=0.125
2023-11-23 05:51:11,694 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2200, loss[loss=0.07192, simple_loss=0.09667, pruned_loss=0.01503, audio_tagging_loss=0.008561, over 16202.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09333, pruned_loss=0.01441, audio_tagging_loss=0.009036, over 3039698.00 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:51:19,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2259106.6666666665, ans=0.125
2023-11-23 05:51:50,017 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.527e+01 9.201e+01 9.842e+01 1.279e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 05:51:53,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338900
2023-11-23 05:51:57,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2259306.6666666665, ans=0.125
2023-11-23 05:51:58,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2259306.6666666665, ans=0.125
2023-11-23 05:52:06,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2259373.3333333335, ans=0.125
2023-11-23 05:52:08,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.23 vs. limit=15.0
2023-11-23 05:52:08,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2259373.3333333335, ans=0.0
2023-11-23 05:52:14,300 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=7.12 vs. limit=10.0
2023-11-23 05:52:16,092 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2250, loss[loss=0.06451, simple_loss=0.0817, pruned_loss=0.01436, audio_tagging_loss=0.009301, over 15060.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.0926, pruned_loss=0.01417, audio_tagging_loss=0.009054, over 3043932.96 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:52:17,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2259440.0, ans=0.1
2023-11-23 05:52:32,888 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.28 vs. limit=10.0
2023-11-23 05:52:33,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2259506.6666666665, ans=0.0
2023-11-23 05:52:49,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2259573.3333333335, ans=0.2
2023-11-23 05:52:57,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2259640.0, ans=0.2
2023-11-23 05:52:58,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 338950
2023-11-23 05:52:59,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.56 vs. limit=8.0
2023-11-23 05:53:09,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2259706.6666666665, ans=0.125
2023-11-23 05:53:22,022 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2300, loss[loss=0.07171, simple_loss=0.09636, pruned_loss=0.01585, audio_tagging_loss=0.007673, over 15718.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09308, pruned_loss=0.01441, audio_tagging_loss=0.009042, over 3037598.64 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:53:38,746 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.02 vs. limit=22.5
2023-11-23 05:54:00,273 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.330e+01 8.807e+01 9.666e+01 1.881e+02, threshold=1.761e+02, percent-clipped=1.0
2023-11-23 05:54:02,931 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339000
2023-11-23 05:54:19,385 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:54:22,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2260040.0, ans=0.1
2023-11-23 05:54:27,448 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2350, loss[loss=0.06907, simple_loss=0.09739, pruned_loss=0.01178, audio_tagging_loss=0.0086, over 16147.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09284, pruned_loss=0.01429, audio_tagging_loss=0.009124, over 3051702.31 frames. ], batch size: 62, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:54:38,787 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:54:57,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2260240.0, ans=0.1
2023-11-23 05:55:04,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2260306.6666666665, ans=0.07
2023-11-23 05:55:08,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339050
2023-11-23 05:55:30,948 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2400, loss[loss=0.06616, simple_loss=0.0888, pruned_loss=0.01363, audio_tagging_loss=0.008136, over 15986.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09204, pruned_loss=0.01418, audio_tagging_loss=0.00934, over 3048460.93 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:56:07,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2260573.3333333335, ans=0.0
2023-11-23 05:56:07,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2260573.3333333335, ans=0.07
2023-11-23 05:56:09,719 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.287e+01 8.924e+01 9.833e+01 1.260e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 05:56:12,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339100
2023-11-23 05:56:23,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2260706.6666666665, ans=0.125
2023-11-23 05:56:28,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2260706.6666666665, ans=0.1
2023-11-23 05:56:34,912 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2450, loss[loss=0.08368, simple_loss=0.1197, pruned_loss=0.01621, audio_tagging_loss=0.007615, over 15295.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09241, pruned_loss=0.01425, audio_tagging_loss=0.009367, over 3049785.58 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:57:15,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339150
2023-11-23 05:57:30,400 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.01 vs. limit=12.0
2023-11-23 05:57:38,395 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2500, loss[loss=0.06818, simple_loss=0.09121, pruned_loss=0.01281, audio_tagging_loss=0.009761, over 15174.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09205, pruned_loss=0.01399, audio_tagging_loss=0.009406, over 3049499.94 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:58:01,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2261173.3333333335, ans=0.0
2023-11-23 05:58:18,163 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.151e+01 8.943e+01 9.911e+01 1.178e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 05:58:18,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2261306.6666666665, ans=0.2
2023-11-23 05:58:19,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339200
2023-11-23 05:58:23,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2261306.6666666665, ans=0.125
2023-11-23 05:58:28,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2261373.3333333335, ans=0.125
2023-11-23 05:58:40,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2261373.3333333335, ans=0.0
2023-11-23 05:58:42,698 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2550, loss[loss=0.06291, simple_loss=0.08389, pruned_loss=0.01295, audio_tagging_loss=0.008004, over 15913.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09204, pruned_loss=0.01396, audio_tagging_loss=0.009341, over 3053841.94 frames. ], batch size: 61, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:59:19,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2261640.0, ans=0.0
2023-11-23 05:59:23,154 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339250
2023-11-23 05:59:38,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2261706.6666666665, ans=0.1
2023-11-23 05:59:45,920 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2600, loss[loss=0.07184, simple_loss=0.09931, pruned_loss=0.01571, audio_tagging_loss=0.006478, over 13925.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09191, pruned_loss=0.01397, audio_tagging_loss=0.009266, over 3042991.92 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:59:52,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2261773.3333333335, ans=0.95
2023-11-23 06:00:21,776 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.07 vs. limit=22.5
2023-11-23 06:00:25,724 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.329e+01 8.953e+01 9.650e+01 1.435e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 06:00:27,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339300
2023-11-23 06:00:42,128 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:00:43,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.84 vs. limit=15.0
2023-11-23 06:00:50,468 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2650, loss[loss=0.07163, simple_loss=0.08747, pruned_loss=0.01834, audio_tagging_loss=0.009558, over 14365.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09186, pruned_loss=0.01408, audio_tagging_loss=0.009146, over 3041016.38 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:00:56,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2262106.6666666665, ans=0.2
2023-11-23 06:01:31,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339350
2023-11-23 06:01:33,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.03 vs. limit=15.0
2023-11-23 06:01:50,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2262373.3333333335, ans=0.1
2023-11-23 06:01:53,638 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2700, loss[loss=0.07435, simple_loss=0.09595, pruned_loss=0.0165, audio_tagging_loss=0.009876, over 14166.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09163, pruned_loss=0.01405, audio_tagging_loss=0.009166, over 3039241.66 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:02:02,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.10 vs. limit=22.5
2023-11-23 06:02:14,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2262506.6666666665, ans=0.125
2023-11-23 06:02:23,071 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.97 vs. limit=22.5
2023-11-23 06:02:28,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.76 vs. limit=15.0
2023-11-23 06:02:33,769 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.218e+01 8.946e+01 9.636e+01 1.335e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 06:02:35,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339400
2023-11-23 06:02:58,421 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2750, loss[loss=0.07425, simple_loss=0.0997, pruned_loss=0.01648, audio_tagging_loss=0.007913, over 15155.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09112, pruned_loss=0.01399, audio_tagging_loss=0.009159, over 3037204.11 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:03:20,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2262840.0, ans=0.2
2023-11-23 06:03:24,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2262906.6666666665, ans=0.1
2023-11-23 06:03:39,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339450
2023-11-23 06:03:47,470 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.98 vs. limit=15.0
2023-11-23 06:03:52,619 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:04:00,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2263040.0, ans=0.035
2023-11-23 06:04:03,034 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2800, loss[loss=0.06524, simple_loss=0.0887, pruned_loss=0.01398, audio_tagging_loss=0.00691, over 15562.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09079, pruned_loss=0.01392, audio_tagging_loss=0.009073, over 3037646.39 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 06:04:09,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2263106.6666666665, ans=0.0
2023-11-23 06:04:22,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2263173.3333333335, ans=0.0
2023-11-23 06:04:28,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2263240.0, ans=0.1
2023-11-23 06:04:43,994 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.163e+01 8.713e+01 9.405e+01 1.279e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-23 06:04:44,150 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339500
2023-11-23 06:05:05,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2263440.0, ans=0.125
2023-11-23 06:05:06,457 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2850, loss[loss=0.05847, simple_loss=0.08132, pruned_loss=0.008348, audio_tagging_loss=0.009467, over 14701.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09191, pruned_loss=0.01419, audio_tagging_loss=0.008965, over 3036397.48 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:05:25,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2263506.6666666665, ans=0.125
2023-11-23 06:05:31,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2263506.6666666665, ans=0.2
2023-11-23 06:05:48,741 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339550
2023-11-23 06:05:49,385 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.28 vs. limit=22.5
2023-11-23 06:05:55,508 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.39 vs. limit=10.0
2023-11-23 06:06:11,456 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2900, loss[loss=0.08714, simple_loss=0.1233, pruned_loss=0.01945, audio_tagging_loss=0.006035, over 16595.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09224, pruned_loss=0.01434, audio_tagging_loss=0.00901, over 3040627.96 frames. ], batch size: 63, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:06:29,885 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.41 vs. limit=15.0
2023-11-23 06:06:49,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2263973.3333333335, ans=0.125
2023-11-23 06:06:52,983 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.150e+01 8.750e+01 9.470e+01 1.345e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-23 06:06:53,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339600
2023-11-23 06:06:53,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2263973.3333333335, ans=0.0
2023-11-23 06:07:11,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2264040.0, ans=0.0
2023-11-23 06:07:17,750 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 2950, loss[loss=0.05627, simple_loss=0.07632, pruned_loss=0.009952, audio_tagging_loss=0.008154, over 16180.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.0919, pruned_loss=0.01427, audio_tagging_loss=0.009069, over 3046304.19 frames. ], batch size: 62, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:07:21,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2264106.6666666665, ans=0.1
2023-11-23 06:07:25,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2264106.6666666665, ans=0.1
2023-11-23 06:07:49,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2264240.0, ans=0.0
2023-11-23 06:07:50,649 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.61 vs. limit=15.0
2023-11-23 06:07:59,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339650
2023-11-23 06:08:22,225 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3000, loss[loss=0.08661, simple_loss=0.1198, pruned_loss=0.02025, audio_tagging_loss=0.006467, over 14391.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09198, pruned_loss=0.01423, audio_tagging_loss=0.009092, over 3048297.12 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:08:22,226 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 06:09:05,320 INFO [train_asr.py:1253] (2/4) Epoch 29, validation: loss=0.05823, simple_loss=0.05127, pruned_loss=0.005185, audio_tagging_loss=0.02741, over 4681554.00 frames. 
2023-11-23 06:09:05,321 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 06:09:08,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2264440.0, ans=0.1
2023-11-23 06:09:13,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2264440.0, ans=0.2
2023-11-23 06:09:23,883 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.24 vs. limit=22.5
2023-11-23 06:09:33,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2264573.3333333335, ans=0.1
2023-11-23 06:09:45,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2264640.0, ans=0.125
2023-11-23 06:09:46,390 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.265e+01 8.432e+01 8.978e+01 9.876e+01 1.396e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 06:09:46,547 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339700
2023-11-23 06:09:47,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2264640.0, ans=0.0
2023-11-23 06:10:02,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2264706.6666666665, ans=0.125
2023-11-23 06:10:10,841 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3050, loss[loss=0.06129, simple_loss=0.08216, pruned_loss=0.009948, audio_tagging_loss=0.01026, over 15962.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09295, pruned_loss=0.01443, audio_tagging_loss=0.009167, over 3039378.85 frames. ], batch size: 61, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:10:11,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2264773.3333333335, ans=0.04949747468305833
2023-11-23 06:10:23,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2264840.0, ans=0.125
2023-11-23 06:10:31,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2264840.0, ans=0.125
2023-11-23 06:10:35,781 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.42 vs. limit=15.0
2023-11-23 06:10:39,475 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0
2023-11-23 06:10:46,335 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:10:49,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2264973.3333333335, ans=0.07
2023-11-23 06:10:52,471 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339750
2023-11-23 06:10:56,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2264973.3333333335, ans=0.0
2023-11-23 06:11:14,626 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3100, loss[loss=0.08219, simple_loss=0.1073, pruned_loss=0.01993, audio_tagging_loss=0.008617, over 14808.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09299, pruned_loss=0.01433, audio_tagging_loss=0.009122, over 3040734.05 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 8.0
2023-11-23 06:11:19,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.44 vs. limit=12.0
2023-11-23 06:11:19,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2265106.6666666665, ans=0.0
2023-11-23 06:11:26,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2265173.3333333335, ans=0.125
2023-11-23 06:11:55,548 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339800
2023-11-23 06:11:56,580 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.703e+01 8.232e+01 8.826e+01 9.441e+01 1.388e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 06:12:02,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2265306.6666666665, ans=0.025
2023-11-23 06:12:05,040 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.67 vs. limit=12.0
2023-11-23 06:12:18,032 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3150, loss[loss=0.08174, simple_loss=0.1145, pruned_loss=0.01773, audio_tagging_loss=0.006765, over 15627.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.0934, pruned_loss=0.01427, audio_tagging_loss=0.00915, over 3041197.25 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 8.0
2023-11-23 06:12:22,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2265440.0, ans=0.2
2023-11-23 06:12:42,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.42 vs. limit=10.0
2023-11-23 06:12:58,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2265640.0, ans=0.0
2023-11-23 06:12:59,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339850
2023-11-23 06:13:04,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2265640.0, ans=0.125
2023-11-23 06:13:10,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2265706.6666666665, ans=0.1
2023-11-23 06:13:14,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2265706.6666666665, ans=0.125
2023-11-23 06:13:18,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2265706.6666666665, ans=0.2
2023-11-23 06:13:23,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2265773.3333333335, ans=0.125
2023-11-23 06:13:23,918 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3200, loss[loss=0.05972, simple_loss=0.07082, pruned_loss=0.01379, audio_tagging_loss=0.01052, over 15475.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.0939, pruned_loss=0.01437, audio_tagging_loss=0.009222, over 3047186.75 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:13:34,054 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:13:38,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2265840.0, ans=0.5
2023-11-23 06:13:42,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2265840.0, ans=0.125
2023-11-23 06:14:04,486 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339900
2023-11-23 06:14:06,192 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.473e+01 8.333e+01 9.166e+01 9.819e+01 1.242e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 06:14:10,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2265973.3333333335, ans=0.125
2023-11-23 06:14:18,900 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.14 vs. limit=22.5
2023-11-23 06:14:20,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2266040.0, ans=0.125
2023-11-23 06:14:23,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2266040.0, ans=0.1
2023-11-23 06:14:26,817 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3250, loss[loss=0.05649, simple_loss=0.06614, pruned_loss=0.01023, audio_tagging_loss=0.0132, over 14802.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09355, pruned_loss=0.01427, audio_tagging_loss=0.009227, over 3046279.48 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:14:28,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2266106.6666666665, ans=0.125
2023-11-23 06:14:33,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2266106.6666666665, ans=0.125
2023-11-23 06:14:33,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2266106.6666666665, ans=0.0
2023-11-23 06:14:38,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2266173.3333333335, ans=0.5
2023-11-23 06:14:41,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2266173.3333333335, ans=0.125
2023-11-23 06:14:43,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2266173.3333333335, ans=0.0
2023-11-23 06:14:51,252 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.70 vs. limit=22.5
2023-11-23 06:15:00,442 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.15 vs. limit=15.0
2023-11-23 06:15:06,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2266306.6666666665, ans=0.1
2023-11-23 06:15:08,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 339950
2023-11-23 06:15:16,758 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.67 vs. limit=8.0
2023-11-23 06:15:19,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2266373.3333333335, ans=0.0
2023-11-23 06:15:30,522 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3300, loss[loss=0.06395, simple_loss=0.08291, pruned_loss=0.01265, audio_tagging_loss=0.009843, over 16815.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09275, pruned_loss=0.01419, audio_tagging_loss=0.009331, over 3053811.67 frames. ], batch size: 64, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:15:54,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.93 vs. limit=15.0
2023-11-23 06:16:01,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2266573.3333333335, ans=0.125
2023-11-23 06:16:08,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2266640.0, ans=0.1
2023-11-23 06:16:12,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340000
2023-11-23 06:16:13,718 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.298e+01 8.900e+01 9.803e+01 1.322e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 06:16:30,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2266706.6666666665, ans=0.0
2023-11-23 06:16:34,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2266706.6666666665, ans=0.125
2023-11-23 06:16:40,084 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3350, loss[loss=0.09792, simple_loss=0.1319, pruned_loss=0.02472, audio_tagging_loss=0.007276, over 16890.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09364, pruned_loss=0.01447, audio_tagging_loss=0.009259, over 3050329.74 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:16:48,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2266773.3333333335, ans=0.125
2023-11-23 06:16:57,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2266840.0, ans=0.125
2023-11-23 06:16:57,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2266840.0, ans=0.125
2023-11-23 06:17:01,360 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0
2023-11-23 06:17:20,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340050
2023-11-23 06:17:23,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2266973.3333333335, ans=0.0
2023-11-23 06:17:33,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2267040.0, ans=0.0
2023-11-23 06:17:35,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.32 vs. limit=22.5
2023-11-23 06:17:43,969 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3400, loss[loss=0.05604, simple_loss=0.0765, pruned_loss=0.0106, audio_tagging_loss=0.007188, over 14503.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09387, pruned_loss=0.01454, audio_tagging_loss=0.009148, over 3055894.75 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:18:25,569 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340100
2023-11-23 06:18:26,635 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.359e+01 8.997e+01 9.733e+01 1.260e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 06:18:27,366 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.77 vs. limit=15.0
2023-11-23 06:18:43,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2267373.3333333335, ans=0.0
2023-11-23 06:18:47,386 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3450, loss[loss=0.06048, simple_loss=0.07552, pruned_loss=0.01293, audio_tagging_loss=0.009791, over 14533.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09399, pruned_loss=0.01454, audio_tagging_loss=0.009066, over 3053219.00 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:18:51,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2267440.0, ans=0.0
2023-11-23 06:19:29,745 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340150
2023-11-23 06:19:32,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.64 vs. limit=22.5
2023-11-23 06:19:53,634 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3500, loss[loss=0.07185, simple_loss=0.1023, pruned_loss=0.01336, audio_tagging_loss=0.007353, over 15044.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.0934, pruned_loss=0.01431, audio_tagging_loss=0.008981, over 3045865.77 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:19:59,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2267773.3333333335, ans=0.125
2023-11-23 06:20:07,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2267840.0, ans=0.125
2023-11-23 06:20:08,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2267840.0, ans=0.2
2023-11-23 06:20:23,009 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.17 vs. limit=22.5
2023-11-23 06:20:25,089 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:20:27,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2267906.6666666665, ans=0.0
2023-11-23 06:20:34,338 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340200
2023-11-23 06:20:35,330 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.175e+01 8.873e+01 9.452e+01 1.244e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 06:20:58,520 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3550, loss[loss=0.07291, simple_loss=0.09261, pruned_loss=0.01875, audio_tagging_loss=0.007865, over 15182.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09309, pruned_loss=0.01422, audio_tagging_loss=0.009001, over 3043885.72 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:21:09,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2268173.3333333335, ans=0.0
2023-11-23 06:21:37,658 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.37 vs. limit=15.0
2023-11-23 06:21:39,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340250
2023-11-23 06:21:58,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2268373.3333333335, ans=0.2
2023-11-23 06:22:01,958 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3600, loss[loss=0.1065, simple_loss=0.1465, pruned_loss=0.02721, audio_tagging_loss=0.00602, over 15227.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09328, pruned_loss=0.0142, audio_tagging_loss=0.009048, over 3052969.76 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:22:02,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2268440.0, ans=0.125
2023-11-23 06:22:02,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2268440.0, ans=0.0
2023-11-23 06:22:21,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2268506.6666666665, ans=0.125
2023-11-23 06:22:23,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2268506.6666666665, ans=0.0
2023-11-23 06:22:43,587 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340300
2023-11-23 06:22:44,650 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.233e+01 8.921e+01 9.911e+01 1.235e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 06:22:50,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.67 vs. limit=15.0
2023-11-23 06:22:54,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2268706.6666666665, ans=0.125
2023-11-23 06:22:57,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2268706.6666666665, ans=0.2
2023-11-23 06:23:02,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2268706.6666666665, ans=0.125
2023-11-23 06:23:07,291 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3650, loss[loss=0.08127, simple_loss=0.1106, pruned_loss=0.01741, audio_tagging_loss=0.008538, over 16109.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09332, pruned_loss=0.01438, audio_tagging_loss=0.009036, over 3058509.51 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:23:18,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2268840.0, ans=0.1
2023-11-23 06:23:47,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340350
2023-11-23 06:24:03,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.82 vs. limit=22.5
2023-11-23 06:24:10,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2269106.6666666665, ans=0.0
2023-11-23 06:24:11,247 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3700, loss[loss=0.0789, simple_loss=0.1087, pruned_loss=0.0167, audio_tagging_loss=0.007844, over 16893.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09288, pruned_loss=0.01421, audio_tagging_loss=0.009045, over 3056067.18 frames. ], batch size: 62, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:24:19,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2269106.6666666665, ans=0.125
2023-11-23 06:24:24,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2269173.3333333335, ans=0.2
2023-11-23 06:24:35,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2269240.0, ans=0.5
2023-11-23 06:24:52,588 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340400
2023-11-23 06:24:52,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2269306.6666666665, ans=0.2
2023-11-23 06:24:55,303 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.367e+01 8.831e+01 9.767e+01 1.153e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 06:25:16,101 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3750, loss[loss=0.0544, simple_loss=0.07044, pruned_loss=0.007728, audio_tagging_loss=0.01146, over 14874.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09291, pruned_loss=0.01414, audio_tagging_loss=0.009072, over 3048050.29 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:25:18,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2269440.0, ans=0.125
2023-11-23 06:25:57,882 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340450
2023-11-23 06:25:59,030 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:26:08,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2269706.6666666665, ans=0.125
2023-11-23 06:26:20,470 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3800, loss[loss=0.06429, simple_loss=0.08786, pruned_loss=0.01073, audio_tagging_loss=0.009638, over 14140.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09272, pruned_loss=0.01414, audio_tagging_loss=0.009189, over 3053478.92 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:26:47,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2269906.6666666665, ans=0.125
2023-11-23 06:27:01,992 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340500
2023-11-23 06:27:04,359 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.380e+01 8.994e+01 9.826e+01 1.163e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 06:27:12,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2270040.0, ans=0.1
2023-11-23 06:27:22,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2270040.0, ans=0.125
2023-11-23 06:27:26,029 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3850, loss[loss=0.06368, simple_loss=0.0799, pruned_loss=0.0149, audio_tagging_loss=0.008824, over 15310.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09246, pruned_loss=0.014, audio_tagging_loss=0.00924, over 3051038.54 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:27:41,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2270173.3333333335, ans=0.125
2023-11-23 06:27:48,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2270173.3333333335, ans=0.125
2023-11-23 06:27:50,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2270240.0, ans=0.2
2023-11-23 06:27:54,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2270240.0, ans=0.125
2023-11-23 06:28:07,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2270306.6666666665, ans=0.95
2023-11-23 06:28:07,176 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:28:08,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340550
2023-11-23 06:28:13,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2270306.6666666665, ans=0.0
2023-11-23 06:28:26,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2270373.3333333335, ans=0.125
2023-11-23 06:28:31,118 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3900, loss[loss=0.06471, simple_loss=0.08272, pruned_loss=0.01331, audio_tagging_loss=0.01004, over 13528.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09274, pruned_loss=0.01408, audio_tagging_loss=0.009245, over 3045131.15 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:28:38,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2270440.0, ans=0.0
2023-11-23 06:29:00,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2270573.3333333335, ans=0.125
2023-11-23 06:29:11,716 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340600
2023-11-23 06:29:14,347 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.444e+01 8.394e+01 8.868e+01 9.858e+01 1.292e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 06:29:20,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2270640.0, ans=0.0
2023-11-23 06:29:26,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2270706.6666666665, ans=0.0
2023-11-23 06:29:31,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2270706.6666666665, ans=0.125
2023-11-23 06:29:35,331 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 3950, loss[loss=0.06694, simple_loss=0.0926, pruned_loss=0.01365, audio_tagging_loss=0.006989, over 14455.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09305, pruned_loss=0.01424, audio_tagging_loss=0.009342, over 3048250.53 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:29:38,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.66 vs. limit=10.0
2023-11-23 06:30:07,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.26 vs. limit=15.0
2023-11-23 06:30:08,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2270906.6666666665, ans=0.1
2023-11-23 06:30:15,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2270973.3333333335, ans=0.0
2023-11-23 06:30:16,663 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340650
2023-11-23 06:30:40,561 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4000, loss[loss=0.05614, simple_loss=0.07229, pruned_loss=0.009972, audio_tagging_loss=0.01002, over 13805.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09384, pruned_loss=0.01449, audio_tagging_loss=0.009351, over 3047318.50 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:30:48,718 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.60 vs. limit=6.0
2023-11-23 06:30:56,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2271173.3333333335, ans=0.125
2023-11-23 06:31:15,296 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.33 vs. limit=15.0
2023-11-23 06:31:22,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340700
2023-11-23 06:31:23,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2271306.6666666665, ans=0.125
2023-11-23 06:31:25,686 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.305e+01 8.425e+01 8.951e+01 9.689e+01 1.650e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 06:31:27,455 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.91 vs. limit=15.0
2023-11-23 06:31:28,809 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=12.0
2023-11-23 06:31:32,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2271373.3333333335, ans=0.0
2023-11-23 06:31:40,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2271373.3333333335, ans=0.125
2023-11-23 06:31:43,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2271440.0, ans=0.125
2023-11-23 06:31:44,109 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4050, loss[loss=0.06725, simple_loss=0.09088, pruned_loss=0.01376, audio_tagging_loss=0.008045, over 14185.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.0934, pruned_loss=0.0144, audio_tagging_loss=0.00941, over 3040157.26 frames. ], batch size: 52, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:31:45,423 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:31:48,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2271440.0, ans=0.125
2023-11-23 06:31:49,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2271440.0, ans=0.0
2023-11-23 06:32:07,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2271506.6666666665, ans=0.125
2023-11-23 06:32:21,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2271573.3333333335, ans=0.125
2023-11-23 06:32:26,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340750
2023-11-23 06:32:48,646 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4100, loss[loss=0.06434, simple_loss=0.08538, pruned_loss=0.01323, audio_tagging_loss=0.008423, over 14870.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09349, pruned_loss=0.01449, audio_tagging_loss=0.009369, over 3043574.56 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:33:01,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2271840.0, ans=0.2
2023-11-23 06:33:12,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2271840.0, ans=0.1
2023-11-23 06:33:27,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2271973.3333333335, ans=0.125
2023-11-23 06:33:30,057 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340800
2023-11-23 06:33:33,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2271973.3333333335, ans=0.1
2023-11-23 06:33:34,562 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.442e+01 8.896e+01 9.495e+01 1.100e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 06:33:49,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2272040.0, ans=0.95
2023-11-23 06:33:54,768 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4150, loss[loss=0.05865, simple_loss=0.0786, pruned_loss=0.008924, audio_tagging_loss=0.01043, over 14972.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09332, pruned_loss=0.01452, audio_tagging_loss=0.009262, over 3040844.83 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:34:18,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2272240.0, ans=0.125
2023-11-23 06:34:36,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340850
2023-11-23 06:34:37,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2272306.6666666665, ans=0.125
2023-11-23 06:34:38,717 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:34:38,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2272306.6666666665, ans=0.0
2023-11-23 06:34:40,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2272306.6666666665, ans=0.1
2023-11-23 06:34:58,251 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4200, loss[loss=0.06423, simple_loss=0.08618, pruned_loss=0.01188, audio_tagging_loss=0.009261, over 14978.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09348, pruned_loss=0.01445, audio_tagging_loss=0.009109, over 3044893.98 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:35:14,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2272506.6666666665, ans=0.125
2023-11-23 06:35:30,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2272573.3333333335, ans=0.1
2023-11-23 06:35:40,237 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340900
2023-11-23 06:35:43,731 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.311e+01 8.855e+01 9.584e+01 1.424e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 06:36:02,077 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4250, loss[loss=0.08118, simple_loss=0.1098, pruned_loss=0.0168, audio_tagging_loss=0.009483, over 15434.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09359, pruned_loss=0.01443, audio_tagging_loss=0.009051, over 3048939.98 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:36:41,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2272973.3333333335, ans=0.125
2023-11-23 06:36:43,957 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 340950
2023-11-23 06:36:44,574 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.29 vs. limit=15.0
2023-11-23 06:36:45,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2272973.3333333335, ans=0.125
2023-11-23 06:36:49,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2272973.3333333335, ans=0.125
2023-11-23 06:36:50,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2272973.3333333335, ans=0.0
2023-11-23 06:37:07,788 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4300, loss[loss=0.07022, simple_loss=0.09723, pruned_loss=0.01513, audio_tagging_loss=0.006467, over 14422.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09427, pruned_loss=0.01461, audio_tagging_loss=0.009083, over 3044249.72 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:37:14,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2273106.6666666665, ans=0.125
2023-11-23 06:37:48,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341000
2023-11-23 06:37:53,076 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.426e+01 8.312e+01 8.993e+01 9.823e+01 1.635e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 06:37:55,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2273306.6666666665, ans=0.125
2023-11-23 06:38:11,583 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4350, loss[loss=0.09441, simple_loss=0.1349, pruned_loss=0.02027, audio_tagging_loss=0.006682, over 15988.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09453, pruned_loss=0.01444, audio_tagging_loss=0.009065, over 3049179.28 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:38:20,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2273440.0, ans=0.5
2023-11-23 06:38:35,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2273573.3333333335, ans=0.5
2023-11-23 06:38:52,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341050
2023-11-23 06:38:58,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2273640.0, ans=0.2
2023-11-23 06:39:00,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2273640.0, ans=0.125
2023-11-23 06:39:04,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2273706.6666666665, ans=0.125
2023-11-23 06:39:14,626 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4400, loss[loss=0.06701, simple_loss=0.08876, pruned_loss=0.01299, audio_tagging_loss=0.009643, over 15156.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09418, pruned_loss=0.01438, audio_tagging_loss=0.009079, over 3053756.30 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:39:27,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2273840.0, ans=0.125
2023-11-23 06:39:27,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.87 vs. limit=22.5
2023-11-23 06:39:50,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2273906.6666666665, ans=0.2
2023-11-23 06:39:55,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341100
2023-11-23 06:40:00,189 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.327e+01 8.740e+01 9.683e+01 1.205e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 06:40:19,124 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4450, loss[loss=0.07121, simple_loss=0.1005, pruned_loss=0.01377, audio_tagging_loss=0.007184, over 13483.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09313, pruned_loss=0.0142, audio_tagging_loss=0.009046, over 3049768.12 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:40:40,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.03 vs. limit=15.0
2023-11-23 06:40:51,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2274240.0, ans=0.125
2023-11-23 06:40:54,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2274306.6666666665, ans=0.2
2023-11-23 06:40:57,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2274306.6666666665, ans=0.125
2023-11-23 06:40:58,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341150
2023-11-23 06:41:17,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2274373.3333333335, ans=0.1
2023-11-23 06:41:19,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2274373.3333333335, ans=0.1
2023-11-23 06:41:21,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2274440.0, ans=0.125
2023-11-23 06:41:22,094 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4500, loss[loss=0.06723, simple_loss=0.0881, pruned_loss=0.0153, audio_tagging_loss=0.007885, over 15160.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09271, pruned_loss=0.014, audio_tagging_loss=0.009025, over 3058194.94 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:41:23,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=2274440.0, ans=0.2
2023-11-23 06:41:32,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2274440.0, ans=0.125
2023-11-23 06:41:51,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2274573.3333333335, ans=0.125
2023-11-23 06:42:03,433 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341200
2023-11-23 06:42:08,635 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.193e+01 8.685e+01 9.597e+01 1.191e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 06:42:10,635 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.30 vs. limit=15.0
2023-11-23 06:42:14,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2274706.6666666665, ans=0.0
2023-11-23 06:42:17,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2274706.6666666665, ans=0.1
2023-11-23 06:42:25,802 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4550, loss[loss=0.07449, simple_loss=0.1042, pruned_loss=0.01585, audio_tagging_loss=0.006558, over 16046.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09181, pruned_loss=0.01378, audio_tagging_loss=0.009048, over 3047224.94 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:42:45,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2274840.0, ans=0.125
2023-11-23 06:42:47,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2274840.0, ans=0.125
2023-11-23 06:43:04,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2274973.3333333335, ans=0.1
2023-11-23 06:43:06,629 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341250
2023-11-23 06:43:11,441 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:43:15,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2275040.0, ans=0.2
2023-11-23 06:43:28,950 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4600, loss[loss=0.09727, simple_loss=0.1297, pruned_loss=0.0231, audio_tagging_loss=0.009327, over 14394.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09288, pruned_loss=0.01396, audio_tagging_loss=0.009088, over 3057927.59 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:44:07,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2275306.6666666665, ans=0.125
2023-11-23 06:44:08,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341300
2023-11-23 06:44:13,201 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.33 vs. limit=15.0
2023-11-23 06:44:13,567 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 8.306e+01 8.944e+01 9.725e+01 1.153e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 06:44:17,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2275306.6666666665, ans=0.0
2023-11-23 06:44:28,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2275373.3333333335, ans=0.2
2023-11-23 06:44:32,729 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4650, loss[loss=0.06226, simple_loss=0.08135, pruned_loss=0.01173, audio_tagging_loss=0.009854, over 15702.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09243, pruned_loss=0.01406, audio_tagging_loss=0.009185, over 3048908.82 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:44:36,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2275440.0, ans=0.125
2023-11-23 06:45:00,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2275573.3333333335, ans=0.1
2023-11-23 06:45:01,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2275573.3333333335, ans=0.125
2023-11-23 06:45:13,365 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341350
2023-11-23 06:45:36,211 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4700, loss[loss=0.06351, simple_loss=0.08657, pruned_loss=0.01354, audio_tagging_loss=0.006688, over 14132.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09199, pruned_loss=0.0141, audio_tagging_loss=0.009163, over 3046576.17 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:45:47,941 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.59 vs. limit=12.0
2023-11-23 06:46:00,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.54 vs. limit=12.0
2023-11-23 06:46:05,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2275906.6666666665, ans=0.0
2023-11-23 06:46:08,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.65 vs. limit=15.0
2023-11-23 06:46:17,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2275973.3333333335, ans=0.125
2023-11-23 06:46:18,068 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341400
2023-11-23 06:46:23,127 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.995e+01 8.437e+01 9.019e+01 9.512e+01 1.168e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 06:46:34,861 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.25 vs. limit=22.5
2023-11-23 06:46:40,572 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4750, loss[loss=0.0822, simple_loss=0.114, pruned_loss=0.01712, audio_tagging_loss=0.008099, over 15285.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.092, pruned_loss=0.01399, audio_tagging_loss=0.009262, over 3043466.63 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:46:47,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2276106.6666666665, ans=0.0
2023-11-23 06:47:22,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341450
2023-11-23 06:47:46,390 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4800, loss[loss=0.09727, simple_loss=0.1283, pruned_loss=0.02388, audio_tagging_loss=0.009225, over 14768.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09239, pruned_loss=0.01411, audio_tagging_loss=0.009418, over 3043615.62 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:47:56,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.84 vs. limit=15.0
2023-11-23 06:48:20,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2276573.3333333335, ans=0.0
2023-11-23 06:48:27,930 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341500
2023-11-23 06:48:32,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2276640.0, ans=0.1
2023-11-23 06:48:33,941 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.380e+01 8.789e+01 9.333e+01 1.267e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-23 06:48:50,500 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4850, loss[loss=0.07558, simple_loss=0.1074, pruned_loss=0.01225, audio_tagging_loss=0.009607, over 14850.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09334, pruned_loss=0.01442, audio_tagging_loss=0.00954, over 3044216.77 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:49:06,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2276840.0, ans=0.125
2023-11-23 06:49:09,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.93 vs. limit=15.0
2023-11-23 06:49:18,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2276906.6666666665, ans=0.1
2023-11-23 06:49:30,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2276973.3333333335, ans=0.125
2023-11-23 06:49:32,090 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341550
2023-11-23 06:49:34,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2276973.3333333335, ans=0.05
2023-11-23 06:49:37,584 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0
2023-11-23 06:49:40,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2277040.0, ans=0.125
2023-11-23 06:49:51,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2277040.0, ans=0.0
2023-11-23 06:49:51,909 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:49:54,002 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4900, loss[loss=0.06213, simple_loss=0.08472, pruned_loss=0.01173, audio_tagging_loss=0.008044, over 15145.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09322, pruned_loss=0.01435, audio_tagging_loss=0.009441, over 3045115.64 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:50:09,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.01 vs. limit=15.0
2023-11-23 06:50:32,696 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.60 vs. limit=12.0
2023-11-23 06:50:35,581 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341600
2023-11-23 06:50:41,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2277306.6666666665, ans=0.0
2023-11-23 06:50:41,947 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.445e+01 8.899e+01 9.661e+01 1.338e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 06:50:59,326 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 4950, loss[loss=0.06893, simple_loss=0.09227, pruned_loss=0.01396, audio_tagging_loss=0.008834, over 15653.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09287, pruned_loss=0.01424, audio_tagging_loss=0.009281, over 3050938.58 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:51:09,644 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.27 vs. limit=15.0
2023-11-23 06:51:12,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2277506.6666666665, ans=0.125
2023-11-23 06:51:13,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2277506.6666666665, ans=0.07
2023-11-23 06:51:40,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341650
2023-11-23 06:52:03,294 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5000, loss[loss=0.07743, simple_loss=0.09912, pruned_loss=0.01937, audio_tagging_loss=0.008494, over 14619.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09258, pruned_loss=0.01432, audio_tagging_loss=0.009111, over 3043733.50 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:52:04,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2277773.3333333335, ans=0.125
2023-11-23 06:52:07,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2277773.3333333335, ans=0.125
2023-11-23 06:52:16,371 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.13 vs. limit=15.0
2023-11-23 06:52:43,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341700
2023-11-23 06:52:45,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2277973.3333333335, ans=0.125
2023-11-23 06:52:50,295 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.065e+01 8.894e+01 1.001e+02 1.433e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 06:53:03,416 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.21 vs. limit=12.0
2023-11-23 06:53:06,447 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5050, loss[loss=0.05555, simple_loss=0.06745, pruned_loss=0.01209, audio_tagging_loss=0.009735, over 15690.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09255, pruned_loss=0.01431, audio_tagging_loss=0.009055, over 3043730.47 frames. ], batch size: 62, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:53:36,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2278240.0, ans=0.125
2023-11-23 06:53:42,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.11 vs. limit=15.0
2023-11-23 06:53:47,869 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341750
2023-11-23 06:53:49,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2278306.6666666665, ans=0.07
2023-11-23 06:53:58,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2278373.3333333335, ans=0.1
2023-11-23 06:54:10,956 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5100, loss[loss=0.06462, simple_loss=0.08473, pruned_loss=0.01144, audio_tagging_loss=0.01082, over 14860.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09236, pruned_loss=0.01422, audio_tagging_loss=0.009105, over 3043499.66 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:54:15,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2278440.0, ans=0.05
2023-11-23 06:54:21,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2278440.0, ans=0.125
2023-11-23 06:54:22,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2278506.6666666665, ans=0.2
2023-11-23 06:54:25,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.81 vs. limit=22.5
2023-11-23 06:54:29,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=15.0
2023-11-23 06:54:31,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.44 vs. limit=22.5
2023-11-23 06:54:32,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2278506.6666666665, ans=0.1
2023-11-23 06:54:36,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2278573.3333333335, ans=0.1
2023-11-23 06:54:37,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2278573.3333333335, ans=0.0
2023-11-23 06:54:41,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2278573.3333333335, ans=0.1
2023-11-23 06:54:42,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2278573.3333333335, ans=0.0
2023-11-23 06:54:52,226 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341800
2023-11-23 06:54:54,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2278640.0, ans=0.125
2023-11-23 06:54:57,531 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.72 vs. limit=15.0
2023-11-23 06:54:59,226 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 7.874e+01 8.463e+01 9.175e+01 1.215e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-23 06:55:02,375 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.01 vs. limit=12.0
2023-11-23 06:55:05,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2278706.6666666665, ans=0.5
2023-11-23 06:55:15,520 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5150, loss[loss=0.07392, simple_loss=0.09798, pruned_loss=0.01553, audio_tagging_loss=0.009406, over 15273.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09126, pruned_loss=0.01394, audio_tagging_loss=0.009168, over 3045011.56 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:55:15,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2278773.3333333335, ans=0.125
2023-11-23 06:55:16,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2278773.3333333335, ans=0.125
2023-11-23 06:55:17,309 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.24 vs. limit=6.0
2023-11-23 06:55:50,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2278906.6666666665, ans=0.125
2023-11-23 06:55:57,813 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341850
2023-11-23 06:56:06,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2279040.0, ans=0.2
2023-11-23 06:56:07,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2279040.0, ans=15.0
2023-11-23 06:56:18,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.03 vs. limit=6.0
2023-11-23 06:56:20,319 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5200, loss[loss=0.08644, simple_loss=0.1154, pruned_loss=0.02098, audio_tagging_loss=0.007783, over 14548.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09269, pruned_loss=0.0142, audio_tagging_loss=0.009048, over 3039072.16 frames. ], batch size: 52, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:56:20,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2279106.6666666665, ans=0.125
2023-11-23 06:56:25,744 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:56:42,386 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.01 vs. limit=10.0
2023-11-23 06:56:45,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=15.0
2023-11-23 06:56:46,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.24 vs. limit=10.0
2023-11-23 06:56:50,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2279240.0, ans=0.125
2023-11-23 06:56:55,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2279240.0, ans=0.125
2023-11-23 06:57:01,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341900
2023-11-23 06:57:02,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2279306.6666666665, ans=0.025
2023-11-23 06:57:09,207 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.360e+01 8.790e+01 9.384e+01 1.181e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-23 06:57:25,757 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5250, loss[loss=0.07224, simple_loss=0.1027, pruned_loss=0.01277, audio_tagging_loss=0.008128, over 13732.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09411, pruned_loss=0.01454, audio_tagging_loss=0.008906, over 3040071.26 frames. ], batch size: 50, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:57:26,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2279440.0, ans=0.07
2023-11-23 06:57:42,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2279506.6666666665, ans=0.125
2023-11-23 06:58:01,628 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.99 vs. limit=15.0
2023-11-23 06:58:06,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 341950
2023-11-23 06:58:13,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.28 vs. limit=22.5
2023-11-23 06:58:20,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2279706.6666666665, ans=0.125
2023-11-23 06:58:25,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2279706.6666666665, ans=0.05
2023-11-23 06:58:29,708 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5300, loss[loss=0.06229, simple_loss=0.08206, pruned_loss=0.0143, audio_tagging_loss=0.00696, over 15051.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09413, pruned_loss=0.01444, audio_tagging_loss=0.008853, over 3041935.07 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:58:39,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2279773.3333333335, ans=0.1
2023-11-23 06:58:48,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5
2023-11-23 06:58:48,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2279840.0, ans=0.07
2023-11-23 06:58:50,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.75 vs. limit=15.0
2023-11-23 06:59:00,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2279906.6666666665, ans=0.0
2023-11-23 06:59:01,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2279906.6666666665, ans=0.125
2023-11-23 06:59:11,263 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342000
2023-11-23 06:59:18,969 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.762e+01 8.363e+01 8.739e+01 9.478e+01 1.273e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 06:59:25,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2280040.0, ans=0.125
2023-11-23 06:59:33,712 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5350, loss[loss=0.08614, simple_loss=0.106, pruned_loss=0.02131, audio_tagging_loss=0.01185, over 14962.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09394, pruned_loss=0.01429, audio_tagging_loss=0.008871, over 3038474.61 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:59:35,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2280106.6666666665, ans=0.125
2023-11-23 06:59:46,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2280173.3333333335, ans=0.0
2023-11-23 07:00:00,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2280240.0, ans=0.125
2023-11-23 07:00:00,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2280240.0, ans=0.125
2023-11-23 07:00:15,458 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342050
2023-11-23 07:00:38,805 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5400, loss[loss=0.08402, simple_loss=0.1107, pruned_loss=0.02068, audio_tagging_loss=0.008014, over 14906.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09332, pruned_loss=0.01428, audio_tagging_loss=0.00896, over 3039146.36 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:00:42,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2280440.0, ans=0.0
2023-11-23 07:00:52,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2280506.6666666665, ans=0.125
2023-11-23 07:01:19,571 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342100
2023-11-23 07:01:27,618 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.345e+01 9.025e+01 9.492e+01 1.163e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 07:01:43,277 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5450, loss[loss=0.08449, simple_loss=0.1115, pruned_loss=0.01979, audio_tagging_loss=0.008959, over 15166.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09339, pruned_loss=0.01425, audio_tagging_loss=0.009101, over 3033499.00 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:02:01,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2280840.0, ans=0.125
2023-11-23 07:02:11,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2280906.6666666665, ans=0.0
2023-11-23 07:02:22,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2280973.3333333335, ans=0.0
2023-11-23 07:02:24,767 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342150
2023-11-23 07:02:27,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2280973.3333333335, ans=0.125
2023-11-23 07:02:28,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2280973.3333333335, ans=0.125
2023-11-23 07:02:40,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2281040.0, ans=0.0
2023-11-23 07:02:46,737 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5500, loss[loss=0.05776, simple_loss=0.0759, pruned_loss=0.01258, audio_tagging_loss=0.007238, over 15030.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09343, pruned_loss=0.01431, audio_tagging_loss=0.009084, over 3035837.69 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:02:59,267 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.50 vs. limit=15.0
2023-11-23 07:03:08,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2281173.3333333335, ans=0.1
2023-11-23 07:03:18,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2281240.0, ans=0.0
2023-11-23 07:03:18,854 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.81 vs. limit=15.0
2023-11-23 07:03:28,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342200
2023-11-23 07:03:32,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2281306.6666666665, ans=0.125
2023-11-23 07:03:35,748 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.376e+01 8.901e+01 9.455e+01 1.073e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 07:03:51,692 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5550, loss[loss=0.0655, simple_loss=0.0825, pruned_loss=0.01603, audio_tagging_loss=0.008227, over 14938.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.0934, pruned_loss=0.01431, audio_tagging_loss=0.009162, over 3033206.65 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:03:51,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2281440.0, ans=0.0
2023-11-23 07:04:06,455 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.69 vs. limit=15.0
2023-11-23 07:04:13,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2281506.6666666665, ans=0.125
2023-11-23 07:04:31,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342250
2023-11-23 07:04:55,522 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5600, loss[loss=0.07724, simple_loss=0.1004, pruned_loss=0.01419, audio_tagging_loss=0.01284, over 15671.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09286, pruned_loss=0.01419, audio_tagging_loss=0.009303, over 3042348.17 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 07:05:05,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2281773.3333333335, ans=0.125
2023-11-23 07:05:36,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342300
2023-11-23 07:05:38,725 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:05:43,429 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.528e+01 8.225e+01 8.879e+01 9.451e+01 1.680e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 07:05:52,419 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0
2023-11-23 07:05:57,944 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5650, loss[loss=0.07144, simple_loss=0.09541, pruned_loss=0.01533, audio_tagging_loss=0.008403, over 15800.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09274, pruned_loss=0.01405, audio_tagging_loss=0.009346, over 3047697.46 frames. ], batch size: 61, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 07:06:09,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2282173.3333333335, ans=0.125
2023-11-23 07:06:11,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2282173.3333333335, ans=0.1
2023-11-23 07:06:13,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2282173.3333333335, ans=0.1
2023-11-23 07:06:17,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2282173.3333333335, ans=0.04949747468305833
2023-11-23 07:06:19,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2282173.3333333335, ans=0.05
2023-11-23 07:06:35,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2282306.6666666665, ans=0.125
2023-11-23 07:06:39,130 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342350
2023-11-23 07:06:41,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2282306.6666666665, ans=0.125
2023-11-23 07:06:42,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.19 vs. limit=15.0
2023-11-23 07:06:57,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2282373.3333333335, ans=0.125
2023-11-23 07:06:59,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2282373.3333333335, ans=0.125
2023-11-23 07:07:01,226 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5700, loss[loss=0.06161, simple_loss=0.08022, pruned_loss=0.009748, audio_tagging_loss=0.01175, over 15039.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09191, pruned_loss=0.01397, audio_tagging_loss=0.009369, over 3046093.18 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:07:16,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2282506.6666666665, ans=0.1
2023-11-23 07:07:18,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2282506.6666666665, ans=0.0
2023-11-23 07:07:34,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2282573.3333333335, ans=0.1
2023-11-23 07:07:41,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342400
2023-11-23 07:07:48,842 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.83 vs. limit=15.0
2023-11-23 07:07:50,742 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.447e+01 8.964e+01 9.780e+01 1.182e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 07:07:56,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2282706.6666666665, ans=0.0
2023-11-23 07:08:06,329 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5750, loss[loss=0.07308, simple_loss=0.1063, pruned_loss=0.01322, audio_tagging_loss=0.0067, over 15447.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09173, pruned_loss=0.01406, audio_tagging_loss=0.009222, over 3048222.21 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:08:09,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=2282773.3333333335, ans=0.2
2023-11-23 07:08:12,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2282773.3333333335, ans=0.1
2023-11-23 07:08:20,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.53 vs. limit=22.5
2023-11-23 07:08:23,799 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:08:37,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2282906.6666666665, ans=0.125
2023-11-23 07:08:48,052 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342450
2023-11-23 07:08:55,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2282973.3333333335, ans=0.2
2023-11-23 07:09:06,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2283040.0, ans=0.125
2023-11-23 07:09:09,970 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5800, loss[loss=0.07196, simple_loss=0.09195, pruned_loss=0.01683, audio_tagging_loss=0.009159, over 14539.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09219, pruned_loss=0.01419, audio_tagging_loss=0.009069, over 3047004.80 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:09:10,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2283106.6666666665, ans=0.1
2023-11-23 07:09:51,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342500
2023-11-23 07:10:00,540 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.596e+01 8.186e+01 8.952e+01 9.751e+01 1.304e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 07:10:11,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2283373.3333333335, ans=0.125
2023-11-23 07:10:14,075 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5850, loss[loss=0.0717, simple_loss=0.08785, pruned_loss=0.01562, audio_tagging_loss=0.01215, over 14880.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09216, pruned_loss=0.01405, audio_tagging_loss=0.009093, over 3051927.36 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:10:14,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2283440.0, ans=0.1
2023-11-23 07:10:27,420 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=15.0
2023-11-23 07:10:34,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2283506.6666666665, ans=0.0
2023-11-23 07:10:41,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2283573.3333333335, ans=10.0
2023-11-23 07:10:43,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2283573.3333333335, ans=0.125
2023-11-23 07:10:49,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2283573.3333333335, ans=0.125
2023-11-23 07:10:55,922 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342550
2023-11-23 07:11:20,086 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5900, loss[loss=0.05979, simple_loss=0.07681, pruned_loss=0.01383, audio_tagging_loss=0.007562, over 16426.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09239, pruned_loss=0.01413, audio_tagging_loss=0.009108, over 3054288.71 frames. ], batch size: 63, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:11:28,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2283773.3333333335, ans=0.0
2023-11-23 07:11:29,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.15 vs. limit=15.0
2023-11-23 07:11:35,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2283840.0, ans=0.2
2023-11-23 07:12:01,270 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342600
2023-11-23 07:12:05,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2283973.3333333335, ans=0.125
2023-11-23 07:12:10,804 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.390e+01 8.881e+01 9.843e+01 1.391e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 07:12:18,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2284040.0, ans=0.125
2023-11-23 07:12:21,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=15.0
2023-11-23 07:12:24,213 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 5950, loss[loss=0.07117, simple_loss=0.09035, pruned_loss=0.01553, audio_tagging_loss=0.01046, over 14766.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09233, pruned_loss=0.01414, audio_tagging_loss=0.009068, over 3051514.67 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:12:36,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2284173.3333333335, ans=0.1
2023-11-23 07:12:58,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2284240.0, ans=0.125
2023-11-23 07:13:05,421 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342650
2023-11-23 07:13:05,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2284306.6666666665, ans=0.125
2023-11-23 07:13:06,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2284306.6666666665, ans=0.125
2023-11-23 07:13:10,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2284306.6666666665, ans=0.2
2023-11-23 07:13:27,466 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6000, loss[loss=0.05504, simple_loss=0.06714, pruned_loss=0.01121, audio_tagging_loss=0.01026, over 15219.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09229, pruned_loss=0.01409, audio_tagging_loss=0.009064, over 3046222.11 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:13:27,467 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 07:14:10,555 INFO [train_asr.py:1253] (2/4) Epoch 29, validation: loss=0.05847, simple_loss=0.05123, pruned_loss=0.005068, audio_tagging_loss=0.02778, over 4681554.00 frames. 
2023-11-23 07:14:10,556 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 07:14:10,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2284440.0, ans=0.125
2023-11-23 07:14:16,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2284440.0, ans=0.09899494936611666
2023-11-23 07:14:51,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342700
2023-11-23 07:14:55,092 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:15:00,584 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.161e+01 8.758e+01 9.491e+01 1.345e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 07:15:01,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2284706.6666666665, ans=0.0
2023-11-23 07:15:14,036 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6050, loss[loss=0.06989, simple_loss=0.1005, pruned_loss=0.009559, audio_tagging_loss=0.01008, over 15185.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09222, pruned_loss=0.01422, audio_tagging_loss=0.009045, over 3045224.32 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:15:36,590 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0
2023-11-23 07:15:52,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2284973.3333333335, ans=0.0
2023-11-23 07:15:55,591 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342750
2023-11-23 07:16:16,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2285106.6666666665, ans=0.2
2023-11-23 07:16:17,500 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6100, loss[loss=0.07797, simple_loss=0.1067, pruned_loss=0.01412, audio_tagging_loss=0.0105, over 15085.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09287, pruned_loss=0.0142, audio_tagging_loss=0.008982, over 3046087.23 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:16:28,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2285106.6666666665, ans=0.0
2023-11-23 07:16:30,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2285173.3333333335, ans=0.125
2023-11-23 07:16:51,335 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.08 vs. limit=15.0
2023-11-23 07:16:57,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2285306.6666666665, ans=0.0
2023-11-23 07:16:59,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342800
2023-11-23 07:16:59,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2285306.6666666665, ans=0.125
2023-11-23 07:17:05,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2285306.6666666665, ans=0.125
2023-11-23 07:17:06,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2285306.6666666665, ans=0.125
2023-11-23 07:17:08,160 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.337e+01 9.020e+01 9.718e+01 1.256e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 07:17:08,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2285373.3333333335, ans=0.1
2023-11-23 07:17:23,752 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6150, loss[loss=0.06788, simple_loss=0.09233, pruned_loss=0.01255, audio_tagging_loss=0.009172, over 14649.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09261, pruned_loss=0.01419, audio_tagging_loss=0.008963, over 3047842.00 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:17:27,120 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.09 vs. limit=15.0
2023-11-23 07:17:30,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2285440.0, ans=0.125
2023-11-23 07:17:35,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2285506.6666666665, ans=0.125
2023-11-23 07:17:40,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2285506.6666666665, ans=0.2
2023-11-23 07:17:41,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2285506.6666666665, ans=0.0
2023-11-23 07:17:45,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff2.min_abs, batch_count=2285506.6666666665, ans=0.1
2023-11-23 07:17:56,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2285573.3333333335, ans=0.1
2023-11-23 07:18:05,134 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342850
2023-11-23 07:18:28,761 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6200, loss[loss=0.07089, simple_loss=0.08818, pruned_loss=0.01497, audio_tagging_loss=0.01183, over 15457.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09273, pruned_loss=0.01417, audio_tagging_loss=0.009119, over 3046696.30 frames. ], batch size: 62, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:18:32,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2285773.3333333335, ans=0.0
2023-11-23 07:18:40,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2285840.0, ans=0.1
2023-11-23 07:18:41,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2285840.0, ans=0.125
2023-11-23 07:18:58,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2285906.6666666665, ans=0.125
2023-11-23 07:19:06,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2285973.3333333335, ans=0.125
2023-11-23 07:19:09,334 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342900
2023-11-23 07:19:19,545 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.276e+01 8.850e+01 9.566e+01 1.179e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 07:19:31,986 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6250, loss[loss=0.07595, simple_loss=0.09801, pruned_loss=0.01861, audio_tagging_loss=0.008333, over 14703.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.0922, pruned_loss=0.01424, audio_tagging_loss=0.009288, over 3050867.43 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:19:37,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2286106.6666666665, ans=0.0
2023-11-23 07:20:01,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2286240.0, ans=0.025
2023-11-23 07:20:14,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 342950
2023-11-23 07:20:37,723 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6300, loss[loss=0.07443, simple_loss=0.09828, pruned_loss=0.01509, audio_tagging_loss=0.0102, over 14896.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09307, pruned_loss=0.01443, audio_tagging_loss=0.009299, over 3043186.27 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:20:56,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.05 vs. limit=22.5
2023-11-23 07:21:02,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2286573.3333333335, ans=0.0
2023-11-23 07:21:04,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2286573.3333333335, ans=0.0
2023-11-23 07:21:15,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2286640.0, ans=0.125
2023-11-23 07:21:18,297 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343000
2023-11-23 07:21:26,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2286640.0, ans=0.125
2023-11-23 07:21:27,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.46 vs. limit=6.0
2023-11-23 07:21:29,495 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.995e+01 8.227e+01 8.680e+01 9.495e+01 1.255e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 07:21:32,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2286706.6666666665, ans=0.1
2023-11-23 07:21:34,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2286706.6666666665, ans=0.0
2023-11-23 07:21:42,430 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6350, loss[loss=0.09141, simple_loss=0.1165, pruned_loss=0.02464, audio_tagging_loss=0.008512, over 15496.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09229, pruned_loss=0.01426, audio_tagging_loss=0.009397, over 3038759.98 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:21:44,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2286773.3333333335, ans=0.1
2023-11-23 07:22:11,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.77 vs. limit=15.0
2023-11-23 07:22:12,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2286906.6666666665, ans=0.2
2023-11-23 07:22:23,564 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343050
2023-11-23 07:22:46,284 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6400, loss[loss=0.07963, simple_loss=0.09937, pruned_loss=0.01779, audio_tagging_loss=0.01216, over 14745.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.0924, pruned_loss=0.01439, audio_tagging_loss=0.009458, over 3035031.13 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:22:56,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2287106.6666666665, ans=0.125
2023-11-23 07:22:58,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2287173.3333333335, ans=0.125
2023-11-23 07:23:01,539 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.67 vs. limit=22.5
2023-11-23 07:23:28,276 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343100
2023-11-23 07:23:37,926 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.257e+01 8.871e+01 9.376e+01 1.262e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 07:23:51,286 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6450, loss[loss=0.05444, simple_loss=0.07047, pruned_loss=0.009493, audio_tagging_loss=0.00971, over 15851.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09226, pruned_loss=0.01441, audio_tagging_loss=0.009543, over 3030056.45 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:24:12,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.32 vs. limit=10.0
2023-11-23 07:24:16,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2287573.3333333335, ans=0.1
2023-11-23 07:24:20,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.21 vs. limit=22.5
2023-11-23 07:24:29,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2287640.0, ans=0.2
2023-11-23 07:24:32,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343150
2023-11-23 07:24:34,008 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.65 vs. limit=10.0
2023-11-23 07:24:34,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2287640.0, ans=0.0
2023-11-23 07:24:51,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2287706.6666666665, ans=0.125
2023-11-23 07:24:55,833 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6500, loss[loss=0.07878, simple_loss=0.1077, pruned_loss=0.0165, audio_tagging_loss=0.008406, over 16402.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09299, pruned_loss=0.01456, audio_tagging_loss=0.009454, over 3042567.46 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:25:27,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2287906.6666666665, ans=0.125
2023-11-23 07:25:32,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2287906.6666666665, ans=0.125
2023-11-23 07:25:37,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343200
2023-11-23 07:25:47,697 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.151e+01 8.781e+01 9.682e+01 1.216e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-23 07:25:49,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2288040.0, ans=0.125
2023-11-23 07:25:56,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2288040.0, ans=0.0
2023-11-23 07:26:00,748 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6550, loss[loss=0.07821, simple_loss=0.1104, pruned_loss=0.01676, audio_tagging_loss=0.006253, over 15483.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09353, pruned_loss=0.01449, audio_tagging_loss=0.009283, over 3046022.89 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:26:23,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.58 vs. limit=5.0
2023-11-23 07:26:28,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2288240.0, ans=10.0
2023-11-23 07:26:30,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2288240.0, ans=0.0
2023-11-23 07:26:39,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2288306.6666666665, ans=0.125
2023-11-23 07:26:42,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343250
2023-11-23 07:26:59,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2288373.3333333335, ans=0.1
2023-11-23 07:27:05,401 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6600, loss[loss=0.07681, simple_loss=0.1, pruned_loss=0.01579, audio_tagging_loss=0.01103, over 14126.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09385, pruned_loss=0.01447, audio_tagging_loss=0.009106, over 3042743.22 frames. ], batch size: 53, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:27:46,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343300
2023-11-23 07:27:48,828 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.74 vs. limit=10.0
2023-11-23 07:27:53,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2288640.0, ans=0.125
2023-11-23 07:27:59,914 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.340e+01 9.003e+01 9.613e+01 1.234e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 07:28:00,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2288706.6666666665, ans=0.125
2023-11-23 07:28:10,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2288773.3333333335, ans=0.0
2023-11-23 07:28:11,247 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6650, loss[loss=0.06789, simple_loss=0.08341, pruned_loss=0.01471, audio_tagging_loss=0.01147, over 15348.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09436, pruned_loss=0.01454, audio_tagging_loss=0.009071, over 3037197.39 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:28:19,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2288773.3333333335, ans=10.0
2023-11-23 07:28:26,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2288840.0, ans=0.1
2023-11-23 07:28:35,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2288906.6666666665, ans=0.125
2023-11-23 07:28:52,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343350
2023-11-23 07:29:14,903 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6700, loss[loss=0.07635, simple_loss=0.09664, pruned_loss=0.02063, audio_tagging_loss=0.007401, over 15936.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09347, pruned_loss=0.01449, audio_tagging_loss=0.009132, over 3035532.38 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:29:33,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2289173.3333333335, ans=0.2
2023-11-23 07:29:55,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2289306.6666666665, ans=0.1
2023-11-23 07:29:56,367 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343400
2023-11-23 07:30:00,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2289306.6666666665, ans=0.0
2023-11-23 07:30:07,554 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.148e+01 8.906e+01 9.681e+01 1.677e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 07:30:08,334 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.13 vs. limit=15.0
2023-11-23 07:30:15,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2289373.3333333335, ans=0.2
2023-11-23 07:30:19,374 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6750, loss[loss=0.05979, simple_loss=0.07311, pruned_loss=0.01325, audio_tagging_loss=0.009983, over 16241.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.0923, pruned_loss=0.01433, audio_tagging_loss=0.009198, over 3030866.94 frames. ], batch size: 62, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:30:40,312 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.79 vs. limit=15.0
2023-11-23 07:31:00,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343450
2023-11-23 07:31:14,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2289706.6666666665, ans=0.0
2023-11-23 07:31:24,897 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6800, loss[loss=0.06381, simple_loss=0.08796, pruned_loss=0.01037, audio_tagging_loss=0.009461, over 15948.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09264, pruned_loss=0.01428, audio_tagging_loss=0.009136, over 3030314.43 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:31:26,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2289773.3333333335, ans=0.125
2023-11-23 07:31:32,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2289773.3333333335, ans=0.125
2023-11-23 07:31:34,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2289773.3333333335, ans=0.125
2023-11-23 07:32:03,038 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.74 vs. limit=12.0
2023-11-23 07:32:06,724 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343500
2023-11-23 07:32:17,723 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.151e+01 8.874e+01 9.532e+01 1.488e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 07:32:20,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2290040.0, ans=0.0
2023-11-23 07:32:21,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2290040.0, ans=0.2
2023-11-23 07:32:28,857 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6850, loss[loss=0.07915, simple_loss=0.1052, pruned_loss=0.02012, audio_tagging_loss=0.0064, over 14584.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09193, pruned_loss=0.01403, audio_tagging_loss=0.009063, over 3028324.68 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:32:37,038 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.61 vs. limit=22.5
2023-11-23 07:33:11,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343550
2023-11-23 07:33:17,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.98 vs. limit=22.5
2023-11-23 07:33:22,912 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.16 vs. limit=15.0
2023-11-23 07:33:33,144 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6900, loss[loss=0.06296, simple_loss=0.09015, pruned_loss=0.01062, audio_tagging_loss=0.00726, over 14679.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09205, pruned_loss=0.01394, audio_tagging_loss=0.009085, over 3029596.63 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:33:47,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2290506.6666666665, ans=0.0
2023-11-23 07:33:57,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.95 vs. limit=15.0
2023-11-23 07:34:06,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2290573.3333333335, ans=0.0
2023-11-23 07:34:07,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2290573.3333333335, ans=10.0
2023-11-23 07:34:14,574 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343600
2023-11-23 07:34:22,089 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:34:26,257 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.590e+01 8.208e+01 8.939e+01 9.660e+01 1.242e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 07:34:34,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2290706.6666666665, ans=0.2
2023-11-23 07:34:39,255 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 6950, loss[loss=0.0681, simple_loss=0.09165, pruned_loss=0.01562, audio_tagging_loss=0.006651, over 15023.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09186, pruned_loss=0.0139, audio_tagging_loss=0.009096, over 3032730.62 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:34:44,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2290773.3333333335, ans=0.1
2023-11-23 07:34:46,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2290773.3333333335, ans=0.035
2023-11-23 07:34:53,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.91 vs. limit=6.0
2023-11-23 07:35:15,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2290973.3333333335, ans=0.035
2023-11-23 07:35:17,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2290973.3333333335, ans=0.125
2023-11-23 07:35:19,878 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343650
2023-11-23 07:35:30,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2291040.0, ans=0.125
2023-11-23 07:35:41,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2291106.6666666665, ans=0.125
2023-11-23 07:35:42,388 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7000, loss[loss=0.09539, simple_loss=0.1212, pruned_loss=0.02584, audio_tagging_loss=0.008925, over 14824.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09265, pruned_loss=0.01411, audio_tagging_loss=0.009083, over 3032029.96 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:35:44,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2291106.6666666665, ans=0.125
2023-11-23 07:35:53,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2291173.3333333335, ans=0.125
2023-11-23 07:35:58,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2291173.3333333335, ans=0.125
2023-11-23 07:36:04,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2291173.3333333335, ans=0.04949747468305833
2023-11-23 07:36:07,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2291240.0, ans=0.2
2023-11-23 07:36:07,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2291240.0, ans=0.125
2023-11-23 07:36:20,641 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:36:24,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343700
2023-11-23 07:36:34,905 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.255e+01 8.925e+01 9.735e+01 1.377e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 07:36:36,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.18 vs. limit=15.0
2023-11-23 07:36:37,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2291373.3333333335, ans=0.04949747468305833
2023-11-23 07:36:45,868 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7050, loss[loss=0.06781, simple_loss=0.07888, pruned_loss=0.01816, audio_tagging_loss=0.01022, over 14191.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.092, pruned_loss=0.01403, audio_tagging_loss=0.009146, over 3027928.85 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:36:48,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2291440.0, ans=0.0
2023-11-23 07:36:52,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2291440.0, ans=0.1
2023-11-23 07:37:04,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2291506.6666666665, ans=0.125
2023-11-23 07:37:27,567 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343750
2023-11-23 07:37:52,037 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7100, loss[loss=0.07142, simple_loss=0.08364, pruned_loss=0.01884, audio_tagging_loss=0.01076, over 14754.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09264, pruned_loss=0.01422, audio_tagging_loss=0.009119, over 3027146.62 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:38:07,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2291840.0, ans=0.0
2023-11-23 07:38:12,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2291840.0, ans=0.0
2023-11-23 07:38:14,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2291840.0, ans=0.1
2023-11-23 07:38:32,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343800
2023-11-23 07:38:43,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2292040.0, ans=0.125
2023-11-23 07:38:45,437 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.319e+01 8.862e+01 9.684e+01 1.135e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 07:38:51,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2292040.0, ans=0.0
2023-11-23 07:38:56,646 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7150, loss[loss=0.08463, simple_loss=0.1078, pruned_loss=0.01978, audio_tagging_loss=0.01098, over 14816.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09226, pruned_loss=0.01417, audio_tagging_loss=0.009237, over 3040239.26 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:39:08,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2292173.3333333335, ans=0.125
2023-11-23 07:39:15,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2292173.3333333335, ans=0.125
2023-11-23 07:39:38,220 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343850
2023-11-23 07:39:38,664 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.14 vs. limit=15.0
2023-11-23 07:39:40,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2292306.6666666665, ans=0.0
2023-11-23 07:39:42,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-23 07:39:42,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.14 vs. limit=22.5
2023-11-23 07:39:50,800 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.44 vs. limit=15.0
2023-11-23 07:40:00,100 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7200, loss[loss=0.06488, simple_loss=0.08675, pruned_loss=0.01209, audio_tagging_loss=0.009411, over 15651.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09183, pruned_loss=0.01396, audio_tagging_loss=0.009356, over 3043551.98 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:40:17,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2292506.6666666665, ans=0.0
2023-11-23 07:40:22,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2292506.6666666665, ans=0.1
2023-11-23 07:40:32,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2292573.3333333335, ans=0.125
2023-11-23 07:40:41,610 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343900
2023-11-23 07:40:46,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2292640.0, ans=0.0
2023-11-23 07:40:52,452 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.353e+01 8.830e+01 9.779e+01 1.772e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 07:41:05,600 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7250, loss[loss=0.05289, simple_loss=0.06867, pruned_loss=0.007049, audio_tagging_loss=0.01151, over 15731.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09211, pruned_loss=0.01404, audio_tagging_loss=0.009425, over 3047337.53 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:41:10,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2292773.3333333335, ans=0.0
2023-11-23 07:41:11,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2292773.3333333335, ans=0.125
2023-11-23 07:41:15,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2292773.3333333335, ans=0.125
2023-11-23 07:41:37,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2292906.6666666665, ans=0.2
2023-11-23 07:41:37,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.41 vs. limit=22.5
2023-11-23 07:41:45,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 343950
2023-11-23 07:42:01,916 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0
2023-11-23 07:42:09,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2293106.6666666665, ans=0.2
2023-11-23 07:42:10,342 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7300, loss[loss=0.07787, simple_loss=0.1085, pruned_loss=0.01795, audio_tagging_loss=0.005667, over 16565.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09228, pruned_loss=0.0141, audio_tagging_loss=0.0093, over 3043155.90 frames. ], batch size: 62, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:42:11,168 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.49 vs. limit=10.0
2023-11-23 07:42:32,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2293173.3333333335, ans=0.2
2023-11-23 07:42:39,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.91 vs. limit=15.0
2023-11-23 07:42:39,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2293240.0, ans=0.125
2023-11-23 07:42:40,401 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0
2023-11-23 07:42:43,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2293240.0, ans=0.0
2023-11-23 07:42:51,456 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344000
2023-11-23 07:43:00,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2293306.6666666665, ans=0.125
2023-11-23 07:43:04,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2293373.3333333335, ans=0.0
2023-11-23 07:43:06,583 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.353e+01 8.994e+01 9.658e+01 1.142e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 07:43:17,904 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7350, loss[loss=0.0617, simple_loss=0.08295, pruned_loss=0.0126, audio_tagging_loss=0.00763, over 13772.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09292, pruned_loss=0.01425, audio_tagging_loss=0.00924, over 3044998.80 frames. ], batch size: 52, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:43:20,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2293440.0, ans=0.1
2023-11-23 07:43:26,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2293440.0, ans=0.1
2023-11-23 07:43:39,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2293506.6666666665, ans=0.0
2023-11-23 07:43:43,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2293573.3333333335, ans=0.125
2023-11-23 07:43:44,719 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:43:54,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2293573.3333333335, ans=0.07
2023-11-23 07:43:56,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2293640.0, ans=0.05
2023-11-23 07:44:00,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344050
2023-11-23 07:44:03,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.72 vs. limit=22.5
2023-11-23 07:44:15,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2293706.6666666665, ans=0.0
2023-11-23 07:44:19,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2293706.6666666665, ans=0.0
2023-11-23 07:44:23,608 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7400, loss[loss=0.05994, simple_loss=0.08537, pruned_loss=0.009401, audio_tagging_loss=0.00785, over 15295.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09255, pruned_loss=0.01418, audio_tagging_loss=0.009169, over 3047388.87 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:44:36,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2293840.0, ans=0.125
2023-11-23 07:44:54,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2293906.6666666665, ans=0.1
2023-11-23 07:45:04,062 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344100
2023-11-23 07:45:04,296 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:45:10,618 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.16 vs. limit=12.0
2023-11-23 07:45:12,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2293973.3333333335, ans=0.2
2023-11-23 07:45:17,739 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.228e+01 8.684e+01 9.636e+01 1.290e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 07:45:28,108 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7450, loss[loss=0.07476, simple_loss=0.107, pruned_loss=0.01379, audio_tagging_loss=0.007478, over 15772.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09236, pruned_loss=0.01418, audio_tagging_loss=0.009087, over 3043732.18 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:45:30,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2294106.6666666665, ans=0.125
2023-11-23 07:45:50,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2294173.3333333335, ans=0.125
2023-11-23 07:45:50,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=2294173.3333333335, ans=15.0
2023-11-23 07:46:09,188 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344150
2023-11-23 07:46:10,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2294306.6666666665, ans=0.07
2023-11-23 07:46:21,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.58 vs. limit=6.0
2023-11-23 07:46:31,660 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7500, loss[loss=0.06721, simple_loss=0.09173, pruned_loss=0.01357, audio_tagging_loss=0.007767, over 15660.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09095, pruned_loss=0.01401, audio_tagging_loss=0.009092, over 3045380.20 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:46:38,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2294440.0, ans=0.0
2023-11-23 07:47:06,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2294573.3333333335, ans=0.025
2023-11-23 07:47:13,232 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344200
2023-11-23 07:47:25,789 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.286e+01 8.811e+01 9.468e+01 1.775e+02, threshold=1.762e+02, percent-clipped=1.0
2023-11-23 07:47:29,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2294706.6666666665, ans=0.0
2023-11-23 07:47:32,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2294706.6666666665, ans=0.0
2023-11-23 07:47:35,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2294773.3333333335, ans=0.07
2023-11-23 07:47:36,017 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7550, loss[loss=0.0742, simple_loss=0.1098, pruned_loss=0.01246, audio_tagging_loss=0.006856, over 16196.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09091, pruned_loss=0.01411, audio_tagging_loss=0.009049, over 3039394.71 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:47:48,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2294840.0, ans=0.125
2023-11-23 07:48:08,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2294906.6666666665, ans=0.0
2023-11-23 07:48:12,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2294906.6666666665, ans=0.125
2023-11-23 07:48:12,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2294906.6666666665, ans=0.0
2023-11-23 07:48:17,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344250
2023-11-23 07:48:36,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2295040.0, ans=0.125
2023-11-23 07:48:41,144 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7600, loss[loss=0.06711, simple_loss=0.08858, pruned_loss=0.01246, audio_tagging_loss=0.01036, over 15795.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.0914, pruned_loss=0.01427, audio_tagging_loss=0.008997, over 3041019.13 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:48:54,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2295173.3333333335, ans=0.125
2023-11-23 07:49:22,996 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344300
2023-11-23 07:49:27,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.87 vs. limit=22.5
2023-11-23 07:49:28,317 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.40 vs. limit=10.0
2023-11-23 07:49:34,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2295373.3333333335, ans=0.125
2023-11-23 07:49:35,032 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.274e+01 8.656e+01 9.484e+01 1.340e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-23 07:49:36,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2295373.3333333335, ans=0.1
2023-11-23 07:49:37,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2295373.3333333335, ans=0.0
2023-11-23 07:49:43,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2295373.3333333335, ans=0.5
2023-11-23 07:49:45,377 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7650, loss[loss=0.06252, simple_loss=0.08448, pruned_loss=0.008975, audio_tagging_loss=0.01131, over 15693.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09092, pruned_loss=0.01403, audio_tagging_loss=0.008951, over 3047160.35 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:50:01,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2295506.6666666665, ans=0.125
2023-11-23 07:50:11,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2295573.3333333335, ans=0.125
2023-11-23 07:50:19,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2295573.3333333335, ans=0.2
2023-11-23 07:50:24,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2295640.0, ans=0.07
2023-11-23 07:50:26,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344350
2023-11-23 07:50:41,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2295706.6666666665, ans=0.125
2023-11-23 07:50:44,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2295706.6666666665, ans=0.125
2023-11-23 07:50:46,790 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:50:48,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2295773.3333333335, ans=0.125
2023-11-23 07:50:48,907 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7700, loss[loss=0.05649, simple_loss=0.07539, pruned_loss=0.007971, audio_tagging_loss=0.01083, over 14602.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09146, pruned_loss=0.01395, audio_tagging_loss=0.009038, over 3048278.52 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:51:07,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2295840.0, ans=0.0
2023-11-23 07:51:10,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2295840.0, ans=0.0
2023-11-23 07:51:13,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2295906.6666666665, ans=0.0
2023-11-23 07:51:30,229 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344400
2023-11-23 07:51:44,901 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.308e+01 8.349e+01 8.922e+01 9.569e+01 1.181e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 07:51:54,180 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7750, loss[loss=0.06275, simple_loss=0.09501, pruned_loss=0.007977, audio_tagging_loss=0.007271, over 15803.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09301, pruned_loss=0.01433, audio_tagging_loss=0.008929, over 3055477.07 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:51:57,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2296106.6666666665, ans=0.0
2023-11-23 07:52:00,741 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.16 vs. limit=15.0
2023-11-23 07:52:06,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2296173.3333333335, ans=0.0
2023-11-23 07:52:15,865 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0
2023-11-23 07:52:27,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=15.0
2023-11-23 07:52:35,147 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344450
2023-11-23 07:52:39,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2296306.6666666665, ans=0.125
2023-11-23 07:52:44,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2296373.3333333335, ans=0.125
2023-11-23 07:52:44,861 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.84 vs. limit=15.0
2023-11-23 07:52:49,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2296373.3333333335, ans=0.0
2023-11-23 07:52:54,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2296373.3333333335, ans=0.1
2023-11-23 07:52:57,557 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7800, loss[loss=0.06282, simple_loss=0.08047, pruned_loss=0.01361, audio_tagging_loss=0.008982, over 14228.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09353, pruned_loss=0.01438, audio_tagging_loss=0.008899, over 3050343.70 frames. ], batch size: 53, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:53:00,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2296440.0, ans=0.125
2023-11-23 07:53:05,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2296440.0, ans=0.125
2023-11-23 07:53:26,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2296573.3333333335, ans=0.125
2023-11-23 07:53:39,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344500
2023-11-23 07:53:39,913 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0
2023-11-23 07:53:46,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2296640.0, ans=0.1
2023-11-23 07:53:49,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2296706.6666666665, ans=0.125
2023-11-23 07:53:53,390 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.147e+01 8.634e+01 9.399e+01 1.178e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-23 07:54:01,924 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7850, loss[loss=0.06876, simple_loss=0.09468, pruned_loss=0.01259, audio_tagging_loss=0.00883, over 15052.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09459, pruned_loss=0.01458, audio_tagging_loss=0.009023, over 3047251.49 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:54:07,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2296773.3333333335, ans=0.0
2023-11-23 07:54:14,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2296840.0, ans=0.125
2023-11-23 07:54:25,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2296840.0, ans=0.0
2023-11-23 07:54:43,724 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344550
2023-11-23 07:54:55,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=22.5
2023-11-23 07:55:07,324 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7900, loss[loss=0.06598, simple_loss=0.08216, pruned_loss=0.01305, audio_tagging_loss=0.01185, over 14001.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09336, pruned_loss=0.01436, audio_tagging_loss=0.009223, over 3042750.68 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:55:19,088 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.22 vs. limit=10.0
2023-11-23 07:55:23,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2297173.3333333335, ans=0.125
2023-11-23 07:55:40,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2297240.0, ans=0.125
2023-11-23 07:55:41,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2297240.0, ans=0.125
2023-11-23 07:55:42,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2297240.0, ans=0.125
2023-11-23 07:55:48,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344600
2023-11-23 07:55:52,081 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.33 vs. limit=15.0
2023-11-23 07:56:02,996 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.370e+01 9.081e+01 9.662e+01 1.223e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 07:56:11,605 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 7950, loss[loss=0.0715, simple_loss=0.1008, pruned_loss=0.01279, audio_tagging_loss=0.008329, over 15369.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09352, pruned_loss=0.01427, audio_tagging_loss=0.009279, over 3046048.57 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:56:24,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2297506.6666666665, ans=0.125
2023-11-23 07:56:26,813 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:56:52,897 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344650
2023-11-23 07:56:54,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2297640.0, ans=0.125
2023-11-23 07:57:00,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2297640.0, ans=0.125
2023-11-23 07:57:04,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2297706.6666666665, ans=0.125
2023-11-23 07:57:15,710 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8000, loss[loss=0.08637, simple_loss=0.1173, pruned_loss=0.01811, audio_tagging_loss=0.009602, over 14493.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09404, pruned_loss=0.01442, audio_tagging_loss=0.009353, over 3048395.96 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:57:18,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2297773.3333333335, ans=0.125
2023-11-23 07:57:18,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2297773.3333333335, ans=0.1
2023-11-23 07:57:18,876 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.85 vs. limit=6.0
2023-11-23 07:57:20,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2297773.3333333335, ans=0.125
2023-11-23 07:57:41,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2297906.6666666665, ans=0.125
2023-11-23 07:57:42,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2297906.6666666665, ans=0.125
2023-11-23 07:57:56,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344700
2023-11-23 07:58:00,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2297973.3333333335, ans=0.125
2023-11-23 07:58:08,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2298040.0, ans=0.0
2023-11-23 07:58:10,963 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.163e+01 8.824e+01 9.486e+01 1.093e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 07:58:19,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.36 vs. limit=15.0
2023-11-23 07:58:21,504 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8050, loss[loss=0.05652, simple_loss=0.07539, pruned_loss=0.009999, audio_tagging_loss=0.008825, over 14959.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09322, pruned_loss=0.01431, audio_tagging_loss=0.009442, over 3053364.47 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:58:25,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2298106.6666666665, ans=0.025
2023-11-23 07:58:26,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2298106.6666666665, ans=0.0
2023-11-23 07:58:50,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2298240.0, ans=0.125
2023-11-23 07:58:51,663 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:58:52,345 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.05 vs. limit=10.0
2023-11-23 07:58:59,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2298306.6666666665, ans=0.125
2023-11-23 07:59:02,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344750
2023-11-23 07:59:11,722 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.67 vs. limit=15.0
2023-11-23 07:59:25,549 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8100, loss[loss=0.06915, simple_loss=0.09317, pruned_loss=0.01266, audio_tagging_loss=0.009904, over 14900.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09246, pruned_loss=0.01406, audio_tagging_loss=0.009398, over 3051015.69 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:59:28,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2298440.0, ans=0.125
2023-11-23 07:59:37,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.89 vs. limit=12.0
2023-11-23 08:00:07,356 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344800
2023-11-23 08:00:11,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2298640.0, ans=0.2
2023-11-23 08:00:14,270 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.83 vs. limit=10.0
2023-11-23 08:00:21,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2298706.6666666665, ans=0.07
2023-11-23 08:00:22,309 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.280e+01 8.423e+01 8.748e+01 9.445e+01 1.319e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-23 08:00:29,753 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8150, loss[loss=0.06774, simple_loss=0.09343, pruned_loss=0.01252, audio_tagging_loss=0.008502, over 15420.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09248, pruned_loss=0.01411, audio_tagging_loss=0.00928, over 3050994.04 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:00:37,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2298773.3333333335, ans=0.2
2023-11-23 08:00:40,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2298773.3333333335, ans=0.125
2023-11-23 08:00:45,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2298840.0, ans=10.0
2023-11-23 08:01:04,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2298906.6666666665, ans=0.2
2023-11-23 08:01:06,934 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.34 vs. limit=15.0
2023-11-23 08:01:11,339 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344850
2023-11-23 08:01:27,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2299040.0, ans=0.125
2023-11-23 08:01:34,461 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8200, loss[loss=0.07478, simple_loss=0.09505, pruned_loss=0.01769, audio_tagging_loss=0.009568, over 15247.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09231, pruned_loss=0.01407, audio_tagging_loss=0.009142, over 3057544.42 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:01:34,530 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:01:55,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2299173.3333333335, ans=0.0
2023-11-23 08:02:14,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344900
2023-11-23 08:02:24,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2299306.6666666665, ans=0.1
2023-11-23 08:02:28,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2299373.3333333335, ans=0.125
2023-11-23 08:02:31,496 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.892e+01 8.311e+01 8.960e+01 9.864e+01 1.172e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 08:02:38,996 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8250, loss[loss=0.07227, simple_loss=0.09894, pruned_loss=0.01396, audio_tagging_loss=0.008839, over 16241.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09253, pruned_loss=0.01403, audio_tagging_loss=0.009107, over 3058150.07 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:02:54,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2299506.6666666665, ans=0.0
2023-11-23 08:03:03,426 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=12.0
2023-11-23 08:03:07,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2299573.3333333335, ans=0.125
2023-11-23 08:03:21,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 344950
2023-11-23 08:03:28,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2299640.0, ans=0.125
2023-11-23 08:03:29,168 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.90 vs. limit=15.0
2023-11-23 08:03:35,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2299706.6666666665, ans=0.0
2023-11-23 08:03:35,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2299706.6666666665, ans=0.1
2023-11-23 08:03:42,986 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8300, loss[loss=0.07198, simple_loss=0.08938, pruned_loss=0.01629, audio_tagging_loss=0.011, over 14413.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09239, pruned_loss=0.01403, audio_tagging_loss=0.009121, over 3049561.98 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:04:14,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.93 vs. limit=10.0
2023-11-23 08:04:21,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2299973.3333333335, ans=0.0
2023-11-23 08:04:23,897 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345000
2023-11-23 08:04:38,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.704e+01 8.216e+01 8.783e+01 9.423e+01 1.176e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 08:04:46,170 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.35 vs. limit=22.5
2023-11-23 08:04:46,569 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8350, loss[loss=0.06813, simple_loss=0.08464, pruned_loss=0.01686, audio_tagging_loss=0.008954, over 15210.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09376, pruned_loss=0.01427, audio_tagging_loss=0.008957, over 3051783.08 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:04:58,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2300106.6666666665, ans=0.1
2023-11-23 08:05:02,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2300173.3333333335, ans=0.125
2023-11-23 08:05:16,035 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0
2023-11-23 08:05:21,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2300240.0, ans=10.0
2023-11-23 08:05:21,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2300240.0, ans=0.09899494936611666
2023-11-23 08:05:27,875 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345050
2023-11-23 08:05:33,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2300306.6666666665, ans=0.125
2023-11-23 08:05:40,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2300373.3333333335, ans=0.125
2023-11-23 08:05:45,058 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.38 vs. limit=15.0
2023-11-23 08:05:45,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2300373.3333333335, ans=0.0
2023-11-23 08:05:48,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2300373.3333333335, ans=0.125
2023-11-23 08:05:51,643 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8400, loss[loss=0.05678, simple_loss=0.07632, pruned_loss=0.01004, audio_tagging_loss=0.008583, over 15769.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09217, pruned_loss=0.0141, audio_tagging_loss=0.008984, over 3047687.87 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:05:58,136 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:06:00,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2300440.0, ans=0.1
2023-11-23 08:06:08,080 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:06:13,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2300506.6666666665, ans=0.0
2023-11-23 08:06:27,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2300573.3333333335, ans=0.2
2023-11-23 08:06:33,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345100
2023-11-23 08:06:47,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.105e+01 8.014e+01 8.846e+01 9.500e+01 1.669e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 08:06:54,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2300773.3333333335, ans=0.0
2023-11-23 08:06:55,293 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8450, loss[loss=0.05538, simple_loss=0.0647, pruned_loss=0.01119, audio_tagging_loss=0.01184, over 14691.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09287, pruned_loss=0.01412, audio_tagging_loss=0.009018, over 3046977.32 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:06:56,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2300773.3333333335, ans=0.1
2023-11-23 08:07:04,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2300773.3333333335, ans=0.125
2023-11-23 08:07:04,423 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.06 vs. limit=15.0
2023-11-23 08:07:24,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2300906.6666666665, ans=0.1
2023-11-23 08:07:28,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2300906.6666666665, ans=0.0
2023-11-23 08:07:32,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_na.min_abs, batch_count=2300973.3333333335, ans=0.02
2023-11-23 08:07:36,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345150
2023-11-23 08:07:50,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2301040.0, ans=0.125
2023-11-23 08:07:55,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2301040.0, ans=0.125
2023-11-23 08:07:58,686 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8500, loss[loss=0.08138, simple_loss=0.1158, pruned_loss=0.01461, audio_tagging_loss=0.008846, over 16076.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09209, pruned_loss=0.01395, audio_tagging_loss=0.009087, over 3048728.35 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:08:08,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2301106.6666666665, ans=0.125
2023-11-23 08:08:24,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2301240.0, ans=0.125
2023-11-23 08:08:31,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2301240.0, ans=0.2
2023-11-23 08:08:39,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345200
2023-11-23 08:08:51,408 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:08:54,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff3.min_abs, batch_count=2301373.3333333335, ans=0.2
2023-11-23 08:08:55,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.259e+01 8.832e+01 9.401e+01 1.675e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 08:08:58,830 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-23 08:09:03,823 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8550, loss[loss=0.06844, simple_loss=0.08836, pruned_loss=0.01361, audio_tagging_loss=0.01065, over 14499.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.0929, pruned_loss=0.01401, audio_tagging_loss=0.009063, over 3055519.71 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:09:17,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2301506.6666666665, ans=0.1
2023-11-23 08:09:34,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2301573.3333333335, ans=0.0
2023-11-23 08:09:44,921 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345250
2023-11-23 08:09:52,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2301640.0, ans=0.035
2023-11-23 08:10:07,676 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8600, loss[loss=0.05603, simple_loss=0.06573, pruned_loss=0.01152, audio_tagging_loss=0.01165, over 14058.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09309, pruned_loss=0.01407, audio_tagging_loss=0.009024, over 3056235.85 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:10:12,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2301773.3333333335, ans=0.0
2023-11-23 08:10:45,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2301973.3333333335, ans=0.125
2023-11-23 08:10:49,387 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345300
2023-11-23 08:10:50,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2301973.3333333335, ans=0.0
2023-11-23 08:10:50,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2301973.3333333335, ans=0.09899494936611666
2023-11-23 08:10:54,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2301973.3333333335, ans=0.125
2023-11-23 08:11:02,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2302040.0, ans=0.1
2023-11-23 08:11:03,895 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.320e+01 8.873e+01 9.523e+01 1.247e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 08:11:10,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.98 vs. limit=15.0
2023-11-23 08:11:11,300 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8650, loss[loss=0.07344, simple_loss=0.1015, pruned_loss=0.01648, audio_tagging_loss=0.006197, over 15237.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.0931, pruned_loss=0.01401, audio_tagging_loss=0.009121, over 3059078.86 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:11:31,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2302173.3333333335, ans=0.09899494936611666
2023-11-23 08:11:52,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345350
2023-11-23 08:12:16,463 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8700, loss[loss=0.07968, simple_loss=0.1023, pruned_loss=0.01694, audio_tagging_loss=0.01156, over 15717.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.0938, pruned_loss=0.01415, audio_tagging_loss=0.009099, over 3054012.39 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:12:19,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2302440.0, ans=0.125
2023-11-23 08:12:31,199 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.58 vs. limit=12.0
2023-11-23 08:12:40,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2302573.3333333335, ans=0.125
2023-11-23 08:12:40,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.16 vs. limit=12.0
2023-11-23 08:12:45,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2302573.3333333335, ans=0.025
2023-11-23 08:12:57,886 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345400
2023-11-23 08:13:05,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2302640.0, ans=0.1
2023-11-23 08:13:13,353 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.372e+01 9.197e+01 9.763e+01 1.279e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 08:13:18,661 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:13:20,770 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8750, loss[loss=0.07643, simple_loss=0.08942, pruned_loss=0.022, audio_tagging_loss=0.009721, over 14301.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09459, pruned_loss=0.01443, audio_tagging_loss=0.0091, over 3053001.51 frames. ], batch size: 53, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:13:28,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2302773.3333333335, ans=0.0
2023-11-23 08:13:46,153 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.51 vs. limit=15.0
2023-11-23 08:14:01,532 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345450
2023-11-23 08:14:09,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2302973.3333333335, ans=0.125
2023-11-23 08:14:14,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2303040.0, ans=0.1
2023-11-23 08:14:18,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=2303040.0, ans=6.0
2023-11-23 08:14:24,234 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8800, loss[loss=0.06947, simple_loss=0.09433, pruned_loss=0.01209, audio_tagging_loss=0.01022, over 15505.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09422, pruned_loss=0.01436, audio_tagging_loss=0.009242, over 3052897.95 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:14:28,558 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.82 vs. limit=6.0
2023-11-23 08:14:40,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2303173.3333333335, ans=0.0
2023-11-23 08:14:44,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2303173.3333333335, ans=0.125
2023-11-23 08:15:02,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2303306.6666666665, ans=0.0
2023-11-23 08:15:04,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345500
2023-11-23 08:15:12,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2303306.6666666665, ans=0.0
2023-11-23 08:15:20,170 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.339e+01 9.015e+01 9.606e+01 1.190e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 08:15:28,084 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8850, loss[loss=0.08161, simple_loss=0.105, pruned_loss=0.02059, audio_tagging_loss=0.008524, over 14195.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09446, pruned_loss=0.01439, audio_tagging_loss=0.009249, over 3056526.48 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:15:39,129 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:15:51,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2303506.6666666665, ans=0.2
2023-11-23 08:15:53,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2303573.3333333335, ans=0.125
2023-11-23 08:16:08,803 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345550
2023-11-23 08:16:18,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2303706.6666666665, ans=0.1
2023-11-23 08:16:19,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2303706.6666666665, ans=0.125
2023-11-23 08:16:31,258 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8900, loss[loss=0.09809, simple_loss=0.1355, pruned_loss=0.02241, audio_tagging_loss=0.007937, over 15911.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09509, pruned_loss=0.01445, audio_tagging_loss=0.00912, over 3051680.18 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:16:37,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2303773.3333333335, ans=0.125
2023-11-23 08:16:47,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2303840.0, ans=0.125
2023-11-23 08:17:09,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2303973.3333333335, ans=0.0
2023-11-23 08:17:11,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345600
2023-11-23 08:17:14,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2303973.3333333335, ans=0.05
2023-11-23 08:17:16,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2303973.3333333335, ans=0.07
2023-11-23 08:17:27,493 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.364e+01 9.019e+01 9.831e+01 1.164e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 08:17:31,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2304040.0, ans=0.125
2023-11-23 08:17:32,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2304040.0, ans=0.1
2023-11-23 08:17:34,817 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 8950, loss[loss=0.06444, simple_loss=0.09395, pruned_loss=0.01288, audio_tagging_loss=0.004589, over 15948.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09528, pruned_loss=0.01458, audio_tagging_loss=0.00896, over 3050043.82 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:18:11,480 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.29 vs. limit=22.5
2023-11-23 08:18:16,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345650
2023-11-23 08:18:40,006 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9000, loss[loss=0.05825, simple_loss=0.08456, pruned_loss=0.007525, audio_tagging_loss=0.008446, over 16069.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09407, pruned_loss=0.01428, audio_tagging_loss=0.008963, over 3049190.21 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:18:40,007 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 08:19:12,604 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.1257, 2.4552, 5.0612, 2.8773], device='cuda:2')
2023-11-23 08:19:23,271 INFO [train_asr.py:1253] (2/4) Epoch 29, validation: loss=0.05895, simple_loss=0.05118, pruned_loss=0.005121, audio_tagging_loss=0.02824, over 4681554.00 frames. 
2023-11-23 08:19:23,272 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 08:19:26,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2304440.0, ans=0.125
2023-11-23 08:19:38,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2304506.6666666665, ans=0.0
2023-11-23 08:19:52,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2304573.3333333335, ans=0.0
2023-11-23 08:20:04,379 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345700
2023-11-23 08:20:19,507 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.885e+01 8.225e+01 9.124e+01 9.733e+01 1.162e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 08:20:27,046 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9050, loss[loss=0.07444, simple_loss=0.09598, pruned_loss=0.01542, audio_tagging_loss=0.01102, over 14843.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09352, pruned_loss=0.01405, audio_tagging_loss=0.008967, over 3043385.49 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:20:48,298 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:20:53,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2304906.6666666665, ans=0.1
2023-11-23 08:21:08,202 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345750
2023-11-23 08:21:31,409 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9100, loss[loss=0.08449, simple_loss=0.1157, pruned_loss=0.01856, audio_tagging_loss=0.008088, over 15360.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.0933, pruned_loss=0.0139, audio_tagging_loss=0.008867, over 3043179.43 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:21:51,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2305173.3333333335, ans=0.1
2023-11-23 08:21:57,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2305240.0, ans=0.025
2023-11-23 08:22:12,301 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345800
2023-11-23 08:22:13,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2305306.6666666665, ans=10.0
2023-11-23 08:22:28,183 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.178e+01 8.924e+01 9.840e+01 1.213e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 08:22:34,158 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9150, loss[loss=0.05168, simple_loss=0.07537, pruned_loss=0.006098, audio_tagging_loss=0.007899, over 16588.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09332, pruned_loss=0.01396, audio_tagging_loss=0.008894, over 3049010.68 frames. ], batch size: 63, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:22:53,804 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.57 vs. limit=15.0
2023-11-23 08:23:10,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2305573.3333333335, ans=0.1
2023-11-23 08:23:15,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345850
2023-11-23 08:23:25,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.42 vs. limit=22.5
2023-11-23 08:23:31,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2305706.6666666665, ans=0.0
2023-11-23 08:23:37,726 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9200, loss[loss=0.08601, simple_loss=0.1178, pruned_loss=0.02066, audio_tagging_loss=0.006427, over 15409.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09384, pruned_loss=0.01423, audio_tagging_loss=0.008866, over 3047809.95 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:23:46,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2305773.3333333335, ans=0.04949747468305833
2023-11-23 08:24:00,426 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:24:04,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=2305906.6666666665, ans=0.02
2023-11-23 08:24:11,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff2.min_abs, batch_count=2305906.6666666665, ans=0.1
2023-11-23 08:24:18,701 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345900
2023-11-23 08:24:22,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2305973.3333333335, ans=0.0
2023-11-23 08:24:35,642 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.303e+01 8.816e+01 9.419e+01 1.160e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 08:24:40,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2306040.0, ans=0.125
2023-11-23 08:24:42,459 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9250, loss[loss=0.05779, simple_loss=0.08079, pruned_loss=0.009812, audio_tagging_loss=0.007583, over 15620.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09349, pruned_loss=0.01415, audio_tagging_loss=0.008937, over 3047543.98 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:24:52,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0
2023-11-23 08:25:18,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2306306.6666666665, ans=0.0
2023-11-23 08:25:19,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2306306.6666666665, ans=0.0
2023-11-23 08:25:22,701 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 345950
2023-11-23 08:25:28,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2306306.6666666665, ans=0.2
2023-11-23 08:25:41,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2306373.3333333335, ans=0.0
2023-11-23 08:25:44,891 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9300, loss[loss=0.08072, simple_loss=0.106, pruned_loss=0.01707, audio_tagging_loss=0.01065, over 15326.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09338, pruned_loss=0.01427, audio_tagging_loss=0.008916, over 3047270.30 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:25:45,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2306440.0, ans=0.125
2023-11-23 08:25:49,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.76 vs. limit=6.0
2023-11-23 08:25:57,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2306506.6666666665, ans=0.2
2023-11-23 08:26:01,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0
2023-11-23 08:26:17,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2306573.3333333335, ans=0.125
2023-11-23 08:26:25,813 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346000
2023-11-23 08:26:34,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2306706.6666666665, ans=0.125
2023-11-23 08:26:36,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2306706.6666666665, ans=0.0
2023-11-23 08:26:36,179 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:26:41,830 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.507e+01 8.976e+01 9.625e+01 1.274e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 08:26:46,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2306773.3333333335, ans=0.1
2023-11-23 08:26:48,010 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9350, loss[loss=0.05321, simple_loss=0.07029, pruned_loss=0.008801, audio_tagging_loss=0.009269, over 14380.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.0933, pruned_loss=0.01426, audio_tagging_loss=0.008935, over 3052985.22 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:26:51,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0
2023-11-23 08:26:52,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2306773.3333333335, ans=0.125
2023-11-23 08:27:23,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2306906.6666666665, ans=0.1
2023-11-23 08:27:29,176 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346050
2023-11-23 08:27:31,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2306973.3333333335, ans=0.125
2023-11-23 08:27:39,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2307040.0, ans=0.1
2023-11-23 08:27:40,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2307040.0, ans=0.125
2023-11-23 08:27:44,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2307040.0, ans=0.1
2023-11-23 08:27:44,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2307040.0, ans=0.0
2023-11-23 08:27:53,088 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9400, loss[loss=0.07254, simple_loss=0.09789, pruned_loss=0.01222, audio_tagging_loss=0.01137, over 15514.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.0929, pruned_loss=0.01432, audio_tagging_loss=0.00908, over 3048632.38 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:27:57,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2307106.6666666665, ans=0.125
2023-11-23 08:28:02,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2307106.6666666665, ans=0.125
2023-11-23 08:28:15,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2307173.3333333335, ans=0.07
2023-11-23 08:28:27,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2307240.0, ans=0.1
2023-11-23 08:28:29,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2307306.6666666665, ans=0.0
2023-11-23 08:28:29,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.13 vs. limit=15.0
2023-11-23 08:28:30,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.52 vs. limit=22.5
2023-11-23 08:28:33,034 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346100
2023-11-23 08:28:51,278 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.336e+01 8.834e+01 9.625e+01 1.227e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 08:28:53,804 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:28:56,223 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9450, loss[loss=0.06801, simple_loss=0.08448, pruned_loss=0.01142, audio_tagging_loss=0.01434, over 15034.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.092, pruned_loss=0.01399, audio_tagging_loss=0.009297, over 3046162.86 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:28:58,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.85 vs. limit=10.0
2023-11-23 08:29:02,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2307440.0, ans=0.1
2023-11-23 08:29:30,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2307573.3333333335, ans=0.125
2023-11-23 08:29:36,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346150
2023-11-23 08:29:39,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2307640.0, ans=0.125
2023-11-23 08:29:50,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2307706.6666666665, ans=0.04949747468305833
2023-11-23 08:29:58,746 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9500, loss[loss=0.08263, simple_loss=0.1178, pruned_loss=0.01527, audio_tagging_loss=0.008451, over 15550.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09155, pruned_loss=0.01403, audio_tagging_loss=0.009403, over 3043747.46 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:30:06,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2307773.3333333335, ans=0.125
2023-11-23 08:30:10,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2307840.0, ans=0.125
2023-11-23 08:30:32,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2307906.6666666665, ans=0.04949747468305833
2023-11-23 08:30:39,579 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346200
2023-11-23 08:30:44,238 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.16 vs. limit=12.0
2023-11-23 08:30:56,980 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.410e+01 8.909e+01 9.826e+01 1.512e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 08:31:03,154 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9550, loss[loss=0.08328, simple_loss=0.1129, pruned_loss=0.01577, audio_tagging_loss=0.01106, over 14252.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.0922, pruned_loss=0.01406, audio_tagging_loss=0.009385, over 3040162.00 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:31:43,439 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346250
2023-11-23 08:32:05,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2308373.3333333335, ans=0.2
2023-11-23 08:32:07,623 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9600, loss[loss=0.05531, simple_loss=0.07099, pruned_loss=0.01036, audio_tagging_loss=0.009458, over 15772.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09185, pruned_loss=0.01393, audio_tagging_loss=0.009427, over 3038494.81 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:32:09,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2308440.0, ans=0.125
2023-11-23 08:32:12,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2308440.0, ans=0.125
2023-11-23 08:32:24,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2308506.6666666665, ans=0.125
2023-11-23 08:32:49,811 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346300
2023-11-23 08:32:54,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2308640.0, ans=0.05
2023-11-23 08:33:02,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2308706.6666666665, ans=0.125
2023-11-23 08:33:06,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2308706.6666666665, ans=0.125
2023-11-23 08:33:06,931 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.195e+01 8.302e+01 9.179e+01 9.827e+01 1.321e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-23 08:33:11,996 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9650, loss[loss=0.05635, simple_loss=0.07378, pruned_loss=0.008832, audio_tagging_loss=0.01062, over 15212.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09073, pruned_loss=0.01363, audio_tagging_loss=0.009417, over 3039311.48 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:33:15,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2308773.3333333335, ans=0.125
2023-11-23 08:33:18,879 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.60 vs. limit=15.0
2023-11-23 08:33:27,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2308840.0, ans=0.125
2023-11-23 08:33:34,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2308840.0, ans=0.125
2023-11-23 08:33:38,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2308906.6666666665, ans=15.0
2023-11-23 08:33:53,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346350
2023-11-23 08:33:56,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2308973.3333333335, ans=0.125
2023-11-23 08:33:58,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2308973.3333333335, ans=0.1
2023-11-23 08:34:15,592 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9700, loss[loss=0.07774, simple_loss=0.1092, pruned_loss=0.01402, audio_tagging_loss=0.009127, over 14688.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09114, pruned_loss=0.01367, audio_tagging_loss=0.009224, over 3037759.61 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:34:26,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2309106.6666666665, ans=0.125
2023-11-23 08:34:26,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309106.6666666665, ans=0.1
2023-11-23 08:34:32,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309173.3333333335, ans=0.1
2023-11-23 08:34:33,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2309173.3333333335, ans=0.0
2023-11-23 08:34:42,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2309240.0, ans=0.125
2023-11-23 08:34:46,439 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.36 vs. limit=15.0
2023-11-23 08:34:48,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.23 vs. limit=15.0
2023-11-23 08:34:54,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2309306.6666666665, ans=0.0
2023-11-23 08:34:58,267 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346400
2023-11-23 08:34:58,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2309306.6666666665, ans=0.125
2023-11-23 08:35:16,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2309373.3333333335, ans=0.09899494936611666
2023-11-23 08:35:19,604 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.905e+01 8.371e+01 8.992e+01 9.698e+01 1.313e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 08:35:23,345 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9750, loss[loss=0.06447, simple_loss=0.08365, pruned_loss=0.01482, audio_tagging_loss=0.007823, over 14676.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.0913, pruned_loss=0.01382, audio_tagging_loss=0.009081, over 3040144.84 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:35:24,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2309440.0, ans=0.2
2023-11-23 08:35:33,758 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.09 vs. limit=10.0
2023-11-23 08:35:45,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2309506.6666666665, ans=0.04949747468305833
2023-11-23 08:35:48,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-23 08:36:04,817 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346450
2023-11-23 08:36:25,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2309706.6666666665, ans=0.125
2023-11-23 08:36:27,482 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9800, loss[loss=0.06764, simple_loss=0.08912, pruned_loss=0.01321, audio_tagging_loss=0.00987, over 14635.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09252, pruned_loss=0.01405, audio_tagging_loss=0.008938, over 3046157.53 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:36:29,331 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=15.0
2023-11-23 08:36:30,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309773.3333333335, ans=0.1
2023-11-23 08:36:32,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2309773.3333333335, ans=0.0
2023-11-23 08:36:54,576 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.66 vs. limit=22.5
2023-11-23 08:37:04,896 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0
2023-11-23 08:37:09,359 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346500
2023-11-23 08:37:11,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2309973.3333333335, ans=0.125
2023-11-23 08:37:15,666 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:37:19,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2310040.0, ans=15.0
2023-11-23 08:37:23,978 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:37:27,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.413e+01 9.129e+01 9.676e+01 1.290e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 08:37:28,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=22.5
2023-11-23 08:37:31,277 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9850, loss[loss=0.07083, simple_loss=0.1035, pruned_loss=0.01279, audio_tagging_loss=0.00627, over 15349.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09217, pruned_loss=0.01392, audio_tagging_loss=0.009043, over 3046800.33 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:37:33,143 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.82 vs. limit=15.0
2023-11-23 08:37:38,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=12.0
2023-11-23 08:37:41,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2310106.6666666665, ans=0.0
2023-11-23 08:38:06,282 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.32 vs. limit=12.0
2023-11-23 08:38:12,920 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346550
2023-11-23 08:38:36,335 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9900, loss[loss=0.05577, simple_loss=0.07034, pruned_loss=0.009361, audio_tagging_loss=0.01124, over 15223.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09239, pruned_loss=0.01405, audio_tagging_loss=0.009007, over 3043238.91 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:39:17,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346600
2023-11-23 08:39:36,795 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.313e+01 8.245e+01 9.085e+01 9.612e+01 1.420e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 08:39:37,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2310706.6666666665, ans=0.2
2023-11-23 08:39:40,503 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 9950, loss[loss=0.04828, simple_loss=0.06033, pruned_loss=0.006092, audio_tagging_loss=0.01203, over 17119.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09182, pruned_loss=0.0139, audio_tagging_loss=0.008976, over 3048437.84 frames. ], batch size: 65, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:39:43,849 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.82 vs. limit=15.0
2023-11-23 08:40:13,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2310906.6666666665, ans=0.0
2023-11-23 08:40:22,027 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346650
2023-11-23 08:40:39,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2311040.0, ans=0.2
2023-11-23 08:40:43,948 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10000, loss[loss=0.1043, simple_loss=0.1429, pruned_loss=0.02696, audio_tagging_loss=0.005866, over 15257.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09218, pruned_loss=0.01405, audio_tagging_loss=0.008842, over 3054964.97 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:40:55,589 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.50 vs. limit=15.0
2023-11-23 08:41:01,319 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=15.0
2023-11-23 08:41:13,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2311240.0, ans=0.0
2023-11-23 08:41:24,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346700
2023-11-23 08:41:24,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2311306.6666666665, ans=0.1
2023-11-23 08:41:41,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2311373.3333333335, ans=0.125
2023-11-23 08:41:44,038 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.283e+01 8.893e+01 9.595e+01 1.369e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 08:41:47,684 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10050, loss[loss=0.09047, simple_loss=0.1183, pruned_loss=0.01865, audio_tagging_loss=0.01265, over 15511.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09241, pruned_loss=0.01395, audio_tagging_loss=0.008935, over 3051868.59 frames. ], batch size: 54, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:41:50,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2311440.0, ans=0.2
2023-11-23 08:42:04,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2311506.6666666665, ans=0.0
2023-11-23 08:42:19,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.05 vs. limit=6.0
2023-11-23 08:42:27,027 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.86 vs. limit=10.0
2023-11-23 08:42:29,085 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346750
2023-11-23 08:42:30,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2311640.0, ans=0.125
2023-11-23 08:42:30,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2311640.0, ans=0.125
2023-11-23 08:42:51,628 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10100, loss[loss=0.0532, simple_loss=0.06201, pruned_loss=0.01262, audio_tagging_loss=0.009575, over 15232.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09242, pruned_loss=0.014, audio_tagging_loss=0.009079, over 3054614.97 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:42:59,790 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=12.0
2023-11-23 08:43:03,009 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:43:04,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2311840.0, ans=0.2
2023-11-23 08:43:06,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2311840.0, ans=0.125
2023-11-23 08:43:24,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2311906.6666666665, ans=0.125
2023-11-23 08:43:28,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2311973.3333333335, ans=0.1
2023-11-23 08:43:32,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346800
2023-11-23 08:43:43,422 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:43:53,037 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.400e+01 8.955e+01 9.679e+01 1.203e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 08:43:55,584 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10150, loss[loss=0.09518, simple_loss=0.1293, pruned_loss=0.02123, audio_tagging_loss=0.009299, over 16220.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09329, pruned_loss=0.01422, audio_tagging_loss=0.009142, over 3059904.80 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:43:57,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2312106.6666666665, ans=0.125
2023-11-23 08:44:00,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2312106.6666666665, ans=0.125
2023-11-23 08:44:19,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.67 vs. limit=22.5
2023-11-23 08:44:25,131 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:44:26,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2312240.0, ans=0.0
2023-11-23 08:44:36,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346850
2023-11-23 08:44:42,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2312306.6666666665, ans=0.0
2023-11-23 08:44:43,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2312306.6666666665, ans=0.125
2023-11-23 08:44:45,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2312373.3333333335, ans=0.1
2023-11-23 08:45:00,224 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10200, loss[loss=0.07706, simple_loss=0.1016, pruned_loss=0.01857, audio_tagging_loss=0.007677, over 15734.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09318, pruned_loss=0.01417, audio_tagging_loss=0.009162, over 3057820.18 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:45:23,122 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:45:34,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2312573.3333333335, ans=0.125
2023-11-23 08:45:40,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346900
2023-11-23 08:45:52,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2312706.6666666665, ans=0.0
2023-11-23 08:45:55,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=15.0
2023-11-23 08:45:57,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2312706.6666666665, ans=0.0
2023-11-23 08:45:58,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2312706.6666666665, ans=0.0
2023-11-23 08:46:01,771 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.340e+01 8.896e+01 9.645e+01 1.207e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 08:46:04,274 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10250, loss[loss=0.07532, simple_loss=0.09979, pruned_loss=0.01358, audio_tagging_loss=0.01185, over 15555.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.0934, pruned_loss=0.01424, audio_tagging_loss=0.009234, over 3055974.64 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:46:07,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2312773.3333333335, ans=0.125
2023-11-23 08:46:09,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2312773.3333333335, ans=0.125
2023-11-23 08:46:12,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2312773.3333333335, ans=0.0
2023-11-23 08:46:16,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2312840.0, ans=0.125
2023-11-23 08:46:45,691 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 346950
2023-11-23 08:47:08,013 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10300, loss[loss=0.07453, simple_loss=0.09706, pruned_loss=0.01522, audio_tagging_loss=0.01078, over 16404.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09336, pruned_loss=0.0143, audio_tagging_loss=0.009336, over 3054683.58 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:47:25,475 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.07 vs. limit=15.0
2023-11-23 08:47:37,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2313240.0, ans=0.125
2023-11-23 08:47:48,577 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347000
2023-11-23 08:47:50,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2313306.6666666665, ans=0.125
2023-11-23 08:48:10,245 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.826e+01 8.491e+01 9.106e+01 9.880e+01 1.580e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-23 08:48:12,722 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10350, loss[loss=0.05134, simple_loss=0.06422, pruned_loss=0.006609, audio_tagging_loss=0.01262, over 15205.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.0922, pruned_loss=0.01407, audio_tagging_loss=0.009462, over 3055377.01 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:48:35,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2313506.6666666665, ans=0.2
2023-11-23 08:48:49,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2313640.0, ans=0.1
2023-11-23 08:48:52,772 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347050
2023-11-23 08:48:53,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2313640.0, ans=0.125
2023-11-23 08:48:56,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2313640.0, ans=0.125
2023-11-23 08:49:16,478 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10400, loss[loss=0.09004, simple_loss=0.121, pruned_loss=0.02138, audio_tagging_loss=0.008169, over 14654.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09202, pruned_loss=0.01405, audio_tagging_loss=0.009493, over 3048592.15 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:49:18,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.59 vs. limit=15.0
2023-11-23 08:49:27,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2313840.0, ans=0.1
2023-11-23 08:49:27,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2313840.0, ans=0.2
2023-11-23 08:49:29,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2313840.0, ans=0.125
2023-11-23 08:49:52,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2313906.6666666665, ans=0.035
2023-11-23 08:49:58,392 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347100
2023-11-23 08:50:04,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.27 vs. limit=10.0
2023-11-23 08:50:10,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2314040.0, ans=0.2
2023-11-23 08:50:18,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2314040.0, ans=0.125
2023-11-23 08:50:18,923 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.096e+01 8.361e+01 8.771e+01 9.632e+01 1.204e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-23 08:50:20,799 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10450, loss[loss=0.04155, simple_loss=0.04628, pruned_loss=0.005562, audio_tagging_loss=0.01285, over 15149.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.0918, pruned_loss=0.01408, audio_tagging_loss=0.009504, over 3044661.44 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:50:27,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2314106.6666666665, ans=0.125
2023-11-23 08:50:31,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2314173.3333333335, ans=0.125
2023-11-23 08:50:34,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2314173.3333333335, ans=0.125
2023-11-23 08:50:52,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2314240.0, ans=10.0
2023-11-23 08:50:52,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2314240.0, ans=0.125
2023-11-23 08:50:54,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.63 vs. limit=15.0
2023-11-23 08:51:02,146 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347150
2023-11-23 08:51:09,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2314306.6666666665, ans=0.125
2023-11-23 08:51:11,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2314373.3333333335, ans=0.125
2023-11-23 08:51:17,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2314373.3333333335, ans=0.04949747468305833
2023-11-23 08:51:21,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2314373.3333333335, ans=0.1
2023-11-23 08:51:25,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.80 vs. limit=10.0
2023-11-23 08:51:26,483 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10500, loss[loss=0.06806, simple_loss=0.09147, pruned_loss=0.0141, audio_tagging_loss=0.008225, over 14272.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09189, pruned_loss=0.01403, audio_tagging_loss=0.009388, over 3044409.02 frames. ], batch size: 54, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:51:27,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2314440.0, ans=0.125
2023-11-23 08:52:06,411 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347200
2023-11-23 08:52:21,943 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:52:29,015 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.754e+01 8.399e+01 8.762e+01 9.493e+01 1.186e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 08:52:30,289 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10550, loss[loss=0.07445, simple_loss=0.09135, pruned_loss=0.01726, audio_tagging_loss=0.01151, over 14910.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09136, pruned_loss=0.01391, audio_tagging_loss=0.009226, over 3048954.61 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:52:39,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2314773.3333333335, ans=0.125
2023-11-23 08:52:45,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2314840.0, ans=0.125
2023-11-23 08:53:07,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2314906.6666666665, ans=0.5
2023-11-23 08:53:12,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347250
2023-11-23 08:53:13,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2314973.3333333335, ans=0.125
2023-11-23 08:53:15,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2314973.3333333335, ans=0.125
2023-11-23 08:53:21,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2315040.0, ans=0.0
2023-11-23 08:53:25,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2315040.0, ans=0.0
2023-11-23 08:53:30,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2315040.0, ans=0.0
2023-11-23 08:53:31,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2315040.0, ans=0.2
2023-11-23 08:53:33,964 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10600, loss[loss=0.0642, simple_loss=0.08604, pruned_loss=0.01224, audio_tagging_loss=0.008937, over 13904.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09164, pruned_loss=0.01383, audio_tagging_loss=0.009059, over 3044473.32 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:53:58,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2315173.3333333335, ans=0.0
2023-11-23 08:53:58,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2315173.3333333335, ans=0.125
2023-11-23 08:54:03,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2315240.0, ans=0.0
2023-11-23 08:54:05,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2315240.0, ans=0.0
2023-11-23 08:54:09,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2315240.0, ans=0.04949747468305833
2023-11-23 08:54:15,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347300
2023-11-23 08:54:35,318 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.59 vs. limit=15.0
2023-11-23 08:54:37,203 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.247e+01 8.943e+01 9.697e+01 1.268e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 08:54:39,144 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10650, loss[loss=0.0695, simple_loss=0.09384, pruned_loss=0.01316, audio_tagging_loss=0.009415, over 14345.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.0911, pruned_loss=0.01387, audio_tagging_loss=0.009118, over 3050505.31 frames. ], batch size: 54, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:54:59,355 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.89 vs. limit=15.0
2023-11-23 08:55:04,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2315573.3333333335, ans=0.125
2023-11-23 08:55:11,716 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.21 vs. limit=15.0
2023-11-23 08:55:19,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347350
2023-11-23 08:55:30,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2315706.6666666665, ans=0.2
2023-11-23 08:55:40,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2315706.6666666665, ans=0.1
2023-11-23 08:55:42,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.60 vs. limit=15.0
2023-11-23 08:55:44,091 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10700, loss[loss=0.09682, simple_loss=0.1355, pruned_loss=0.02105, audio_tagging_loss=0.007999, over 15567.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09141, pruned_loss=0.01394, audio_tagging_loss=0.009135, over 3050410.89 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:56:01,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2315840.0, ans=0.125
2023-11-23 08:56:06,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2315840.0, ans=0.125
2023-11-23 08:56:25,724 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347400
2023-11-23 08:56:30,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2315973.3333333335, ans=0.015
2023-11-23 08:56:36,338 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.81 vs. limit=10.0
2023-11-23 08:56:46,529 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.151e+01 8.772e+01 9.535e+01 1.207e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-23 08:56:47,776 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10750, loss[loss=0.0642, simple_loss=0.07839, pruned_loss=0.0144, audio_tagging_loss=0.0106, over 15072.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09223, pruned_loss=0.0141, audio_tagging_loss=0.009048, over 3052321.39 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:56:58,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2316173.3333333335, ans=0.125
2023-11-23 08:57:00,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2316173.3333333335, ans=0.07
2023-11-23 08:57:02,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2316173.3333333335, ans=0.125
2023-11-23 08:57:04,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2316173.3333333335, ans=0.0
2023-11-23 08:57:11,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2316173.3333333335, ans=0.125
2023-11-23 08:57:28,751 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347450
2023-11-23 08:57:30,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2316306.6666666665, ans=0.125
2023-11-23 08:57:50,634 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10800, loss[loss=0.07431, simple_loss=0.09645, pruned_loss=0.01513, audio_tagging_loss=0.01095, over 15127.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09221, pruned_loss=0.01404, audio_tagging_loss=0.009058, over 3046919.63 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:57:50,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2316440.0, ans=0.1
2023-11-23 08:57:52,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2316440.0, ans=0.125
2023-11-23 08:58:01,247 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:58:31,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347500
2023-11-23 08:58:43,282 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:58:55,089 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.196e+01 8.848e+01 9.514e+01 1.233e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 08:58:56,333 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10850, loss[loss=0.06782, simple_loss=0.09814, pruned_loss=0.01103, audio_tagging_loss=0.007721, over 15964.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09344, pruned_loss=0.01422, audio_tagging_loss=0.009028, over 3050714.87 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:59:01,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2316773.3333333335, ans=0.125
2023-11-23 08:59:11,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2316840.0, ans=0.2
2023-11-23 08:59:11,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2316840.0, ans=0.1
2023-11-23 08:59:25,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2316906.6666666665, ans=0.125
2023-11-23 08:59:38,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347550
2023-11-23 08:59:44,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2316973.3333333335, ans=0.2
2023-11-23 08:59:46,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2316973.3333333335, ans=0.125
2023-11-23 08:59:57,024 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:59:57,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2317040.0, ans=0.125
2023-11-23 09:00:00,730 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10900, loss[loss=0.08345, simple_loss=0.1114, pruned_loss=0.01813, audio_tagging_loss=0.009621, over 14599.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09319, pruned_loss=0.01424, audio_tagging_loss=0.009053, over 3040065.68 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:00:19,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2317173.3333333335, ans=0.09899494936611666
2023-11-23 09:00:25,190 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:00:43,171 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347600
2023-11-23 09:00:58,440 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:01:04,201 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.420e+01 8.915e+01 9.673e+01 1.270e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 09:01:05,503 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 10950, loss[loss=0.08551, simple_loss=0.125, pruned_loss=0.01687, audio_tagging_loss=0.006137, over 16101.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09334, pruned_loss=0.01414, audio_tagging_loss=0.008981, over 3041659.08 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:01:21,862 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:01:27,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2317506.6666666665, ans=0.125
2023-11-23 09:01:30,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2317506.6666666665, ans=0.125
2023-11-23 09:01:46,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2317640.0, ans=0.1
2023-11-23 09:01:47,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347650
2023-11-23 09:01:55,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.13 vs. limit=10.0
2023-11-23 09:02:11,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2317773.3333333335, ans=0.0
2023-11-23 09:02:12,001 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11000, loss[loss=0.06845, simple_loss=0.08452, pruned_loss=0.01523, audio_tagging_loss=0.01096, over 14620.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09374, pruned_loss=0.0141, audio_tagging_loss=0.009082, over 3044126.94 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:02:21,746 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:02:32,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2317840.0, ans=0.0
2023-11-23 09:02:40,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2317906.6666666665, ans=0.0
2023-11-23 09:02:47,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2317973.3333333335, ans=0.2
2023-11-23 09:02:53,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347700
2023-11-23 09:03:05,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2318040.0, ans=0.0
2023-11-23 09:03:15,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2318106.6666666665, ans=0.025
2023-11-23 09:03:16,062 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.585e+01 9.370e+01 1.001e+02 1.222e+02, threshold=1.874e+02, percent-clipped=0.0
2023-11-23 09:03:16,108 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11050, loss[loss=0.05963, simple_loss=0.07609, pruned_loss=0.01218, audio_tagging_loss=0.009401, over 15196.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09481, pruned_loss=0.01433, audio_tagging_loss=0.009126, over 3044213.72 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:03:23,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2318106.6666666665, ans=0.125
2023-11-23 09:03:57,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347750
2023-11-23 09:03:58,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2318306.6666666665, ans=0.025
2023-11-23 09:04:04,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2318306.6666666665, ans=0.125
2023-11-23 09:04:04,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2318306.6666666665, ans=0.125
2023-11-23 09:04:14,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2318373.3333333335, ans=0.0
2023-11-23 09:04:19,448 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11100, loss[loss=0.06467, simple_loss=0.07873, pruned_loss=0.01453, audio_tagging_loss=0.01078, over 13918.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09383, pruned_loss=0.01431, audio_tagging_loss=0.009235, over 3042425.24 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:04:23,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2318440.0, ans=0.04949747468305833
2023-11-23 09:04:45,299 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:04:47,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2318573.3333333335, ans=0.2
2023-11-23 09:05:01,246 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347800
2023-11-23 09:05:01,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2318640.0, ans=0.0
2023-11-23 09:05:11,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2318706.6666666665, ans=0.1
2023-11-23 09:05:17,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2318706.6666666665, ans=0.0
2023-11-23 09:05:25,374 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.253e+01 8.535e+01 9.196e+01 1.008e+02 1.260e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 09:05:25,436 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11150, loss[loss=0.05736, simple_loss=0.06781, pruned_loss=0.01061, audio_tagging_loss=0.01284, over 15079.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09364, pruned_loss=0.01424, audio_tagging_loss=0.009304, over 3047447.54 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:05:34,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2318773.3333333335, ans=0.125
2023-11-23 09:05:37,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2318840.0, ans=0.1
2023-11-23 09:05:56,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2318906.6666666665, ans=0.035
2023-11-23 09:05:56,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2318906.6666666665, ans=0.09899494936611666
2023-11-23 09:06:06,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347850
2023-11-23 09:06:06,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2318973.3333333335, ans=0.0
2023-11-23 09:06:20,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2319040.0, ans=0.125
2023-11-23 09:06:29,285 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11200, loss[loss=0.06045, simple_loss=0.08416, pruned_loss=0.01, audio_tagging_loss=0.008373, over 15589.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09349, pruned_loss=0.01426, audio_tagging_loss=0.00934, over 3045999.26 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:06:34,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2319106.6666666665, ans=0.1
2023-11-23 09:06:47,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2319173.3333333335, ans=0.125
2023-11-23 09:06:55,652 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.50 vs. limit=5.0
2023-11-23 09:07:05,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2319240.0, ans=0.125
2023-11-23 09:07:09,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347900
2023-11-23 09:07:18,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2319306.6666666665, ans=0.125
2023-11-23 09:07:19,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2319373.3333333335, ans=0.125
2023-11-23 09:07:32,463 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.786e+01 8.283e+01 8.930e+01 9.840e+01 1.502e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 09:07:32,510 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11250, loss[loss=0.07735, simple_loss=0.107, pruned_loss=0.01791, audio_tagging_loss=0.005931, over 15576.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.0932, pruned_loss=0.0142, audio_tagging_loss=0.009314, over 3055572.18 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:07:51,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2319506.6666666665, ans=0.0
2023-11-23 09:08:06,260 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=15.0
2023-11-23 09:08:13,090 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 347950
2023-11-23 09:08:22,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2319706.6666666665, ans=0.125
2023-11-23 09:08:32,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2319706.6666666665, ans=0.0
2023-11-23 09:08:36,522 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11300, loss[loss=0.07538, simple_loss=0.1002, pruned_loss=0.01692, audio_tagging_loss=0.008372, over 16302.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09316, pruned_loss=0.01431, audio_tagging_loss=0.009147, over 3051987.06 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:08:39,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2319773.3333333335, ans=0.0
2023-11-23 09:08:44,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2319773.3333333335, ans=0.1
2023-11-23 09:08:54,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2319840.0, ans=0.125
2023-11-23 09:09:14,312 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.00 vs. limit=10.0
2023-11-23 09:09:16,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348000
2023-11-23 09:09:38,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2320040.0, ans=0.0
2023-11-23 09:09:42,687 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11350, loss[loss=0.06058, simple_loss=0.0727, pruned_loss=0.0118, audio_tagging_loss=0.01243, over 15600.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09285, pruned_loss=0.01424, audio_tagging_loss=0.009105, over 3051989.36 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:09:43,921 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.379e+01 8.172e+01 9.052e+01 9.730e+01 1.154e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 09:09:44,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2320106.6666666665, ans=0.0
2023-11-23 09:09:53,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2320173.3333333335, ans=0.125
2023-11-23 09:10:22,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2320306.6666666665, ans=0.0
2023-11-23 09:10:23,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348050
2023-11-23 09:10:45,526 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11400, loss[loss=0.06196, simple_loss=0.09192, pruned_loss=0.0102, audio_tagging_loss=0.0058, over 15593.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09273, pruned_loss=0.0142, audio_tagging_loss=0.009085, over 3051483.44 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:10:50,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2320440.0, ans=0.125
2023-11-23 09:10:54,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2320440.0, ans=0.125
2023-11-23 09:11:08,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2320506.6666666665, ans=0.0
2023-11-23 09:11:15,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2320573.3333333335, ans=0.125
2023-11-23 09:11:16,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2320573.3333333335, ans=0.2
2023-11-23 09:11:26,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348100
2023-11-23 09:11:27,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2320640.0, ans=15.0
2023-11-23 09:11:30,499 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:11:41,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2320706.6666666665, ans=0.125
2023-11-23 09:11:49,204 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11450, loss[loss=0.05186, simple_loss=0.07011, pruned_loss=0.009231, audio_tagging_loss=0.00757, over 14405.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09229, pruned_loss=0.01412, audio_tagging_loss=0.009105, over 3050279.90 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 8.0
2023-11-23 09:11:52,261 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.268e+01 8.173e+01 8.697e+01 9.540e+01 1.271e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-23 09:11:56,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2320773.3333333335, ans=0.125
2023-11-23 09:11:56,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2320773.3333333335, ans=0.1
2023-11-23 09:12:04,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2320840.0, ans=0.125
2023-11-23 09:12:23,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2320906.6666666665, ans=0.125
2023-11-23 09:12:29,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348150
2023-11-23 09:12:44,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2321040.0, ans=0.125
2023-11-23 09:12:52,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.45 vs. limit=15.0
2023-11-23 09:12:52,879 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11500, loss[loss=0.08365, simple_loss=0.1063, pruned_loss=0.02037, audio_tagging_loss=0.01011, over 17432.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09267, pruned_loss=0.01419, audio_tagging_loss=0.009132, over 3049675.37 frames. ], batch size: 64, lr: 2.33e-03, grad_scale: 8.0
2023-11-23 09:12:57,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2321106.6666666665, ans=0.125
2023-11-23 09:12:58,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2321106.6666666665, ans=0.0
2023-11-23 09:12:59,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2321106.6666666665, ans=0.0
2023-11-23 09:13:09,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2321173.3333333335, ans=0.125
2023-11-23 09:13:13,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2321173.3333333335, ans=0.125
2023-11-23 09:13:18,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2321240.0, ans=0.125
2023-11-23 09:13:31,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2321306.6666666665, ans=0.1
2023-11-23 09:13:33,952 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348200
2023-11-23 09:13:54,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2321373.3333333335, ans=0.0
2023-11-23 09:13:56,836 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11550, loss[loss=0.0508, simple_loss=0.06814, pruned_loss=0.007718, audio_tagging_loss=0.009014, over 15812.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09207, pruned_loss=0.01413, audio_tagging_loss=0.009164, over 3044538.12 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 8.0
2023-11-23 09:13:59,200 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.220e+01 8.786e+01 9.512e+01 1.197e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 09:14:07,303 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.66 vs. limit=10.0
2023-11-23 09:14:08,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2321506.6666666665, ans=0.125
2023-11-23 09:14:19,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2321506.6666666665, ans=0.125
2023-11-23 09:14:26,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2321573.3333333335, ans=0.1
2023-11-23 09:14:35,099 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:14:37,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348250
2023-11-23 09:14:41,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2321640.0, ans=0.1
2023-11-23 09:14:49,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2321706.6666666665, ans=0.125
2023-11-23 09:14:54,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2321706.6666666665, ans=0.0
2023-11-23 09:14:56,906 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.21 vs. limit=15.0
2023-11-23 09:14:57,738 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:15:00,560 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11600, loss[loss=0.09169, simple_loss=0.1251, pruned_loss=0.02118, audio_tagging_loss=0.007964, over 15052.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09294, pruned_loss=0.01426, audio_tagging_loss=0.009106, over 3046259.25 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:15:06,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2321773.3333333335, ans=0.1
2023-11-23 09:15:07,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2321773.3333333335, ans=0.125
2023-11-23 09:15:34,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2321906.6666666665, ans=0.125
2023-11-23 09:15:35,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2321906.6666666665, ans=0.2
2023-11-23 09:15:37,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2321973.3333333335, ans=0.1
2023-11-23 09:15:41,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348300
2023-11-23 09:16:04,479 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11650, loss[loss=0.06735, simple_loss=0.08422, pruned_loss=0.01579, audio_tagging_loss=0.00945, over 16776.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09416, pruned_loss=0.01446, audio_tagging_loss=0.009018, over 3047466.06 frames. ], batch size: 63, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:16:06,807 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.255e+01 8.853e+01 9.730e+01 1.242e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 09:16:22,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2322173.3333333335, ans=0.125
2023-11-23 09:16:32,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2322240.0, ans=0.0
2023-11-23 09:16:46,101 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348350
2023-11-23 09:17:08,253 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11700, loss[loss=0.06365, simple_loss=0.0837, pruned_loss=0.01364, audio_tagging_loss=0.008162, over 14548.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09353, pruned_loss=0.01444, audio_tagging_loss=0.009053, over 3046918.92 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:17:14,233 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:17:14,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.59 vs. limit=15.0
2023-11-23 09:17:22,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2322506.6666666665, ans=0.125
2023-11-23 09:17:25,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2322506.6666666665, ans=0.09899494936611666
2023-11-23 09:17:27,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2322506.6666666665, ans=0.1
2023-11-23 09:17:48,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2322640.0, ans=0.1
2023-11-23 09:17:50,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348400
2023-11-23 09:17:51,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2322640.0, ans=0.125
2023-11-23 09:17:52,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2322640.0, ans=0.1
2023-11-23 09:17:54,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2322640.0, ans=0.125
2023-11-23 09:18:01,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2322706.6666666665, ans=0.0
2023-11-23 09:18:13,159 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11750, loss[loss=0.06222, simple_loss=0.09091, pruned_loss=0.008779, audio_tagging_loss=0.007986, over 16769.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09295, pruned_loss=0.01444, audio_tagging_loss=0.009169, over 3049255.72 frames. ], batch size: 64, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:18:16,769 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.187e+01 8.216e+01 8.845e+01 9.440e+01 1.082e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 09:18:17,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.24 vs. limit=15.0
2023-11-23 09:18:34,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2322840.0, ans=0.125
2023-11-23 09:18:55,042 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348450
2023-11-23 09:18:57,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2322973.3333333335, ans=0.0
2023-11-23 09:19:16,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2323040.0, ans=0.0
2023-11-23 09:19:18,710 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11800, loss[loss=0.07389, simple_loss=0.09644, pruned_loss=0.01646, audio_tagging_loss=0.009211, over 14443.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.0928, pruned_loss=0.01444, audio_tagging_loss=0.009228, over 3042345.73 frames. ], batch size: 57, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:19:36,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2323173.3333333335, ans=10.0
2023-11-23 09:19:50,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2323240.0, ans=0.0
2023-11-23 09:19:57,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.15 vs. limit=15.0
2023-11-23 09:19:59,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348500
2023-11-23 09:20:13,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2323373.3333333335, ans=0.0
2023-11-23 09:20:16,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2323373.3333333335, ans=0.0
2023-11-23 09:20:22,058 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11850, loss[loss=0.05322, simple_loss=0.06308, pruned_loss=0.01188, audio_tagging_loss=0.009792, over 16106.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09235, pruned_loss=0.01418, audio_tagging_loss=0.009172, over 3039817.66 frames. ], batch size: 62, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:20:24,368 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.461e+01 9.109e+01 9.788e+01 1.263e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-23 09:20:29,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2323440.0, ans=0.125
2023-11-23 09:20:32,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2323440.0, ans=0.125
2023-11-23 09:20:57,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2323573.3333333335, ans=0.125
2023-11-23 09:21:03,610 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348550
2023-11-23 09:21:14,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2323706.6666666665, ans=15.0
2023-11-23 09:21:21,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.50 vs. limit=15.0
2023-11-23 09:21:21,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2323706.6666666665, ans=0.2
2023-11-23 09:21:24,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2323773.3333333335, ans=0.125
2023-11-23 09:21:25,834 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11900, loss[loss=0.07968, simple_loss=0.1003, pruned_loss=0.01724, audio_tagging_loss=0.01228, over 15852.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.0929, pruned_loss=0.01411, audio_tagging_loss=0.009199, over 3044980.36 frames. ], batch size: 64, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:21:29,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2323773.3333333335, ans=0.125
2023-11-23 09:21:29,104 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:21:46,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2323840.0, ans=0.125
2023-11-23 09:21:56,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2323906.6666666665, ans=0.95
2023-11-23 09:22:06,671 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348600
2023-11-23 09:22:09,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2323973.3333333335, ans=0.0
2023-11-23 09:22:12,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2323973.3333333335, ans=0.1
2023-11-23 09:22:31,800 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 11950, loss[loss=0.05287, simple_loss=0.05979, pruned_loss=0.01067, audio_tagging_loss=0.0123, over 15755.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09262, pruned_loss=0.01398, audio_tagging_loss=0.009377, over 3037419.50 frames. ], batch size: 62, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:22:34,226 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.393e+01 8.390e+01 8.982e+01 9.534e+01 1.652e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 09:22:46,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2324173.3333333335, ans=0.125
2023-11-23 09:22:51,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2324173.3333333335, ans=0.125
2023-11-23 09:23:07,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2324306.6666666665, ans=0.125
2023-11-23 09:23:11,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348650
2023-11-23 09:23:17,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.42 vs. limit=15.0
2023-11-23 09:23:33,696 INFO [train_asr.py:1221] (2/4) Epoch 29, batch 12000, loss[loss=0.07032, simple_loss=0.1011, pruned_loss=0.01086, audio_tagging_loss=0.008889, over 14321.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09444, pruned_loss=0.01429, audio_tagging_loss=0.009264, over 3045864.19 frames. ], batch size: 54, lr: 2.32e-03, grad_scale: 32.0
2023-11-23 09:23:33,697 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 09:23:58,442 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.2386, 4.9309, 4.6554, 5.0507], device='cuda:2')
2023-11-23 09:24:15,957 INFO [train_asr.py:1253] (2/4) Epoch 29, validation: loss=0.05844, simple_loss=0.05118, pruned_loss=0.005114, audio_tagging_loss=0.02774, over 4681554.00 frames. 
2023-11-23 09:24:15,957 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 09:24:19,979 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.48 vs. limit=15.0
2023-11-23 09:24:20,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2324440.0, ans=0.125
2023-11-23 09:24:23,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2324440.0, ans=0.1
2023-11-23 09:24:23,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.12 vs. limit=15.0
2023-11-23 09:25:23,498 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 0, loss[loss=0.07479, simple_loss=0.08092, pruned_loss=0.01122, audio_tagging_loss=0.02312, over 15755.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.08092, pruned_loss=0.01122, audio_tagging_loss=0.02312, over 15755.00 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:25:23,499 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 09:26:02,092 INFO [train_asr.py:1253] (2/4) Epoch 30, validation: loss=0.05824, simple_loss=0.05113, pruned_loss=0.005061, audio_tagging_loss=0.02761, over 4681554.00 frames. 
2023-11-23 09:26:02,093 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 09:26:11,936 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348700
2023-11-23 09:26:13,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2324666.6666666665, ans=0.0
2023-11-23 09:26:14,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.76 vs. limit=12.0
2023-11-23 09:26:23,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.93 vs. limit=6.0
2023-11-23 09:26:26,898 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.48 vs. limit=15.0
2023-11-23 09:26:35,744 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.954e+01 9.660e+01 1.053e+02 1.291e+02, threshold=1.932e+02, percent-clipped=0.0
2023-11-23 09:27:05,494 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 50, loss[loss=0.07403, simple_loss=0.09085, pruned_loss=0.01247, audio_tagging_loss=0.01613, over 15651.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.08988, pruned_loss=0.01298, audio_tagging_loss=0.01772, over 691084.72 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:27:15,525 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348750
2023-11-23 09:27:16,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2325000.0, ans=0.125
2023-11-23 09:27:24,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2325000.0, ans=0.125
2023-11-23 09:27:48,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.37 vs. limit=15.0
2023-11-23 09:27:57,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2325200.0, ans=0.125
2023-11-23 09:28:01,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2325200.0, ans=0.125
2023-11-23 09:28:06,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2325200.0, ans=0.0
2023-11-23 09:28:08,362 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 100, loss[loss=0.07423, simple_loss=0.08381, pruned_loss=0.01596, audio_tagging_loss=0.01636, over 14904.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09035, pruned_loss=0.01401, audio_tagging_loss=0.01691, over 1209394.75 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:28:09,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2325266.6666666665, ans=0.0
2023-11-23 09:28:18,714 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348800
2023-11-23 09:28:46,357 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.439e+01 8.985e+01 9.565e+01 1.028e+02 2.272e+02, threshold=1.913e+02, percent-clipped=1.0
2023-11-23 09:28:51,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2325466.6666666665, ans=0.125
2023-11-23 09:29:05,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2325533.3333333335, ans=0.125
2023-11-23 09:29:13,427 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 150, loss[loss=0.05189, simple_loss=0.05792, pruned_loss=0.00918, audio_tagging_loss=0.01374, over 15360.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.08996, pruned_loss=0.01352, audio_tagging_loss=0.01504, over 1610140.73 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:29:17,287 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:29:24,404 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348850
2023-11-23 09:29:25,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2325666.6666666665, ans=0.125
2023-11-23 09:29:28,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2325666.6666666665, ans=0.125
2023-11-23 09:29:36,065 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.43 vs. limit=15.0
2023-11-23 09:29:46,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2325733.3333333335, ans=0.0
2023-11-23 09:29:54,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2325800.0, ans=0.125
2023-11-23 09:30:08,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2325866.6666666665, ans=0.125
2023-11-23 09:30:18,434 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 200, loss[loss=0.05353, simple_loss=0.07379, pruned_loss=0.006903, audio_tagging_loss=0.009733, over 15140.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09108, pruned_loss=0.01373, audio_tagging_loss=0.01325, over 1923222.14 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:30:28,214 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348900
2023-11-23 09:30:29,908 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.70 vs. limit=15.0
2023-11-23 09:30:55,303 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.392e+01 9.007e+01 9.652e+01 1.196e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 09:31:01,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2326133.3333333335, ans=0.0
2023-11-23 09:31:21,701 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 250, loss[loss=0.07955, simple_loss=0.1064, pruned_loss=0.01671, audio_tagging_loss=0.009647, over 15290.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09046, pruned_loss=0.01351, audio_tagging_loss=0.01216, over 2169484.96 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:31:29,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2326266.6666666665, ans=22.5
2023-11-23 09:31:32,245 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 348950
2023-11-23 09:31:33,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2326333.3333333335, ans=0.125
2023-11-23 09:31:59,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2326466.6666666665, ans=0.125
2023-11-23 09:32:09,770 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.63 vs. limit=15.0
2023-11-23 09:32:16,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2326533.3333333335, ans=0.1
2023-11-23 09:32:18,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2326533.3333333335, ans=0.0
2023-11-23 09:32:22,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2326533.3333333335, ans=0.125
2023-11-23 09:32:23,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2326533.3333333335, ans=0.0
2023-11-23 09:32:26,043 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 300, loss[loss=0.05155, simple_loss=0.06251, pruned_loss=0.009801, audio_tagging_loss=0.0105, over 14893.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09083, pruned_loss=0.01375, audio_tagging_loss=0.01131, over 2361585.52 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:32:26,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2326600.0, ans=0.0
2023-11-23 09:32:36,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349000
2023-11-23 09:32:36,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2326600.0, ans=0.2
2023-11-23 09:32:52,736 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.84 vs. limit=15.0
2023-11-23 09:32:56,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2326733.3333333335, ans=0.2
2023-11-23 09:33:00,027 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.62 vs. limit=12.0
2023-11-23 09:33:02,990 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.407e+01 8.907e+01 9.577e+01 1.241e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 09:33:08,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2326800.0, ans=0.0
2023-11-23 09:33:18,329 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.75 vs. limit=15.0
2023-11-23 09:33:24,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2326866.6666666665, ans=0.1
2023-11-23 09:33:30,927 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 350, loss[loss=0.06091, simple_loss=0.08841, pruned_loss=0.0104, audio_tagging_loss=0.0063, over 15293.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09139, pruned_loss=0.01362, audio_tagging_loss=0.01066, over 2508848.81 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 09:33:40,787 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349050
2023-11-23 09:33:46,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2327000.0, ans=0.125
2023-11-23 09:33:54,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2327066.6666666665, ans=0.2
2023-11-23 09:33:57,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2327066.6666666665, ans=0.1
2023-11-23 09:34:14,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2327133.3333333335, ans=0.2
2023-11-23 09:34:33,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2327266.6666666665, ans=0.125
2023-11-23 09:34:34,500 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 400, loss[loss=0.05219, simple_loss=0.06686, pruned_loss=0.0106, audio_tagging_loss=0.008162, over 14186.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09152, pruned_loss=0.01393, audio_tagging_loss=0.01025, over 2629724.73 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:34:44,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349100
2023-11-23 09:35:10,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2327400.0, ans=0.0
2023-11-23 09:35:13,635 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.415e+01 9.145e+01 9.820e+01 1.379e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 09:35:33,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2327533.3333333335, ans=0.125
2023-11-23 09:35:39,513 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 450, loss[loss=0.07199, simple_loss=0.09558, pruned_loss=0.01799, audio_tagging_loss=0.006207, over 15032.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09208, pruned_loss=0.01405, audio_tagging_loss=0.009886, over 2719578.10 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:35:50,122 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349150
2023-11-23 09:35:51,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2327666.6666666665, ans=0.09899494936611666
2023-11-23 09:36:10,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2327733.3333333335, ans=0.125
2023-11-23 09:36:25,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2327800.0, ans=0.125
2023-11-23 09:36:43,353 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 500, loss[loss=0.06579, simple_loss=0.08955, pruned_loss=0.01121, audio_tagging_loss=0.009809, over 15234.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09214, pruned_loss=0.01401, audio_tagging_loss=0.009656, over 2787957.39 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:36:44,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2327933.3333333335, ans=0.0
2023-11-23 09:36:48,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2327933.3333333335, ans=0.125
2023-11-23 09:36:53,761 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349200
2023-11-23 09:37:22,398 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 8.359e+01 8.809e+01 9.526e+01 1.262e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 09:37:43,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2328200.0, ans=0.125
2023-11-23 09:37:47,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.71 vs. limit=22.5
2023-11-23 09:37:48,450 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 550, loss[loss=0.06136, simple_loss=0.0873, pruned_loss=0.01093, audio_tagging_loss=0.006779, over 15981.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09219, pruned_loss=0.01408, audio_tagging_loss=0.00954, over 2842707.90 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:37:58,392 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349250
2023-11-23 09:38:24,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2328400.0, ans=0.125
2023-11-23 09:38:27,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2328466.6666666665, ans=0.125
2023-11-23 09:38:29,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2328466.6666666665, ans=0.1
2023-11-23 09:38:38,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2328533.3333333335, ans=0.2
2023-11-23 09:38:52,668 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 600, loss[loss=0.08036, simple_loss=0.102, pruned_loss=0.01916, audio_tagging_loss=0.01022, over 16250.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09231, pruned_loss=0.014, audio_tagging_loss=0.009414, over 2886679.86 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:39:03,379 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349300
2023-11-23 09:39:16,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2328666.6666666665, ans=0.0
2023-11-23 09:39:19,224 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.21 vs. limit=22.5
2023-11-23 09:39:27,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2328733.3333333335, ans=0.0
2023-11-23 09:39:31,019 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.142e+01 8.700e+01 9.475e+01 1.259e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-23 09:39:39,938 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:39:52,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2328866.6666666665, ans=0.125
2023-11-23 09:39:57,388 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 650, loss[loss=0.06871, simple_loss=0.09392, pruned_loss=0.01344, audio_tagging_loss=0.008305, over 15924.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09267, pruned_loss=0.0142, audio_tagging_loss=0.009444, over 2928512.37 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:39:58,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.06 vs. limit=15.0
2023-11-23 09:40:07,346 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349350
2023-11-23 09:40:08,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2329000.0, ans=0.125
2023-11-23 09:40:11,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2329000.0, ans=0.125
2023-11-23 09:40:18,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2329000.0, ans=10.0
2023-11-23 09:40:24,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2329066.6666666665, ans=0.0
2023-11-23 09:40:35,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2329133.3333333335, ans=0.1
2023-11-23 09:40:48,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2329200.0, ans=0.1
2023-11-23 09:40:49,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2329200.0, ans=0.1
2023-11-23 09:41:02,195 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 700, loss[loss=0.05336, simple_loss=0.06553, pruned_loss=0.01213, audio_tagging_loss=0.008461, over 14009.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09303, pruned_loss=0.01414, audio_tagging_loss=0.00934, over 2957480.62 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:41:12,059 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349400
2023-11-23 09:41:23,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2329333.3333333335, ans=0.1
2023-11-23 09:41:41,167 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.086e+01 8.675e+01 9.792e+01 1.198e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 09:41:46,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2329466.6666666665, ans=0.0
2023-11-23 09:41:46,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2329466.6666666665, ans=0.1
2023-11-23 09:41:54,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2329533.3333333335, ans=0.0
2023-11-23 09:42:06,209 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 750, loss[loss=0.07181, simple_loss=0.1043, pruned_loss=0.01148, audio_tagging_loss=0.008166, over 15017.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09436, pruned_loss=0.01438, audio_tagging_loss=0.009255, over 2982080.21 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:42:06,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.30 vs. limit=22.5
2023-11-23 09:42:17,143 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349450
2023-11-23 09:43:12,045 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 800, loss[loss=0.06432, simple_loss=0.09179, pruned_loss=0.01148, audio_tagging_loss=0.006946, over 15609.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09405, pruned_loss=0.01431, audio_tagging_loss=0.009292, over 3004151.79 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:43:21,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349500
2023-11-23 09:43:26,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2330000.0, ans=0.1
2023-11-23 09:43:50,463 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.139e+01 8.858e+01 9.489e+01 1.259e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 09:43:59,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2330133.3333333335, ans=0.0
2023-11-23 09:44:13,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2330200.0, ans=0.125
2023-11-23 09:44:15,738 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 850, loss[loss=0.07944, simple_loss=0.1002, pruned_loss=0.018, audio_tagging_loss=0.01136, over 15747.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09473, pruned_loss=0.0144, audio_tagging_loss=0.009333, over 3018856.14 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:44:17,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2330266.6666666665, ans=0.0
2023-11-23 09:44:26,115 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349550
2023-11-23 09:44:26,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2330266.6666666665, ans=0.125
2023-11-23 09:44:45,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2330400.0, ans=0.125
2023-11-23 09:44:47,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2330400.0, ans=0.125
2023-11-23 09:44:49,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2330400.0, ans=0.125
2023-11-23 09:45:03,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2330466.6666666665, ans=0.09899494936611666
2023-11-23 09:45:04,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2330466.6666666665, ans=0.125
2023-11-23 09:45:10,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2330533.3333333335, ans=0.0
2023-11-23 09:45:17,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2330533.3333333335, ans=0.0
2023-11-23 09:45:19,619 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 900, loss[loss=0.06216, simple_loss=0.08759, pruned_loss=0.01058, audio_tagging_loss=0.007786, over 15598.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09405, pruned_loss=0.01434, audio_tagging_loss=0.009435, over 3022269.34 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:45:30,751 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349600
2023-11-23 09:45:43,317 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.44 vs. limit=15.0
2023-11-23 09:45:48,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2330733.3333333335, ans=0.1
2023-11-23 09:45:49,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2330733.3333333335, ans=0.0
2023-11-23 09:45:50,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2330733.3333333335, ans=0.0
2023-11-23 09:45:52,435 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:45:58,084 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.328e+01 8.818e+01 9.334e+01 1.944e+02, threshold=1.764e+02, percent-clipped=1.0
2023-11-23 09:45:59,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2330800.0, ans=0.125
2023-11-23 09:45:59,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2330800.0, ans=0.125
2023-11-23 09:46:09,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2330866.6666666665, ans=0.125
2023-11-23 09:46:15,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2330866.6666666665, ans=0.125
2023-11-23 09:46:18,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2330866.6666666665, ans=0.0
2023-11-23 09:46:24,194 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 950, loss[loss=0.0836, simple_loss=0.1097, pruned_loss=0.02102, audio_tagging_loss=0.007734, over 14810.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09338, pruned_loss=0.01421, audio_tagging_loss=0.00944, over 3025169.08 frames. ], batch size: 53, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:46:34,453 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349650
2023-11-23 09:46:34,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2330933.3333333335, ans=0.5
2023-11-23 09:46:39,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2331000.0, ans=0.05
2023-11-23 09:46:56,045 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:47:04,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2331133.3333333335, ans=0.125
2023-11-23 09:47:22,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.79 vs. limit=15.0
2023-11-23 09:47:25,129 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.08 vs. limit=15.0
2023-11-23 09:47:28,209 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1000, loss[loss=0.07135, simple_loss=0.08936, pruned_loss=0.01536, audio_tagging_loss=0.01131, over 16201.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09272, pruned_loss=0.01406, audio_tagging_loss=0.009299, over 3027528.38 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:47:36,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2331266.6666666665, ans=0.1
2023-11-23 09:47:37,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.54 vs. limit=12.0
2023-11-23 09:47:38,041 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349700
2023-11-23 09:47:57,296 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:48:07,584 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.508e+01 8.244e+01 9.173e+01 9.794e+01 1.161e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-23 09:48:32,078 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1050, loss[loss=0.05553, simple_loss=0.07494, pruned_loss=0.009127, audio_tagging_loss=0.00893, over 16307.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09221, pruned_loss=0.01397, audio_tagging_loss=0.009267, over 3035265.87 frames. ], batch size: 61, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:48:43,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349750
2023-11-23 09:48:57,313 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.83 vs. limit=15.0
2023-11-23 09:48:59,922 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.10 vs. limit=22.5
2023-11-23 09:49:01,084 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.67 vs. limit=15.0
2023-11-23 09:49:07,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2331733.3333333335, ans=0.125
2023-11-23 09:49:08,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.55 vs. limit=5.0
2023-11-23 09:49:16,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2331800.0, ans=0.0
2023-11-23 09:49:27,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2331866.6666666665, ans=0.125
2023-11-23 09:49:35,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2331866.6666666665, ans=0.07
2023-11-23 09:49:37,732 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1100, loss[loss=0.06505, simple_loss=0.08971, pruned_loss=0.01144, audio_tagging_loss=0.008752, over 15863.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09275, pruned_loss=0.01396, audio_tagging_loss=0.009161, over 3035784.48 frames. ], batch size: 61, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:49:39,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2331933.3333333335, ans=0.125
2023-11-23 09:49:41,471 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:49:42,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2331933.3333333335, ans=0.2
2023-11-23 09:49:48,150 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349800
2023-11-23 09:50:10,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2332066.6666666665, ans=0.125
2023-11-23 09:50:15,122 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 8.049e+01 8.629e+01 9.475e+01 1.161e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-23 09:50:25,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2332133.3333333335, ans=0.125
2023-11-23 09:50:42,001 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1150, loss[loss=0.07181, simple_loss=0.09624, pruned_loss=0.01642, audio_tagging_loss=0.007265, over 15384.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09231, pruned_loss=0.01375, audio_tagging_loss=0.009167, over 3034071.70 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:50:43,868 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.04 vs. limit=22.5
2023-11-23 09:50:51,323 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.52 vs. limit=15.0
2023-11-23 09:50:52,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349850
2023-11-23 09:51:08,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2332400.0, ans=0.125
2023-11-23 09:51:42,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2332533.3333333335, ans=0.125
2023-11-23 09:51:45,112 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1200, loss[loss=0.04612, simple_loss=0.06132, pruned_loss=0.006128, audio_tagging_loss=0.009338, over 14416.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09221, pruned_loss=0.01382, audio_tagging_loss=0.009124, over 3029959.90 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:51:54,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349900
2023-11-23 09:52:16,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2332733.3333333335, ans=0.125
2023-11-23 09:52:21,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2332733.3333333335, ans=0.95
2023-11-23 09:52:24,720 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.340e+01 9.039e+01 9.535e+01 1.288e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-23 09:52:34,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2332866.6666666665, ans=0.05
2023-11-23 09:52:38,317 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:52:49,392 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1250, loss[loss=0.05996, simple_loss=0.07908, pruned_loss=0.00999, audio_tagging_loss=0.01043, over 14738.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09247, pruned_loss=0.01393, audio_tagging_loss=0.009027, over 3029122.01 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:52:50,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2332933.3333333335, ans=0.0
2023-11-23 09:52:59,707 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 349950
2023-11-23 09:53:14,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2333066.6666666665, ans=0.1
2023-11-23 09:53:22,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2333066.6666666665, ans=0.125
2023-11-23 09:53:27,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.71 vs. limit=12.0
2023-11-23 09:53:52,862 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1300, loss[loss=0.0642, simple_loss=0.08505, pruned_loss=0.01398, audio_tagging_loss=0.0077, over 15399.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09202, pruned_loss=0.01379, audio_tagging_loss=0.009025, over 3030953.53 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:54:03,049 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350000
2023-11-23 09:54:14,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2333333.3333333335, ans=0.0
2023-11-23 09:54:28,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2333400.0, ans=0.0
2023-11-23 09:54:32,752 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.017e+01 7.999e+01 8.654e+01 9.564e+01 1.101e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-23 09:54:53,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2333533.3333333335, ans=0.0
2023-11-23 09:54:56,652 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1350, loss[loss=0.08837, simple_loss=0.1276, pruned_loss=0.01655, audio_tagging_loss=0.008014, over 15237.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09243, pruned_loss=0.01382, audio_tagging_loss=0.008995, over 3037695.69 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:55:03,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2333600.0, ans=0.125
2023-11-23 09:55:06,667 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350050
2023-11-23 09:55:12,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2333666.6666666665, ans=0.125
2023-11-23 09:55:30,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2333733.3333333335, ans=0.0
2023-11-23 09:55:35,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2333800.0, ans=0.1
2023-11-23 09:55:40,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2333800.0, ans=0.125
2023-11-23 09:55:43,907 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:55:45,546 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.99 vs. limit=15.0
2023-11-23 09:55:53,334 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.14 vs. limit=15.0
2023-11-23 09:55:57,742 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:56:00,482 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1400, loss[loss=0.0761, simple_loss=0.1103, pruned_loss=0.01496, audio_tagging_loss=0.005979, over 14801.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09313, pruned_loss=0.01403, audio_tagging_loss=0.008971, over 3040367.93 frames. ], batch size: 53, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:56:10,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.11 vs. limit=15.0
2023-11-23 09:56:11,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350100
2023-11-23 09:56:21,865 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:56:22,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2334000.0, ans=0.125
2023-11-23 09:56:25,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2334066.6666666665, ans=10.0
2023-11-23 09:56:31,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2334066.6666666665, ans=0.0
2023-11-23 09:56:33,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2334066.6666666665, ans=0.035
2023-11-23 09:56:39,708 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 8.084e+01 8.982e+01 9.535e+01 1.222e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 09:56:46,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2334133.3333333335, ans=0.125
2023-11-23 09:56:51,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2334200.0, ans=0.07
2023-11-23 09:56:54,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.87 vs. limit=12.0
2023-11-23 09:57:04,686 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1450, loss[loss=0.07066, simple_loss=0.09614, pruned_loss=0.01332, audio_tagging_loss=0.009263, over 15793.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09345, pruned_loss=0.01408, audio_tagging_loss=0.008991, over 3042527.11 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:57:14,493 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350150
2023-11-23 09:57:19,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2334333.3333333335, ans=0.2
2023-11-23 09:57:30,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2334400.0, ans=0.125
2023-11-23 09:57:40,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2334466.6666666665, ans=0.125
2023-11-23 09:57:53,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2334533.3333333335, ans=0.125
2023-11-23 09:58:06,389 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1500, loss[loss=0.06613, simple_loss=0.09061, pruned_loss=0.009702, audio_tagging_loss=0.01112, over 15299.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09386, pruned_loss=0.01419, audio_tagging_loss=0.009122, over 3049556.25 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:58:11,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2334600.0, ans=0.125
2023-11-23 09:58:16,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350200
2023-11-23 09:58:30,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2334733.3333333335, ans=0.125
2023-11-23 09:58:30,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2334733.3333333335, ans=0.125
2023-11-23 09:58:40,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2334733.3333333335, ans=0.0
2023-11-23 09:58:49,007 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.399e+01 9.056e+01 9.468e+01 1.678e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 09:58:58,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2334866.6666666665, ans=0.0
2023-11-23 09:58:58,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2334866.6666666665, ans=0.0
2023-11-23 09:59:10,067 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1550, loss[loss=0.04932, simple_loss=0.06094, pruned_loss=0.007576, audio_tagging_loss=0.01127, over 14771.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09319, pruned_loss=0.01403, audio_tagging_loss=0.009292, over 3046927.49 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 09:59:17,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2334933.3333333335, ans=0.125
2023-11-23 09:59:20,622 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350250
2023-11-23 09:59:30,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2335000.0, ans=0.1
2023-11-23 09:59:31,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2335000.0, ans=0.5
2023-11-23 09:59:47,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2335133.3333333335, ans=0.0
2023-11-23 09:59:56,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2335133.3333333335, ans=0.05
2023-11-23 10:00:14,401 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1600, loss[loss=0.07674, simple_loss=0.1079, pruned_loss=0.01532, audio_tagging_loss=0.007459, over 15325.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09306, pruned_loss=0.01401, audio_tagging_loss=0.009368, over 3049025.06 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:00:16,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2335266.6666666665, ans=0.1
2023-11-23 10:00:24,799 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350300
2023-11-23 10:00:54,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2335466.6666666665, ans=0.125
2023-11-23 10:00:55,559 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.176e+01 8.486e+01 9.012e+01 9.974e+01 1.250e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 10:01:14,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2335533.3333333335, ans=0.5
2023-11-23 10:01:17,404 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1650, loss[loss=0.0751, simple_loss=0.1058, pruned_loss=0.01415, audio_tagging_loss=0.008051, over 15334.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09216, pruned_loss=0.01396, audio_tagging_loss=0.009491, over 3047493.92 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:01:27,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350350
2023-11-23 10:01:42,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.91 vs. limit=15.0
2023-11-23 10:02:05,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2335800.0, ans=0.125
2023-11-23 10:02:21,327 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1700, loss[loss=0.08206, simple_loss=0.1068, pruned_loss=0.01763, audio_tagging_loss=0.01105, over 15502.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09243, pruned_loss=0.01425, audio_tagging_loss=0.009544, over 3041607.09 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:02:22,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2335933.3333333335, ans=0.0
2023-11-23 10:02:31,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350400
2023-11-23 10:02:33,681 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.32 vs. limit=10.0
2023-11-23 10:03:04,968 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.246e+01 8.760e+01 9.410e+01 1.135e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 10:03:11,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.13 vs. limit=12.0
2023-11-23 10:03:16,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2336200.0, ans=0.0
2023-11-23 10:03:25,718 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1750, loss[loss=0.07764, simple_loss=0.09912, pruned_loss=0.02029, audio_tagging_loss=0.007798, over 14848.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09199, pruned_loss=0.01425, audio_tagging_loss=0.009401, over 3044531.25 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:03:30,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2336266.6666666665, ans=0.1
2023-11-23 10:03:36,509 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350450
2023-11-23 10:03:55,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.85 vs. limit=10.0
2023-11-23 10:04:06,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2336466.6666666665, ans=0.1
2023-11-23 10:04:18,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2336533.3333333335, ans=0.1
2023-11-23 10:04:31,143 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1800, loss[loss=0.05982, simple_loss=0.07811, pruned_loss=0.01032, audio_tagging_loss=0.01044, over 16106.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09207, pruned_loss=0.01426, audio_tagging_loss=0.009304, over 3045078.94 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:04:37,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2336600.0, ans=0.125
2023-11-23 10:04:41,011 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350500
2023-11-23 10:04:43,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2336666.6666666665, ans=0.1
2023-11-23 10:04:43,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2336666.6666666665, ans=0.0
2023-11-23 10:05:15,200 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.590e+01 9.171e+01 9.752e+01 1.397e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 10:05:15,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2336800.0, ans=0.0
2023-11-23 10:05:21,401 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.21 vs. limit=15.0
2023-11-23 10:05:31,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=2336866.6666666665, ans=10.0
2023-11-23 10:05:34,969 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.10 vs. limit=22.5
2023-11-23 10:05:35,398 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1850, loss[loss=0.06026, simple_loss=0.07822, pruned_loss=0.01223, audio_tagging_loss=0.008918, over 15010.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09281, pruned_loss=0.01449, audio_tagging_loss=0.009219, over 3043895.20 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:05:46,521 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350550
2023-11-23 10:05:46,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2336933.3333333335, ans=0.125
2023-11-23 10:05:52,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2337000.0, ans=0.0
2023-11-23 10:05:53,437 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.85 vs. limit=15.0
2023-11-23 10:06:05,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2337066.6666666665, ans=0.125
2023-11-23 10:06:23,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2337133.3333333335, ans=0.125
2023-11-23 10:06:31,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2337200.0, ans=0.0
2023-11-23 10:06:37,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2337200.0, ans=0.0
2023-11-23 10:06:39,854 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1900, loss[loss=0.09903, simple_loss=0.1281, pruned_loss=0.02722, audio_tagging_loss=0.007751, over 15812.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09298, pruned_loss=0.01447, audio_tagging_loss=0.009109, over 3047903.04 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:06:50,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350600
2023-11-23 10:06:51,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2337333.3333333335, ans=0.125
2023-11-23 10:06:54,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2337333.3333333335, ans=0.07
2023-11-23 10:06:56,086 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.37 vs. limit=12.0
2023-11-23 10:07:20,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2337466.6666666665, ans=0.015
2023-11-23 10:07:24,832 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.717e+01 8.255e+01 9.028e+01 9.866e+01 1.250e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 10:07:27,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2337466.6666666665, ans=0.07
2023-11-23 10:07:45,192 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 1950, loss[loss=0.06949, simple_loss=0.09984, pruned_loss=0.013, audio_tagging_loss=0.006577, over 15545.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09317, pruned_loss=0.01435, audio_tagging_loss=0.009001, over 3046264.01 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:07:55,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350650
2023-11-23 10:08:07,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2337666.6666666665, ans=0.125
2023-11-23 10:08:19,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2337733.3333333335, ans=0.125
2023-11-23 10:08:27,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2337800.0, ans=0.125
2023-11-23 10:08:41,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.32 vs. limit=15.0
2023-11-23 10:08:50,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2337933.3333333335, ans=0.0
2023-11-23 10:08:50,764 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.71 vs. limit=22.5
2023-11-23 10:08:51,214 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2000, loss[loss=0.06373, simple_loss=0.08528, pruned_loss=0.009567, audio_tagging_loss=0.01152, over 14269.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09295, pruned_loss=0.01432, audio_tagging_loss=0.00909, over 3045658.92 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:09:02,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350700
2023-11-23 10:09:07,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2338000.0, ans=0.125
2023-11-23 10:09:35,832 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.268e+01 8.767e+01 9.455e+01 1.201e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-23 10:09:42,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2338200.0, ans=0.125
2023-11-23 10:09:57,059 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2050, loss[loss=0.07359, simple_loss=0.09907, pruned_loss=0.01594, audio_tagging_loss=0.008116, over 16467.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09343, pruned_loss=0.01451, audio_tagging_loss=0.009015, over 3042152.67 frames. ], batch size: 61, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:09:58,805 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.27 vs. limit=12.0
2023-11-23 10:10:07,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350750
2023-11-23 10:10:16,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2338333.3333333335, ans=0.2
2023-11-23 10:10:34,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2338466.6666666665, ans=0.0
2023-11-23 10:10:50,750 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.31 vs. limit=15.0
2023-11-23 10:11:01,827 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2100, loss[loss=0.06296, simple_loss=0.08257, pruned_loss=0.01344, audio_tagging_loss=0.008231, over 14664.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09334, pruned_loss=0.01437, audio_tagging_loss=0.008974, over 3044101.95 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:11:04,523 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:11:11,702 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350800
2023-11-23 10:11:12,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2338600.0, ans=0.0
2023-11-23 10:11:30,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.76 vs. limit=22.5
2023-11-23 10:11:38,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2338733.3333333335, ans=0.125
2023-11-23 10:11:39,046 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.80 vs. limit=15.0
2023-11-23 10:11:46,478 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.972e+01 8.232e+01 8.810e+01 9.514e+01 1.231e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 10:12:06,510 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2150, loss[loss=0.06172, simple_loss=0.07817, pruned_loss=0.01329, audio_tagging_loss=0.009348, over 15988.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09187, pruned_loss=0.01413, audio_tagging_loss=0.009059, over 3039852.88 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:12:17,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350850
2023-11-23 10:12:18,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2338933.3333333335, ans=0.125
2023-11-23 10:12:35,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.85 vs. limit=6.0
2023-11-23 10:12:38,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2339066.6666666665, ans=0.125
2023-11-23 10:12:47,112 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:13:12,304 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2200, loss[loss=0.1041, simple_loss=0.1472, pruned_loss=0.02509, audio_tagging_loss=0.005392, over 15313.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09323, pruned_loss=0.01428, audio_tagging_loss=0.008922, over 3048277.41 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:13:12,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.48 vs. limit=10.0
2023-11-23 10:13:16,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2339266.6666666665, ans=0.0
2023-11-23 10:13:22,647 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350900
2023-11-23 10:13:25,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2339333.3333333335, ans=0.0
2023-11-23 10:13:51,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2339466.6666666665, ans=0.125
2023-11-23 10:13:55,238 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.264e+01 8.958e+01 9.575e+01 1.152e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 10:14:13,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2339533.3333333335, ans=0.2
2023-11-23 10:14:17,022 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2250, loss[loss=0.05016, simple_loss=0.06486, pruned_loss=0.006388, audio_tagging_loss=0.01135, over 15113.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09295, pruned_loss=0.01415, audio_tagging_loss=0.008954, over 3049195.69 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:14:18,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2339600.0, ans=0.1
2023-11-23 10:14:22,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2339600.0, ans=0.125
2023-11-23 10:14:26,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 350950
2023-11-23 10:14:42,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2339733.3333333335, ans=0.125
2023-11-23 10:15:02,314 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.13 vs. limit=6.0
2023-11-23 10:15:21,426 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2300, loss[loss=0.05948, simple_loss=0.07207, pruned_loss=0.01125, audio_tagging_loss=0.01219, over 15930.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.0939, pruned_loss=0.01438, audio_tagging_loss=0.009013, over 3048157.80 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:15:26,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2339933.3333333335, ans=0.125
2023-11-23 10:15:31,150 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351000
2023-11-23 10:15:39,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2340000.0, ans=0.125
2023-11-23 10:15:44,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2340000.0, ans=0.125
2023-11-23 10:16:06,088 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.543e+01 9.090e+01 9.740e+01 1.795e+02, threshold=1.818e+02, percent-clipped=1.0
2023-11-23 10:16:13,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2340200.0, ans=0.125
2023-11-23 10:16:15,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2340200.0, ans=0.0
2023-11-23 10:16:19,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2340200.0, ans=0.125
2023-11-23 10:16:19,868 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:16:23,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2340200.0, ans=0.125
2023-11-23 10:16:26,859 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2350, loss[loss=0.06863, simple_loss=0.09875, pruned_loss=0.01204, audio_tagging_loss=0.007212, over 15079.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09326, pruned_loss=0.01409, audio_tagging_loss=0.009221, over 3047669.24 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:16:30,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2340266.6666666665, ans=0.125
2023-11-23 10:16:38,031 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351050
2023-11-23 10:16:59,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2340400.0, ans=0.125
2023-11-23 10:17:14,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2340466.6666666665, ans=0.0
2023-11-23 10:17:32,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.79 vs. limit=12.0
2023-11-23 10:17:33,012 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2400, loss[loss=0.07652, simple_loss=0.1085, pruned_loss=0.01234, audio_tagging_loss=0.009944, over 16237.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09294, pruned_loss=0.01406, audio_tagging_loss=0.009281, over 3046526.77 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:17:42,679 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351100
2023-11-23 10:17:56,852 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.74 vs. limit=10.0
2023-11-23 10:18:06,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0
2023-11-23 10:18:06,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2340733.3333333335, ans=0.1
2023-11-23 10:18:13,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2340800.0, ans=0.125
2023-11-23 10:18:13,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2340800.0, ans=0.0
2023-11-23 10:18:15,894 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.266e+01 8.725e+01 9.443e+01 1.197e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 10:18:22,952 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:18:29,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2340866.6666666665, ans=0.125
2023-11-23 10:18:34,659 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.53 vs. limit=10.0
2023-11-23 10:18:36,483 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2450, loss[loss=0.06723, simple_loss=0.08737, pruned_loss=0.01302, audio_tagging_loss=0.01053, over 14420.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09354, pruned_loss=0.01406, audio_tagging_loss=0.009281, over 3052895.52 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:18:44,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2340933.3333333335, ans=0.125
2023-11-23 10:18:45,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2340933.3333333335, ans=0.0
2023-11-23 10:18:46,593 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351150
2023-11-23 10:18:48,364 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.46 vs. limit=15.0
2023-11-23 10:18:53,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2341000.0, ans=0.125
2023-11-23 10:19:05,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2341066.6666666665, ans=0.0
2023-11-23 10:19:16,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.97 vs. limit=15.0
2023-11-23 10:19:37,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2341200.0, ans=0.125
2023-11-23 10:19:39,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2341200.0, ans=0.125
2023-11-23 10:19:40,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2341266.6666666665, ans=0.125
2023-11-23 10:19:41,847 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2500, loss[loss=0.05999, simple_loss=0.07738, pruned_loss=0.01173, audio_tagging_loss=0.009566, over 14561.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09283, pruned_loss=0.01377, audio_tagging_loss=0.009277, over 3047796.28 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:19:50,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2341266.6666666665, ans=10.0
2023-11-23 10:19:53,996 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351200
2023-11-23 10:19:55,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2341333.3333333335, ans=0.125
2023-11-23 10:20:27,908 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.304e+01 8.850e+01 9.688e+01 1.424e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 10:20:42,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2341533.3333333335, ans=0.1
2023-11-23 10:20:51,093 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2550, loss[loss=0.07339, simple_loss=0.09318, pruned_loss=0.01443, audio_tagging_loss=0.01237, over 15848.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09253, pruned_loss=0.01378, audio_tagging_loss=0.009211, over 3047448.43 frames. ], batch size: 61, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:21:01,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351250
2023-11-23 10:21:14,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2341666.6666666665, ans=15.0
2023-11-23 10:21:24,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.10 vs. limit=15.0
2023-11-23 10:21:37,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2341800.0, ans=0.125
2023-11-23 10:21:38,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2341800.0, ans=0.0
2023-11-23 10:21:47,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2341866.6666666665, ans=0.125
2023-11-23 10:21:49,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.11 vs. limit=15.0
2023-11-23 10:21:53,614 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.93 vs. limit=12.0
2023-11-23 10:21:56,982 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2600, loss[loss=0.06679, simple_loss=0.08384, pruned_loss=0.01484, audio_tagging_loss=0.01003, over 15252.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09211, pruned_loss=0.01377, audio_tagging_loss=0.009142, over 3048467.97 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:22:06,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351300
2023-11-23 10:22:18,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2342000.0, ans=0.125
2023-11-23 10:22:22,564 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:22:40,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2342133.3333333335, ans=0.125
2023-11-23 10:22:42,339 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.380e+01 9.027e+01 9.591e+01 1.211e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 10:22:47,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2342133.3333333335, ans=0.035
2023-11-23 10:23:02,631 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2650, loss[loss=0.08168, simple_loss=0.1076, pruned_loss=0.01995, audio_tagging_loss=0.007934, over 15919.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.0918, pruned_loss=0.01377, audio_tagging_loss=0.009049, over 3051895.79 frames. ], batch size: 61, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:23:14,172 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351350
2023-11-23 10:23:15,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2342333.3333333335, ans=0.125
2023-11-23 10:23:54,037 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.23 vs. limit=15.0
2023-11-23 10:23:58,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.14 vs. limit=6.0
2023-11-23 10:24:09,548 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2700, loss[loss=0.04644, simple_loss=0.05452, pruned_loss=0.007381, audio_tagging_loss=0.0118, over 14567.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09075, pruned_loss=0.0135, audio_tagging_loss=0.009076, over 3050569.10 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:24:09,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2342600.0, ans=0.0
2023-11-23 10:24:11,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.64 vs. limit=22.5
2023-11-23 10:24:15,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.87 vs. limit=15.0
2023-11-23 10:24:17,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2342600.0, ans=0.125
2023-11-23 10:24:18,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2342600.0, ans=0.0
2023-11-23 10:24:20,920 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351400
2023-11-23 10:24:39,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2342733.3333333335, ans=0.0
2023-11-23 10:24:45,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2342733.3333333335, ans=0.2
2023-11-23 10:24:54,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2342800.0, ans=0.2
2023-11-23 10:24:55,398 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.154e+01 8.358e+01 8.964e+01 9.971e+01 1.230e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 10:25:15,062 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2750, loss[loss=0.0581, simple_loss=0.06991, pruned_loss=0.01138, audio_tagging_loss=0.01177, over 15333.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09082, pruned_loss=0.01362, audio_tagging_loss=0.009021, over 3052677.66 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:25:24,977 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351450
2023-11-23 10:25:48,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2343066.6666666665, ans=0.035
2023-11-23 10:26:12,091 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:26:19,251 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2800, loss[loss=0.08007, simple_loss=0.114, pruned_loss=0.01626, audio_tagging_loss=0.006791, over 15962.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.0904, pruned_loss=0.0135, audio_tagging_loss=0.009035, over 3052651.91 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:26:30,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351500
2023-11-23 10:26:35,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2343333.3333333335, ans=0.07
2023-11-23 10:26:50,018 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:26:53,237 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.47 vs. limit=15.0
2023-11-23 10:27:05,086 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.185e+01 8.850e+01 9.511e+01 1.118e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 10:27:05,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2343466.6666666665, ans=0.125
2023-11-23 10:27:18,661 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-23 10:27:24,906 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2850, loss[loss=0.06689, simple_loss=0.08589, pruned_loss=0.01564, audio_tagging_loss=0.008297, over 15383.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09078, pruned_loss=0.01357, audio_tagging_loss=0.008971, over 3049264.99 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:27:35,421 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351550
2023-11-23 10:27:36,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2343666.6666666665, ans=0.0
2023-11-23 10:27:52,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2343733.3333333335, ans=0.125
2023-11-23 10:27:57,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0
2023-11-23 10:28:29,908 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2900, loss[loss=0.0678, simple_loss=0.0981, pruned_loss=0.01214, audio_tagging_loss=0.006598, over 14686.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09116, pruned_loss=0.01364, audio_tagging_loss=0.009049, over 3044791.78 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:28:31,950 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=22.5
2023-11-23 10:28:35,046 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.27 vs. limit=10.0
2023-11-23 10:28:40,805 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351600
2023-11-23 10:28:40,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2343933.3333333335, ans=0.125
2023-11-23 10:29:17,056 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.374e+01 8.898e+01 9.788e+01 1.211e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 10:29:31,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2344200.0, ans=0.125
2023-11-23 10:29:35,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2344266.6666666665, ans=0.1
2023-11-23 10:29:36,340 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 2950, loss[loss=0.04323, simple_loss=0.05592, pruned_loss=0.005315, audio_tagging_loss=0.009954, over 14402.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09246, pruned_loss=0.01394, audio_tagging_loss=0.008961, over 3044409.29 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:29:36,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2344266.6666666665, ans=0.125
2023-11-23 10:29:39,096 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:29:40,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2344266.6666666665, ans=0.0
2023-11-23 10:29:47,252 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351650
2023-11-23 10:29:54,705 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.21 vs. limit=15.0
2023-11-23 10:29:57,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.27 vs. limit=15.0
2023-11-23 10:30:25,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2344466.6666666665, ans=0.125
2023-11-23 10:30:32,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2344533.3333333335, ans=0.07
2023-11-23 10:30:42,040 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3000, loss[loss=0.07874, simple_loss=0.1003, pruned_loss=0.0178, audio_tagging_loss=0.01081, over 14936.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09186, pruned_loss=0.01372, audio_tagging_loss=0.009054, over 3043227.43 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:30:42,041 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 10:30:57,530 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6757, 4.2968, 2.8321, 3.9481], device='cuda:2')
2023-11-23 10:31:05,714 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8104, 4.9439, 5.0519, 4.8818], device='cuda:2')
2023-11-23 10:31:10,583 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9542, 3.7570, 4.9468, 4.3983], device='cuda:2')
2023-11-23 10:31:20,387 INFO [train_asr.py:1253] (2/4) Epoch 30, validation: loss=0.05789, simple_loss=0.05111, pruned_loss=0.005034, audio_tagging_loss=0.0273, over 4681554.00 frames. 
2023-11-23 10:31:20,387 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 10:31:31,331 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351700
2023-11-23 10:32:03,363 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.18 vs. limit=22.5
2023-11-23 10:32:08,863 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.435e+01 9.100e+01 1.011e+02 1.193e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-23 10:32:21,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2344866.6666666665, ans=0.0
2023-11-23 10:32:25,965 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3050, loss[loss=0.07475, simple_loss=0.1049, pruned_loss=0.01466, audio_tagging_loss=0.007646, over 14980.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.0926, pruned_loss=0.01394, audio_tagging_loss=0.009101, over 3047220.50 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:32:37,493 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351750
2023-11-23 10:32:38,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2345000.0, ans=0.05
2023-11-23 10:32:46,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2345000.0, ans=0.2
2023-11-23 10:33:06,557 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:33:09,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2345133.3333333335, ans=0.2
2023-11-23 10:33:16,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2345133.3333333335, ans=0.0
2023-11-23 10:33:32,033 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3100, loss[loss=0.07112, simple_loss=0.09289, pruned_loss=0.01476, audio_tagging_loss=0.009918, over 15426.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09324, pruned_loss=0.01416, audio_tagging_loss=0.00919, over 3054276.98 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:33:33,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2345266.6666666665, ans=0.125
2023-11-23 10:33:33,978 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.49 vs. limit=12.0
2023-11-23 10:33:39,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2345266.6666666665, ans=0.1
2023-11-23 10:33:42,778 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351800
2023-11-23 10:34:21,253 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.167e+01 8.412e+01 9.012e+01 9.624e+01 1.358e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 10:34:31,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2345533.3333333335, ans=0.1
2023-11-23 10:34:32,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2345533.3333333335, ans=0.125
2023-11-23 10:34:34,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2345533.3333333335, ans=0.125
2023-11-23 10:34:38,155 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3150, loss[loss=0.06843, simple_loss=0.08414, pruned_loss=0.01486, audio_tagging_loss=0.0115, over 14781.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09346, pruned_loss=0.01422, audio_tagging_loss=0.009185, over 3059059.04 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:34:48,288 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351850
2023-11-23 10:34:48,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2345600.0, ans=0.125
2023-11-23 10:34:49,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2345666.6666666665, ans=0.0
2023-11-23 10:35:10,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2345733.3333333335, ans=0.0
2023-11-23 10:35:11,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2345733.3333333335, ans=10.0
2023-11-23 10:35:19,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2345800.0, ans=0.125
2023-11-23 10:35:38,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2345866.6666666665, ans=0.0
2023-11-23 10:35:43,473 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3200, loss[loss=0.06632, simple_loss=0.09016, pruned_loss=0.01487, audio_tagging_loss=0.006374, over 14684.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09293, pruned_loss=0.014, audio_tagging_loss=0.009253, over 3053914.58 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:35:54,412 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351900
2023-11-23 10:35:56,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.45 vs. limit=8.0
2023-11-23 10:35:58,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2346000.0, ans=0.125
2023-11-23 10:36:31,821 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.179e+01 8.758e+01 9.576e+01 1.199e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 10:36:36,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2346200.0, ans=0.125
2023-11-23 10:36:49,764 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3250, loss[loss=0.07321, simple_loss=0.08631, pruned_loss=0.01575, audio_tagging_loss=0.0143, over 16193.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09216, pruned_loss=0.01377, audio_tagging_loss=0.00937, over 3057482.77 frames. ], batch size: 63, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:37:00,261 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 351950
2023-11-23 10:37:30,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2346466.6666666665, ans=0.125
2023-11-23 10:37:54,528 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3300, loss[loss=0.08733, simple_loss=0.1133, pruned_loss=0.02106, audio_tagging_loss=0.009595, over 15757.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09171, pruned_loss=0.01374, audio_tagging_loss=0.009378, over 3050179.20 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:37:59,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2346600.0, ans=0.0
2023-11-23 10:38:04,401 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352000
2023-11-23 10:38:13,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2346666.6666666665, ans=0.1
2023-11-23 10:38:19,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2346666.6666666665, ans=0.125
2023-11-23 10:38:35,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.41 vs. limit=22.5
2023-11-23 10:38:46,036 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.301e+01 8.918e+01 9.607e+01 1.152e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 10:38:49,262 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.20 vs. limit=15.0
2023-11-23 10:38:56,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2346866.6666666665, ans=0.2
2023-11-23 10:38:56,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.02 vs. limit=15.0
2023-11-23 10:39:02,155 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3350, loss[loss=0.07932, simple_loss=0.1058, pruned_loss=0.02036, audio_tagging_loss=0.006055, over 14701.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09292, pruned_loss=0.01393, audio_tagging_loss=0.009234, over 3046526.65 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:39:02,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2346933.3333333335, ans=0.2
2023-11-23 10:39:03,097 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.33 vs. limit=15.0
2023-11-23 10:39:12,925 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352050
2023-11-23 10:39:13,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.76 vs. limit=15.0
2023-11-23 10:39:31,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2347066.6666666665, ans=0.125
2023-11-23 10:39:43,714 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-23 10:40:00,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2347200.0, ans=0.125
2023-11-23 10:40:08,306 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3400, loss[loss=0.0595, simple_loss=0.07224, pruned_loss=0.01249, audio_tagging_loss=0.01089, over 15033.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09234, pruned_loss=0.01379, audio_tagging_loss=0.009093, over 3049841.83 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:40:19,051 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352100
2023-11-23 10:40:19,706 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0
2023-11-23 10:40:46,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2347466.6666666665, ans=0.1
2023-11-23 10:40:46,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2347466.6666666665, ans=0.125
2023-11-23 10:40:48,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2347466.6666666665, ans=0.0
2023-11-23 10:40:50,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2347466.6666666665, ans=0.2
2023-11-23 10:40:55,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2347466.6666666665, ans=0.125
2023-11-23 10:40:56,360 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.131e+01 8.875e+01 9.563e+01 1.133e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 10:40:57,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2347466.6666666665, ans=0.2
2023-11-23 10:41:12,941 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3450, loss[loss=0.08221, simple_loss=0.09593, pruned_loss=0.02255, audio_tagging_loss=0.01169, over 15393.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09257, pruned_loss=0.01375, audio_tagging_loss=0.009067, over 3058291.60 frames. ], batch size: 61, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:41:23,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352150
2023-11-23 10:41:23,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2347600.0, ans=0.125
2023-11-23 10:41:31,692 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:41:35,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2347666.6666666665, ans=0.1
2023-11-23 10:41:55,794 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:41:57,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.08 vs. limit=10.0
2023-11-23 10:42:01,513 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.75 vs. limit=10.0
2023-11-23 10:42:02,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2347800.0, ans=0.125
2023-11-23 10:42:15,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2347933.3333333335, ans=0.125
2023-11-23 10:42:16,595 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3500, loss[loss=0.07126, simple_loss=0.09757, pruned_loss=0.0155, audio_tagging_loss=0.006975, over 15489.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09171, pruned_loss=0.0136, audio_tagging_loss=0.008965, over 3052028.76 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:42:17,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.35 vs. limit=22.5
2023-11-23 10:42:26,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352200
2023-11-23 10:42:41,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2348066.6666666665, ans=0.1
2023-11-23 10:42:49,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2348066.6666666665, ans=0.0
2023-11-23 10:42:52,033 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:43:04,281 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.164e+01 8.763e+01 9.572e+01 1.144e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-23 10:43:16,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2348200.0, ans=0.125
2023-11-23 10:43:16,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2348200.0, ans=0.1
2023-11-23 10:43:20,958 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3550, loss[loss=0.07615, simple_loss=0.09439, pruned_loss=0.01881, audio_tagging_loss=0.01014, over 14064.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09108, pruned_loss=0.01359, audio_tagging_loss=0.009021, over 3054236.99 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:43:27,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2348266.6666666665, ans=0.125
2023-11-23 10:43:31,902 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352250
2023-11-23 10:43:40,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.63 vs. limit=15.0
2023-11-23 10:43:57,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2348400.0, ans=0.1
2023-11-23 10:44:00,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.49 vs. limit=22.5
2023-11-23 10:44:12,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2348533.3333333335, ans=0.125
2023-11-23 10:44:12,955 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.03 vs. limit=15.0
2023-11-23 10:44:16,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.36 vs. limit=15.0
2023-11-23 10:44:18,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2348533.3333333335, ans=0.125
2023-11-23 10:44:25,955 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3600, loss[loss=0.07135, simple_loss=0.09265, pruned_loss=0.01526, audio_tagging_loss=0.009768, over 15268.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09083, pruned_loss=0.01359, audio_tagging_loss=0.009026, over 3051013.54 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:44:35,750 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352300
2023-11-23 10:44:44,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2348666.6666666665, ans=0.0
2023-11-23 10:44:49,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2348733.3333333335, ans=0.125
2023-11-23 10:44:50,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2348733.3333333335, ans=0.125
2023-11-23 10:44:52,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2348733.3333333335, ans=0.125
2023-11-23 10:45:08,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2348800.0, ans=0.05
2023-11-23 10:45:13,765 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.915e+01 8.138e+01 8.784e+01 9.716e+01 1.349e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 10:45:15,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2348800.0, ans=0.1
2023-11-23 10:45:20,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2348866.6666666665, ans=0.0
2023-11-23 10:45:27,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2348866.6666666665, ans=0.125
2023-11-23 10:45:29,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.36 vs. limit=15.0
2023-11-23 10:45:29,834 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3650, loss[loss=0.07791, simple_loss=0.1085, pruned_loss=0.01401, audio_tagging_loss=0.009634, over 15602.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09188, pruned_loss=0.01396, audio_tagging_loss=0.009042, over 3055817.56 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:45:39,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352350
2023-11-23 10:45:40,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2348933.3333333335, ans=0.2
2023-11-23 10:45:40,629 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.57 vs. limit=15.0
2023-11-23 10:45:45,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2349000.0, ans=0.125
2023-11-23 10:45:51,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2349000.0, ans=0.0
2023-11-23 10:46:11,476 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:46:21,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2349200.0, ans=0.0
2023-11-23 10:46:34,669 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3700, loss[loss=0.04984, simple_loss=0.06659, pruned_loss=0.009381, audio_tagging_loss=0.007162, over 14853.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09126, pruned_loss=0.01376, audio_tagging_loss=0.008975, over 3056996.82 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:46:35,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2349266.6666666665, ans=0.1
2023-11-23 10:46:36,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2349266.6666666665, ans=0.1
2023-11-23 10:46:46,493 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352400
2023-11-23 10:47:19,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.23 vs. limit=15.0
2023-11-23 10:47:23,650 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.475e+01 8.984e+01 9.756e+01 1.281e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 10:47:42,634 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3750, loss[loss=0.07174, simple_loss=0.09482, pruned_loss=0.01483, audio_tagging_loss=0.009496, over 15315.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09159, pruned_loss=0.01382, audio_tagging_loss=0.008988, over 3057320.10 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:47:44,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2349600.0, ans=0.125
2023-11-23 10:47:49,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2349600.0, ans=0.125
2023-11-23 10:47:49,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2349600.0, ans=0.0
2023-11-23 10:47:53,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352450
2023-11-23 10:48:09,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2349733.3333333335, ans=0.0
2023-11-23 10:48:21,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2349800.0, ans=0.0
2023-11-23 10:48:31,267 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:48:44,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2349866.6666666665, ans=0.0
2023-11-23 10:48:46,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.06 vs. limit=15.0
2023-11-23 10:48:49,436 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3800, loss[loss=0.0649, simple_loss=0.0825, pruned_loss=0.0104, audio_tagging_loss=0.01325, over 15256.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.0924, pruned_loss=0.01384, audio_tagging_loss=0.009035, over 3065869.85 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:48:53,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2349933.3333333335, ans=0.125
2023-11-23 10:48:55,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2349933.3333333335, ans=0.0
2023-11-23 10:48:59,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352500
2023-11-23 10:49:06,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2350000.0, ans=0.1
2023-11-23 10:49:38,108 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.465e+01 8.919e+01 9.665e+01 1.243e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 10:49:41,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2350200.0, ans=0.2
2023-11-23 10:49:44,088 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.90 vs. limit=15.0
2023-11-23 10:49:46,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2350200.0, ans=0.0
2023-11-23 10:49:54,997 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3850, loss[loss=0.06946, simple_loss=0.09622, pruned_loss=0.01342, audio_tagging_loss=0.007932, over 15457.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09176, pruned_loss=0.01375, audio_tagging_loss=0.009179, over 3060610.95 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:50:06,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352550
2023-11-23 10:50:09,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2350333.3333333335, ans=0.1
2023-11-23 10:51:02,822 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3900, loss[loss=0.0719, simple_loss=0.09011, pruned_loss=0.01487, audio_tagging_loss=0.01198, over 14058.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09177, pruned_loss=0.01391, audio_tagging_loss=0.009237, over 3049112.52 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:51:10,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2350600.0, ans=0.07
2023-11-23 10:51:14,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352600
2023-11-23 10:51:20,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2350666.6666666665, ans=0.1
2023-11-23 10:51:24,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2350666.6666666665, ans=0.125
2023-11-23 10:51:32,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2350733.3333333335, ans=0.04949747468305833
2023-11-23 10:51:35,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2350733.3333333335, ans=0.125
2023-11-23 10:51:51,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2350800.0, ans=0.125
2023-11-23 10:51:55,080 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.310e+01 8.820e+01 9.647e+01 1.440e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 10:52:06,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2350866.6666666665, ans=0.125
2023-11-23 10:52:10,481 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 3950, loss[loss=0.06262, simple_loss=0.07401, pruned_loss=0.01089, audio_tagging_loss=0.01472, over 14309.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09185, pruned_loss=0.01402, audio_tagging_loss=0.009321, over 3051883.52 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:52:20,606 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352650
2023-11-23 10:52:31,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2351000.0, ans=0.125
2023-11-23 10:52:37,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2351066.6666666665, ans=0.0
2023-11-23 10:52:46,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.32 vs. limit=22.5
2023-11-23 10:52:48,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2351066.6666666665, ans=0.2
2023-11-23 10:53:16,302 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4000, loss[loss=0.06006, simple_loss=0.08277, pruned_loss=0.00967, audio_tagging_loss=0.009005, over 15086.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09203, pruned_loss=0.0141, audio_tagging_loss=0.009376, over 3049016.87 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:53:27,807 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352700
2023-11-23 10:53:30,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2351333.3333333335, ans=0.09899494936611666
2023-11-23 10:53:54,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2351400.0, ans=0.125
2023-11-23 10:53:55,141 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.50 vs. limit=15.0
2023-11-23 10:54:07,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2351466.6666666665, ans=0.125
2023-11-23 10:54:08,350 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.731e+01 8.440e+01 9.076e+01 9.924e+01 2.102e+02, threshold=1.815e+02, percent-clipped=1.0
2023-11-23 10:54:22,803 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:54:23,937 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4050, loss[loss=0.07749, simple_loss=0.09592, pruned_loss=0.02074, audio_tagging_loss=0.008791, over 14626.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.0923, pruned_loss=0.01407, audio_tagging_loss=0.00941, over 3049627.94 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:54:27,756 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:54:34,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352750
2023-11-23 10:54:37,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2351666.6666666665, ans=0.125
2023-11-23 10:54:45,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.05 vs. limit=15.0
2023-11-23 10:54:51,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2351733.3333333335, ans=0.1
2023-11-23 10:54:52,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2351733.3333333335, ans=0.05
2023-11-23 10:54:52,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2351733.3333333335, ans=0.1
2023-11-23 10:55:13,196 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.05 vs. limit=15.0
2023-11-23 10:55:28,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2351866.6666666665, ans=0.125
2023-11-23 10:55:30,756 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4100, loss[loss=0.0619, simple_loss=0.07745, pruned_loss=0.01368, audio_tagging_loss=0.009493, over 14771.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09239, pruned_loss=0.01411, audio_tagging_loss=0.009339, over 3044784.00 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:55:41,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352800
2023-11-23 10:55:51,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2352000.0, ans=0.125
2023-11-23 10:55:54,595 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.41 vs. limit=15.0
2023-11-23 10:55:57,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2352066.6666666665, ans=0.025
2023-11-23 10:56:07,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2352066.6666666665, ans=0.125
2023-11-23 10:56:15,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2352133.3333333335, ans=0.125
2023-11-23 10:56:20,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2352133.3333333335, ans=0.125
2023-11-23 10:56:22,981 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.387e+01 8.886e+01 9.725e+01 1.263e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 10:56:37,354 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4150, loss[loss=0.0665, simple_loss=0.0858, pruned_loss=0.01323, audio_tagging_loss=0.01037, over 14365.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09249, pruned_loss=0.01414, audio_tagging_loss=0.009243, over 3040906.28 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:56:47,998 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352850
2023-11-23 10:56:48,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2352266.6666666665, ans=0.125
2023-11-23 10:56:49,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2352333.3333333335, ans=0.2
2023-11-23 10:57:03,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2352400.0, ans=22.5
2023-11-23 10:57:07,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.75 vs. limit=15.0
2023-11-23 10:57:25,906 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:57:26,811 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.81 vs. limit=5.0
2023-11-23 10:57:43,203 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4200, loss[loss=0.07269, simple_loss=0.1017, pruned_loss=0.01506, audio_tagging_loss=0.006758, over 15122.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09244, pruned_loss=0.01404, audio_tagging_loss=0.00913, over 3044962.92 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:57:51,626 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:57:52,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2352600.0, ans=0.0
2023-11-23 10:57:53,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352900
2023-11-23 10:57:57,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2352666.6666666665, ans=0.125
2023-11-23 10:58:12,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2352733.3333333335, ans=0.125
2023-11-23 10:58:14,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2352733.3333333335, ans=0.125
2023-11-23 10:58:17,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2352733.3333333335, ans=0.0
2023-11-23 10:58:34,391 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.262e+01 8.280e+01 9.153e+01 9.878e+01 1.174e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 10:58:35,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=2352866.6666666665, ans=15.0
2023-11-23 10:58:39,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2352866.6666666665, ans=0.125
2023-11-23 10:58:48,824 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4250, loss[loss=0.08029, simple_loss=0.1092, pruned_loss=0.01677, audio_tagging_loss=0.008941, over 15689.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09232, pruned_loss=0.01401, audio_tagging_loss=0.009021, over 3050665.46 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:58:58,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 352950
2023-11-23 10:59:32,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2353133.3333333335, ans=0.125
2023-11-23 10:59:38,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2353133.3333333335, ans=0.125
2023-11-23 10:59:54,231 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4300, loss[loss=0.06911, simple_loss=0.09327, pruned_loss=0.01486, audio_tagging_loss=0.00762, over 15233.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09254, pruned_loss=0.01401, audio_tagging_loss=0.008998, over 3056246.01 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:00:05,046 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353000
2023-11-23 11:00:16,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.36 vs. limit=12.0
2023-11-23 11:00:23,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2353400.0, ans=22.5
2023-11-23 11:00:24,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2353400.0, ans=0.125
2023-11-23 11:00:28,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2353400.0, ans=0.07
2023-11-23 11:00:38,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2353466.6666666665, ans=0.125
2023-11-23 11:00:45,968 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.099e+01 8.412e+01 9.137e+01 9.657e+01 1.208e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 11:00:47,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2353533.3333333335, ans=0.1
2023-11-23 11:00:48,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2353533.3333333335, ans=0.025
2023-11-23 11:00:52,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2353533.3333333335, ans=0.125
2023-11-23 11:01:01,192 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4350, loss[loss=0.05882, simple_loss=0.0727, pruned_loss=0.01148, audio_tagging_loss=0.01098, over 14726.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09259, pruned_loss=0.0141, audio_tagging_loss=0.009003, over 3052748.75 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:01:11,777 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353050
2023-11-23 11:01:48,912 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.40 vs. limit=22.5
2023-11-23 11:01:59,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2353866.6666666665, ans=0.95
2023-11-23 11:02:07,046 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4400, loss[loss=0.06319, simple_loss=0.08235, pruned_loss=0.01169, audio_tagging_loss=0.01032, over 16012.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09318, pruned_loss=0.01423, audio_tagging_loss=0.008907, over 3049713.98 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:02:17,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353100
2023-11-23 11:02:22,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2354000.0, ans=0.0
2023-11-23 11:02:26,861 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.87 vs. limit=12.0
2023-11-23 11:02:36,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2354066.6666666665, ans=0.2
2023-11-23 11:02:37,871 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.59 vs. limit=12.0
2023-11-23 11:02:53,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2354133.3333333335, ans=0.0
2023-11-23 11:02:58,602 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.258e+01 8.734e+01 9.523e+01 1.170e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-23 11:03:01,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2354200.0, ans=0.0
2023-11-23 11:03:12,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.33 vs. limit=22.5
2023-11-23 11:03:12,652 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4450, loss[loss=0.0824, simple_loss=0.1083, pruned_loss=0.0193, audio_tagging_loss=0.008962, over 16160.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09448, pruned_loss=0.01431, audio_tagging_loss=0.00878, over 3052066.23 frames. ], batch size: 61, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:03:23,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353150
2023-11-23 11:04:07,886 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:04:08,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.96 vs. limit=15.0
2023-11-23 11:04:18,929 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4500, loss[loss=0.0646, simple_loss=0.08526, pruned_loss=0.01282, audio_tagging_loss=0.009157, over 14965.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09393, pruned_loss=0.01415, audio_tagging_loss=0.008768, over 3048327.37 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:04:26,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2354600.0, ans=0.0
2023-11-23 11:04:29,727 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353200
2023-11-23 11:04:36,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2354666.6666666665, ans=0.1
2023-11-23 11:04:38,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-23 11:04:53,325 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.74 vs. limit=15.0
2023-11-23 11:05:10,689 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.361e+01 8.894e+01 9.678e+01 1.285e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 11:05:25,307 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4550, loss[loss=0.07775, simple_loss=0.1111, pruned_loss=0.01647, audio_tagging_loss=0.005706, over 15722.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09457, pruned_loss=0.01422, audio_tagging_loss=0.008674, over 3043880.52 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:05:35,579 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353250
2023-11-23 11:05:35,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2354933.3333333335, ans=0.125
2023-11-23 11:05:40,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2355000.0, ans=0.0
2023-11-23 11:06:11,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2355133.3333333335, ans=0.125
2023-11-23 11:06:11,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2355133.3333333335, ans=0.125
2023-11-23 11:06:16,955 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:06:30,712 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4600, loss[loss=0.05657, simple_loss=0.07407, pruned_loss=0.009486, audio_tagging_loss=0.01005, over 15407.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09298, pruned_loss=0.01405, audio_tagging_loss=0.008806, over 3038337.08 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:06:31,449 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.68 vs. limit=22.5
2023-11-23 11:06:33,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2355266.6666666665, ans=0.2
2023-11-23 11:06:40,932 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353300
2023-11-23 11:06:57,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2355400.0, ans=0.125
2023-11-23 11:06:59,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2355400.0, ans=0.125
2023-11-23 11:07:08,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2355400.0, ans=0.1
2023-11-23 11:07:10,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2355466.6666666665, ans=0.125
2023-11-23 11:07:11,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.75 vs. limit=6.0
2023-11-23 11:07:20,885 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.46 vs. limit=15.0
2023-11-23 11:07:22,702 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.374e+01 9.113e+01 9.738e+01 1.636e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-23 11:07:25,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten.whitening_limit, batch_count=2355533.3333333335, ans=15.0
2023-11-23 11:07:35,294 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4650, loss[loss=0.05876, simple_loss=0.07864, pruned_loss=0.009839, audio_tagging_loss=0.009599, over 15974.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09314, pruned_loss=0.01408, audio_tagging_loss=0.008928, over 3042812.48 frames. ], batch size: 61, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:07:47,175 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353350
2023-11-23 11:07:59,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2355666.6666666665, ans=0.125
2023-11-23 11:08:22,286 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.54 vs. limit=15.0
2023-11-23 11:08:27,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2355866.6666666665, ans=0.0
2023-11-23 11:08:39,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.82 vs. limit=15.0
2023-11-23 11:08:42,559 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4700, loss[loss=0.08673, simple_loss=0.1143, pruned_loss=0.01994, audio_tagging_loss=0.009645, over 15543.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09239, pruned_loss=0.01401, audio_tagging_loss=0.008982, over 3040269.41 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:08:52,634 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353400
2023-11-23 11:09:05,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2356000.0, ans=0.0
2023-11-23 11:09:08,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2356066.6666666665, ans=0.5
2023-11-23 11:09:08,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2356066.6666666665, ans=0.0
2023-11-23 11:09:13,722 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.26 vs. limit=15.0
2023-11-23 11:09:23,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2356133.3333333335, ans=0.2
2023-11-23 11:09:30,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2356133.3333333335, ans=0.0
2023-11-23 11:09:33,276 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.44 vs. limit=22.5
2023-11-23 11:09:35,055 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.186e+01 8.702e+01 9.591e+01 1.216e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-23 11:09:47,644 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4750, loss[loss=0.05534, simple_loss=0.07167, pruned_loss=0.01064, audio_tagging_loss=0.008868, over 13845.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09125, pruned_loss=0.01367, audio_tagging_loss=0.009151, over 3032355.49 frames. ], batch size: 52, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:09:55,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2356266.6666666665, ans=0.0
2023-11-23 11:09:57,683 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353450
2023-11-23 11:09:57,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2356266.6666666665, ans=0.2
2023-11-23 11:10:01,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2356333.3333333335, ans=0.125
2023-11-23 11:10:22,849 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.62 vs. limit=10.0
2023-11-23 11:10:39,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2356533.3333333335, ans=0.2
2023-11-23 11:10:52,102 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4800, loss[loss=0.07593, simple_loss=0.1069, pruned_loss=0.01343, audio_tagging_loss=0.009066, over 16017.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09127, pruned_loss=0.01365, audio_tagging_loss=0.009266, over 3034239.01 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:11:04,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353500
2023-11-23 11:11:32,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2356800.0, ans=0.125
2023-11-23 11:11:39,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.77 vs. limit=12.0
2023-11-23 11:11:44,943 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.167e+01 8.824e+01 9.550e+01 1.189e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 11:11:57,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2356866.6666666665, ans=0.125
2023-11-23 11:11:59,397 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4850, loss[loss=0.08025, simple_loss=0.09288, pruned_loss=0.01899, audio_tagging_loss=0.01482, over 15352.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09157, pruned_loss=0.01358, audio_tagging_loss=0.009395, over 3035297.80 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:12:10,072 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353550
2023-11-23 11:12:11,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2357000.0, ans=0.0
2023-11-23 11:12:25,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2357066.6666666665, ans=0.125
2023-11-23 11:12:27,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2357066.6666666665, ans=0.2
2023-11-23 11:12:56,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2357200.0, ans=0.125
2023-11-23 11:12:56,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2357200.0, ans=0.125
2023-11-23 11:13:04,996 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4900, loss[loss=0.07277, simple_loss=0.08923, pruned_loss=0.01695, audio_tagging_loss=0.01121, over 15535.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09243, pruned_loss=0.01391, audio_tagging_loss=0.009346, over 3036326.89 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:13:09,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2357266.6666666665, ans=0.0
2023-11-23 11:13:15,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353600
2023-11-23 11:13:32,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2357400.0, ans=0.125
2023-11-23 11:13:54,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2357466.6666666665, ans=0.0
2023-11-23 11:13:57,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.207e+01 8.666e+01 9.499e+01 1.245e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-23 11:14:10,308 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 4950, loss[loss=0.06415, simple_loss=0.08359, pruned_loss=0.01371, audio_tagging_loss=0.008648, over 15177.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09185, pruned_loss=0.01377, audio_tagging_loss=0.009149, over 3031128.29 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:14:12,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2357600.0, ans=0.125
2023-11-23 11:14:21,802 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353650
2023-11-23 11:14:28,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2357666.6666666665, ans=0.125
2023-11-23 11:14:34,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2357666.6666666665, ans=0.0
2023-11-23 11:14:53,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2357800.0, ans=0.125
2023-11-23 11:15:16,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2357933.3333333335, ans=0.2
2023-11-23 11:15:17,634 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5000, loss[loss=0.04385, simple_loss=0.05139, pruned_loss=0.009945, audio_tagging_loss=0.008206, over 15327.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09232, pruned_loss=0.01371, audio_tagging_loss=0.008884, over 3033585.24 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:15:20,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2357933.3333333335, ans=0.1
2023-11-23 11:15:29,489 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353700
2023-11-23 11:16:00,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.53 vs. limit=10.0
2023-11-23 11:16:03,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2358133.3333333335, ans=0.0
2023-11-23 11:16:04,993 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.22 vs. limit=10.0
2023-11-23 11:16:10,295 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.172e+01 8.851e+01 9.559e+01 1.178e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 11:16:16,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2358200.0, ans=0.0
2023-11-23 11:16:24,077 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5050, loss[loss=0.05451, simple_loss=0.06891, pruned_loss=0.01102, audio_tagging_loss=0.009039, over 16267.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09227, pruned_loss=0.01357, audio_tagging_loss=0.008908, over 3039689.35 frames. ], batch size: 62, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:16:34,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353750
2023-11-23 11:16:43,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2358333.3333333335, ans=0.125
2023-11-23 11:16:46,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2358333.3333333335, ans=0.2
2023-11-23 11:16:50,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2358400.0, ans=0.0
2023-11-23 11:17:19,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2358533.3333333335, ans=0.125
2023-11-23 11:17:26,035 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:17:29,508 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5100, loss[loss=0.08259, simple_loss=0.105, pruned_loss=0.01632, audio_tagging_loss=0.01378, over 14986.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09262, pruned_loss=0.01363, audio_tagging_loss=0.008949, over 3044769.94 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:17:32,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2358600.0, ans=0.0
2023-11-23 11:17:40,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353800
2023-11-23 11:17:49,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2358666.6666666665, ans=0.0
2023-11-23 11:18:03,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2358733.3333333335, ans=0.0
2023-11-23 11:18:19,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2358800.0, ans=0.1
2023-11-23 11:18:23,224 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.214e+01 8.798e+01 9.658e+01 1.127e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-23 11:18:35,635 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5150, loss[loss=0.06071, simple_loss=0.08681, pruned_loss=0.009771, audio_tagging_loss=0.007537, over 14999.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09276, pruned_loss=0.01354, audio_tagging_loss=0.008956, over 3041821.55 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:18:35,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2358933.3333333335, ans=0.0
2023-11-23 11:18:46,395 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353850
2023-11-23 11:18:46,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2358933.3333333335, ans=0.125
2023-11-23 11:18:47,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2359000.0, ans=0.125
2023-11-23 11:18:50,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2359000.0, ans=0.0
2023-11-23 11:18:56,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.64 vs. limit=15.0
2023-11-23 11:19:03,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2359066.6666666665, ans=0.125
2023-11-23 11:19:12,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2359066.6666666665, ans=0.125
2023-11-23 11:19:15,619 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.14 vs. limit=22.5
2023-11-23 11:19:27,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2359200.0, ans=0.125
2023-11-23 11:19:39,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2359200.0, ans=0.2
2023-11-23 11:19:42,040 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5200, loss[loss=0.07093, simple_loss=0.08947, pruned_loss=0.01782, audio_tagging_loss=0.008373, over 15508.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09306, pruned_loss=0.01372, audio_tagging_loss=0.009038, over 3043354.44 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:19:47,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2359266.6666666665, ans=0.125
2023-11-23 11:19:52,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353900
2023-11-23 11:19:56,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2359333.3333333335, ans=0.07
2023-11-23 11:20:15,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.47 vs. limit=22.5
2023-11-23 11:20:32,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2359466.6666666665, ans=0.0
2023-11-23 11:20:33,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2359533.3333333335, ans=0.0
2023-11-23 11:20:36,393 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.445e+01 9.037e+01 9.922e+01 1.226e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 11:20:37,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.98 vs. limit=15.0
2023-11-23 11:20:43,601 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.47 vs. limit=10.0
2023-11-23 11:20:47,852 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5250, loss[loss=0.04553, simple_loss=0.05387, pruned_loss=0.007129, audio_tagging_loss=0.01147, over 15026.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09212, pruned_loss=0.01368, audio_tagging_loss=0.009006, over 3032933.82 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:20:53,569 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.68 vs. limit=15.0
2023-11-23 11:20:57,913 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 353950
2023-11-23 11:21:32,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0
2023-11-23 11:21:43,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2359866.6666666665, ans=0.0
2023-11-23 11:21:54,212 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5300, loss[loss=0.07193, simple_loss=0.09546, pruned_loss=0.01355, audio_tagging_loss=0.01065, over 15862.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09353, pruned_loss=0.01417, audio_tagging_loss=0.008944, over 3040794.66 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:22:04,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354000
2023-11-23 11:22:27,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2360066.6666666665, ans=0.125
2023-11-23 11:22:49,349 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.002e+01 8.452e+01 9.049e+01 9.694e+01 1.252e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 11:22:55,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2360200.0, ans=0.125
2023-11-23 11:22:59,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.31 vs. limit=12.0
2023-11-23 11:23:00,033 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5350, loss[loss=0.07674, simple_loss=0.1074, pruned_loss=0.01611, audio_tagging_loss=0.006908, over 15594.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.0934, pruned_loss=0.01427, audio_tagging_loss=0.008888, over 3039734.34 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:23:10,966 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354050
2023-11-23 11:23:24,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2360400.0, ans=0.2
2023-11-23 11:23:41,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2360466.6666666665, ans=0.5
2023-11-23 11:23:43,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2360466.6666666665, ans=0.2
2023-11-23 11:24:00,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.56 vs. limit=10.0
2023-11-23 11:24:01,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.44 vs. limit=15.0
2023-11-23 11:24:02,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2360533.3333333335, ans=0.1
2023-11-23 11:24:06,739 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5400, loss[loss=0.03313, simple_loss=0.03807, pruned_loss=0.003558, audio_tagging_loss=0.01054, over 13996.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09349, pruned_loss=0.0143, audio_tagging_loss=0.008895, over 3037079.99 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:24:16,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354100
2023-11-23 11:24:37,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2360733.3333333335, ans=0.125
2023-11-23 11:24:45,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-23 11:24:49,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2360800.0, ans=0.125
2023-11-23 11:24:54,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2360800.0, ans=0.1
2023-11-23 11:25:02,418 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.248e+01 8.837e+01 9.737e+01 1.214e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 11:25:12,642 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5450, loss[loss=0.07055, simple_loss=0.09178, pruned_loss=0.01545, audio_tagging_loss=0.009215, over 14392.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09333, pruned_loss=0.01426, audio_tagging_loss=0.009058, over 3041079.78 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:25:17,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2360933.3333333335, ans=0.1
2023-11-23 11:25:24,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354150
2023-11-23 11:25:28,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2361000.0, ans=0.125
2023-11-23 11:25:32,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2361000.0, ans=0.2
2023-11-23 11:25:44,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.20 vs. limit=22.5
2023-11-23 11:25:46,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.64 vs. limit=22.5
2023-11-23 11:25:55,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2361133.3333333335, ans=0.5
2023-11-23 11:25:55,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2361133.3333333335, ans=0.125
2023-11-23 11:25:58,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2361133.3333333335, ans=0.0
2023-11-23 11:26:19,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2361266.6666666665, ans=15.0
2023-11-23 11:26:19,493 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5500, loss[loss=0.06215, simple_loss=0.08477, pruned_loss=0.0108, audio_tagging_loss=0.008958, over 15725.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09264, pruned_loss=0.01409, audio_tagging_loss=0.009178, over 3048088.78 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:26:30,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354200
2023-11-23 11:26:37,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2361333.3333333335, ans=0.0
2023-11-23 11:26:51,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2361400.0, ans=0.125
2023-11-23 11:27:16,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.480e+01 8.474e+01 9.083e+01 9.895e+01 1.258e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 11:27:26,425 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5550, loss[loss=0.07697, simple_loss=0.1085, pruned_loss=0.01403, audio_tagging_loss=0.008685, over 15096.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09324, pruned_loss=0.01422, audio_tagging_loss=0.009232, over 3048398.93 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:27:37,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354250
2023-11-23 11:27:38,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2361666.6666666665, ans=0.0
2023-11-23 11:27:40,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2361666.6666666665, ans=0.125
2023-11-23 11:27:51,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2361666.6666666665, ans=0.125
2023-11-23 11:28:00,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2361733.3333333335, ans=0.0
2023-11-23 11:28:07,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2361800.0, ans=0.0
2023-11-23 11:28:32,569 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5600, loss[loss=0.05004, simple_loss=0.06179, pruned_loss=0.008252, audio_tagging_loss=0.01089, over 15661.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.0929, pruned_loss=0.01422, audio_tagging_loss=0.009263, over 3052291.86 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:28:36,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2361933.3333333335, ans=0.0
2023-11-23 11:28:43,743 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354300
2023-11-23 11:28:51,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2362000.0, ans=0.125
2023-11-23 11:28:58,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2362066.6666666665, ans=0.025
2023-11-23 11:29:21,275 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:29:29,325 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.942e+01 8.297e+01 9.078e+01 9.642e+01 1.395e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 11:29:29,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2362200.0, ans=0.125
2023-11-23 11:29:32,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2362200.0, ans=0.125
2023-11-23 11:29:35,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2362200.0, ans=0.125
2023-11-23 11:29:38,654 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5650, loss[loss=0.07673, simple_loss=0.108, pruned_loss=0.01486, audio_tagging_loss=0.007867, over 15362.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09381, pruned_loss=0.01434, audio_tagging_loss=0.009221, over 3048786.71 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:29:44,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2362266.6666666665, ans=0.2
2023-11-23 11:29:48,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2362266.6666666665, ans=0.125
2023-11-23 11:29:49,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354350
2023-11-23 11:30:44,119 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5700, loss[loss=0.0654, simple_loss=0.07927, pruned_loss=0.01583, audio_tagging_loss=0.009931, over 14880.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09295, pruned_loss=0.01425, audio_tagging_loss=0.009291, over 3053452.08 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:30:46,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2362600.0, ans=0.0
2023-11-23 11:30:54,330 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354400
2023-11-23 11:31:00,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2362666.6666666665, ans=0.125
2023-11-23 11:31:11,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2362733.3333333335, ans=0.0
2023-11-23 11:31:21,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2362733.3333333335, ans=0.125
2023-11-23 11:31:27,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2362800.0, ans=0.0
2023-11-23 11:31:36,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2362866.6666666665, ans=10.0
2023-11-23 11:31:40,367 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.224e+01 8.276e+01 8.914e+01 9.745e+01 1.403e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 11:31:46,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2362866.6666666665, ans=0.125
2023-11-23 11:31:49,760 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5750, loss[loss=0.06204, simple_loss=0.08581, pruned_loss=0.009224, audio_tagging_loss=0.009907, over 14952.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09302, pruned_loss=0.01425, audio_tagging_loss=0.009232, over 3045965.20 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:31:52,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2362933.3333333335, ans=0.0
2023-11-23 11:31:54,154 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.32 vs. limit=22.5
2023-11-23 11:32:01,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354450
2023-11-23 11:32:01,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2362933.3333333335, ans=0.07
2023-11-23 11:32:40,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2363133.3333333335, ans=0.1
2023-11-23 11:32:43,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2363200.0, ans=0.125
2023-11-23 11:32:56,515 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5800, loss[loss=0.04819, simple_loss=0.06246, pruned_loss=0.005974, audio_tagging_loss=0.01099, over 16110.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09335, pruned_loss=0.0142, audio_tagging_loss=0.009085, over 3047190.54 frames. ], batch size: 62, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:33:07,347 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354500
2023-11-23 11:33:08,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2363333.3333333335, ans=0.2
2023-11-23 11:33:13,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.74 vs. limit=22.5
2023-11-23 11:33:43,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2363466.6666666665, ans=0.2
2023-11-23 11:33:46,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2363466.6666666665, ans=0.1
2023-11-23 11:33:53,516 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.821e+01 8.322e+01 8.914e+01 9.568e+01 1.180e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 11:34:01,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2363600.0, ans=0.125
2023-11-23 11:34:02,440 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5850, loss[loss=0.08206, simple_loss=0.1117, pruned_loss=0.02032, audio_tagging_loss=0.005917, over 14646.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09304, pruned_loss=0.01424, audio_tagging_loss=0.009018, over 3041607.41 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:34:12,357 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354550
2023-11-23 11:34:17,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2363666.6666666665, ans=0.125
2023-11-23 11:34:19,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2363666.6666666665, ans=0.0
2023-11-23 11:34:21,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2363666.6666666665, ans=0.1
2023-11-23 11:34:37,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2363733.3333333335, ans=0.1
2023-11-23 11:34:41,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2363800.0, ans=0.0
2023-11-23 11:34:55,535 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.71 vs. limit=15.0
2023-11-23 11:35:00,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2363866.6666666665, ans=0.1
2023-11-23 11:35:06,287 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5900, loss[loss=0.07673, simple_loss=0.1107, pruned_loss=0.01411, audio_tagging_loss=0.007259, over 15124.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09403, pruned_loss=0.01432, audio_tagging_loss=0.008987, over 3052971.97 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:35:11,938 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.26 vs. limit=22.5
2023-11-23 11:35:17,680 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354600
2023-11-23 11:35:35,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2364066.6666666665, ans=0.0
2023-11-23 11:35:57,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.68 vs. limit=22.5
2023-11-23 11:36:02,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.354e+01 8.917e+01 9.513e+01 1.247e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 11:36:12,717 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 5950, loss[loss=0.07443, simple_loss=0.1042, pruned_loss=0.01459, audio_tagging_loss=0.007754, over 14985.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09406, pruned_loss=0.01428, audio_tagging_loss=0.008882, over 3060582.75 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:36:23,392 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354650
2023-11-23 11:36:32,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2364333.3333333335, ans=0.0
2023-11-23 11:36:32,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2364333.3333333335, ans=0.125
2023-11-23 11:36:34,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2364333.3333333335, ans=0.1
2023-11-23 11:36:39,871 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.18 vs. limit=22.5
2023-11-23 11:36:40,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2364400.0, ans=0.125
2023-11-23 11:36:51,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2364466.6666666665, ans=0.035
2023-11-23 11:37:17,284 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6000, loss[loss=0.06177, simple_loss=0.07939, pruned_loss=0.01168, audio_tagging_loss=0.0104, over 15084.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09293, pruned_loss=0.01405, audio_tagging_loss=0.00896, over 3056414.17 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:37:17,285 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 11:37:43,011 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2611, 4.2625, 4.4708, 4.4675], device='cuda:2')
2023-11-23 11:37:57,940 INFO [train_asr.py:1253] (2/4) Epoch 30, validation: loss=0.05791, simple_loss=0.05108, pruned_loss=0.005053, audio_tagging_loss=0.02732, over 4681554.00 frames. 
2023-11-23 11:37:57,941 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 11:38:08,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354700
2023-11-23 11:38:32,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.53 vs. limit=15.0
2023-11-23 11:38:38,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2364800.0, ans=0.125
2023-11-23 11:38:44,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2364800.0, ans=0.0
2023-11-23 11:38:45,070 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:38:52,923 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.297e+01 9.002e+01 9.719e+01 1.265e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 11:39:02,741 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6050, loss[loss=0.05838, simple_loss=0.07209, pruned_loss=0.009344, audio_tagging_loss=0.01299, over 15462.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09322, pruned_loss=0.014, audio_tagging_loss=0.008909, over 3052960.53 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:39:09,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2364933.3333333335, ans=0.1
2023-11-23 11:39:13,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354750
2023-11-23 11:39:33,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2365066.6666666665, ans=0.0
2023-11-23 11:40:07,149 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6100, loss[loss=0.06672, simple_loss=0.09079, pruned_loss=0.01129, audio_tagging_loss=0.01004, over 14683.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09286, pruned_loss=0.01389, audio_tagging_loss=0.008925, over 3050765.52 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:40:17,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354800
2023-11-23 11:40:42,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2365400.0, ans=0.125
2023-11-23 11:40:43,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2365400.0, ans=0.1
2023-11-23 11:41:02,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.175e+01 8.198e+01 8.863e+01 9.563e+01 1.175e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 11:41:07,181 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.42 vs. limit=15.0
2023-11-23 11:41:11,363 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6150, loss[loss=0.06344, simple_loss=0.0807, pruned_loss=0.01281, audio_tagging_loss=0.01027, over 15389.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09266, pruned_loss=0.01394, audio_tagging_loss=0.008995, over 3051296.83 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:41:21,736 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354850
2023-11-23 11:41:27,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-23 11:42:16,175 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6200, loss[loss=0.05762, simple_loss=0.07521, pruned_loss=0.009044, audio_tagging_loss=0.01097, over 15753.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09326, pruned_loss=0.01417, audio_tagging_loss=0.009052, over 3051821.17 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:42:18,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2365933.3333333335, ans=0.0
2023-11-23 11:42:19,224 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=15.0
2023-11-23 11:42:21,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2365933.3333333335, ans=0.2
2023-11-23 11:42:27,280 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354900
2023-11-23 11:42:27,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2365933.3333333335, ans=0.0
2023-11-23 11:42:39,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2366000.0, ans=0.1
2023-11-23 11:43:12,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2366200.0, ans=0.125
2023-11-23 11:43:13,566 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.682e+01 8.235e+01 8.845e+01 9.606e+01 1.266e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 11:43:21,052 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6250, loss[loss=0.07709, simple_loss=0.09697, pruned_loss=0.01719, audio_tagging_loss=0.01141, over 14925.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09253, pruned_loss=0.01411, audio_tagging_loss=0.009244, over 3054737.26 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:43:25,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2366266.6666666665, ans=0.125
2023-11-23 11:43:31,014 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 354950
2023-11-23 11:43:33,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2366333.3333333335, ans=0.2
2023-11-23 11:43:35,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.55 vs. limit=22.5
2023-11-23 11:43:44,693 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.11 vs. limit=15.0
2023-11-23 11:43:57,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2366400.0, ans=0.125
2023-11-23 11:43:59,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.28 vs. limit=15.0
2023-11-23 11:44:12,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2366533.3333333335, ans=0.09899494936611666
2023-11-23 11:44:21,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=2366533.3333333335, ans=15.0
2023-11-23 11:44:24,652 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6300, loss[loss=0.05851, simple_loss=0.06825, pruned_loss=0.0141, audio_tagging_loss=0.01029, over 15148.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09191, pruned_loss=0.01407, audio_tagging_loss=0.009345, over 3050233.94 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:44:34,921 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355000
2023-11-23 11:44:43,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=15.0
2023-11-23 11:45:20,956 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.278e+01 8.805e+01 9.594e+01 1.253e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 11:45:25,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2366866.6666666665, ans=0.125
2023-11-23 11:45:25,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2366866.6666666665, ans=0.125
2023-11-23 11:45:28,987 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6350, loss[loss=0.07734, simple_loss=0.09841, pruned_loss=0.01828, audio_tagging_loss=0.009856, over 15220.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09121, pruned_loss=0.01368, audio_tagging_loss=0.009437, over 3041211.15 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:45:38,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2366933.3333333335, ans=0.5
2023-11-23 11:45:39,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355050
2023-11-23 11:45:54,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2367066.6666666665, ans=0.0
2023-11-23 11:46:34,469 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6400, loss[loss=0.0755, simple_loss=0.1019, pruned_loss=0.01628, audio_tagging_loss=0.008242, over 15499.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09122, pruned_loss=0.01363, audio_tagging_loss=0.009457, over 3041818.05 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:46:44,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355100
2023-11-23 11:47:00,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2367400.0, ans=0.07
2023-11-23 11:47:30,741 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.031e+01 8.724e+01 9.486e+01 1.143e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 11:47:32,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2367533.3333333335, ans=0.0
2023-11-23 11:47:38,294 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6450, loss[loss=0.08905, simple_loss=0.118, pruned_loss=0.02303, audio_tagging_loss=0.007041, over 15462.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09191, pruned_loss=0.01372, audio_tagging_loss=0.00944, over 3041344.52 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:47:38,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2367600.0, ans=0.1
2023-11-23 11:47:48,384 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355150
2023-11-23 11:48:11,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2367733.3333333335, ans=0.125
2023-11-23 11:48:33,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2367866.6666666665, ans=0.1
2023-11-23 11:48:43,079 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6500, loss[loss=0.06887, simple_loss=0.09406, pruned_loss=0.014, audio_tagging_loss=0.007844, over 15447.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09253, pruned_loss=0.01377, audio_tagging_loss=0.00938, over 3044091.46 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:48:45,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2367933.3333333335, ans=0.0
2023-11-23 11:48:53,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355200
2023-11-23 11:49:06,126 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.63 vs. limit=15.0
2023-11-23 11:49:39,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2368200.0, ans=0.1
2023-11-23 11:49:40,449 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.456e+01 9.159e+01 9.950e+01 1.283e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-23 11:49:48,680 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6550, loss[loss=0.08235, simple_loss=0.1178, pruned_loss=0.0151, audio_tagging_loss=0.008366, over 16044.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09265, pruned_loss=0.01384, audio_tagging_loss=0.00925, over 3055907.16 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:49:52,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2368266.6666666665, ans=0.0
2023-11-23 11:49:59,202 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355250
2023-11-23 11:50:15,862 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.04 vs. limit=22.5
2023-11-23 11:50:53,261 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6600, loss[loss=0.08191, simple_loss=0.1053, pruned_loss=0.02021, audio_tagging_loss=0.009064, over 15592.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09198, pruned_loss=0.0139, audio_tagging_loss=0.009305, over 3046924.20 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:51:03,306 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355300
2023-11-23 11:51:03,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2368600.0, ans=0.125
2023-11-23 11:51:03,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.34 vs. limit=15.0
2023-11-23 11:51:21,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2368733.3333333335, ans=0.0
2023-11-23 11:51:31,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2368800.0, ans=0.125
2023-11-23 11:51:40,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2368800.0, ans=0.0
2023-11-23 11:51:43,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.80 vs. limit=15.0
2023-11-23 11:51:49,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2368866.6666666665, ans=0.125
2023-11-23 11:51:50,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0
2023-11-23 11:51:50,570 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.216e+01 8.892e+01 9.603e+01 1.376e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 11:51:55,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2368866.6666666665, ans=0.0
2023-11-23 11:51:58,400 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6650, loss[loss=0.07847, simple_loss=0.109, pruned_loss=0.0136, audio_tagging_loss=0.01036, over 16456.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09336, pruned_loss=0.0141, audio_tagging_loss=0.009188, over 3041997.77 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:52:08,802 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355350
2023-11-23 11:52:18,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2369000.0, ans=0.015
2023-11-23 11:52:18,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.73 vs. limit=15.0
2023-11-23 11:52:21,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.73 vs. limit=22.5
2023-11-23 11:52:23,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2369066.6666666665, ans=0.0
2023-11-23 11:52:33,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2369066.6666666665, ans=0.0
2023-11-23 11:52:52,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2369200.0, ans=0.125
2023-11-23 11:53:03,807 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6700, loss[loss=0.06561, simple_loss=0.08155, pruned_loss=0.01449, audio_tagging_loss=0.01034, over 15246.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09355, pruned_loss=0.01425, audio_tagging_loss=0.009045, over 3041738.14 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:53:09,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2369266.6666666665, ans=0.0
2023-11-23 11:53:10,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2369266.6666666665, ans=0.0
2023-11-23 11:53:13,942 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355400
2023-11-23 11:53:29,344 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-23 11:53:53,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2369466.6666666665, ans=0.0
2023-11-23 11:53:55,805 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.38 vs. limit=22.5
2023-11-23 11:54:01,061 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.203e+01 8.988e+01 9.539e+01 1.363e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 11:54:08,644 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6750, loss[loss=0.0714, simple_loss=0.09794, pruned_loss=0.01487, audio_tagging_loss=0.007561, over 15971.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09222, pruned_loss=0.01406, audio_tagging_loss=0.009033, over 3029145.39 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:54:19,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355450
2023-11-23 11:54:21,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2369666.6666666665, ans=0.125
2023-11-23 11:54:26,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.53 vs. limit=15.0
2023-11-23 11:54:32,920 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.23 vs. limit=15.0
2023-11-23 11:54:41,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2369733.3333333335, ans=0.125
2023-11-23 11:55:00,845 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:55:13,494 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6800, loss[loss=0.06206, simple_loss=0.08264, pruned_loss=0.01209, audio_tagging_loss=0.008648, over 15320.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09166, pruned_loss=0.01385, audio_tagging_loss=0.009018, over 3039399.79 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:55:21,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2369933.3333333335, ans=0.1
2023-11-23 11:55:24,548 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355500
2023-11-23 11:55:28,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2370000.0, ans=0.125
2023-11-23 11:55:28,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2370000.0, ans=0.2
2023-11-23 11:55:50,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2370066.6666666665, ans=0.0
2023-11-23 11:55:57,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2370133.3333333335, ans=0.125
2023-11-23 11:55:57,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2370133.3333333335, ans=22.5
2023-11-23 11:56:03,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2370133.3333333335, ans=0.035
2023-11-23 11:56:12,191 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.201e+01 8.354e+01 8.978e+01 9.690e+01 1.235e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 11:56:19,023 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6850, loss[loss=0.07191, simple_loss=0.1006, pruned_loss=0.01503, audio_tagging_loss=0.006596, over 14186.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09234, pruned_loss=0.01394, audio_tagging_loss=0.00888, over 3040761.88 frames. ], batch size: 53, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:56:20,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2370266.6666666665, ans=0.1
2023-11-23 11:56:29,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355550
2023-11-23 11:56:29,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2370266.6666666665, ans=0.125
2023-11-23 11:56:34,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2370333.3333333335, ans=0.2
2023-11-23 11:56:44,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2370400.0, ans=0.125
2023-11-23 11:56:45,215 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.89 vs. limit=22.5
2023-11-23 11:56:59,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2370466.6666666665, ans=0.0
2023-11-23 11:57:24,304 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6900, loss[loss=0.06225, simple_loss=0.08195, pruned_loss=0.01159, audio_tagging_loss=0.009684, over 15779.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09174, pruned_loss=0.01385, audio_tagging_loss=0.008966, over 3039964.41 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:57:29,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2370600.0, ans=0.1
2023-11-23 11:57:34,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355600
2023-11-23 11:57:55,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2370733.3333333335, ans=0.125
2023-11-23 11:57:56,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2370733.3333333335, ans=0.05
2023-11-23 11:58:15,728 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:58:18,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2370866.6666666665, ans=0.1
2023-11-23 11:58:23,524 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.385e+01 8.957e+01 9.598e+01 1.316e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 11:58:29,827 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 6950, loss[loss=0.08958, simple_loss=0.1205, pruned_loss=0.02244, audio_tagging_loss=0.006878, over 15017.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09307, pruned_loss=0.01398, audio_tagging_loss=0.008837, over 3042727.34 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:58:33,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2370933.3333333335, ans=0.2
2023-11-23 11:58:40,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2370933.3333333335, ans=0.125
2023-11-23 11:58:41,254 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355650
2023-11-23 11:58:52,286 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.69 vs. limit=15.0
2023-11-23 11:59:01,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2371066.6666666665, ans=0.125
2023-11-23 11:59:28,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2371200.0, ans=0.07
2023-11-23 11:59:36,263 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7000, loss[loss=0.05764, simple_loss=0.07728, pruned_loss=0.01106, audio_tagging_loss=0.00794, over 15603.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09313, pruned_loss=0.01406, audio_tagging_loss=0.008929, over 3040923.13 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:59:39,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2371266.6666666665, ans=0.125
2023-11-23 11:59:46,810 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355700
2023-11-23 11:59:57,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2371333.3333333335, ans=0.0
2023-11-23 12:00:14,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2371466.6666666665, ans=0.125
2023-11-23 12:00:25,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2371533.3333333335, ans=0.125
2023-11-23 12:00:28,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.67 vs. limit=10.0
2023-11-23 12:00:34,917 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.338e+01 8.854e+01 9.577e+01 1.141e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 12:00:39,858 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7050, loss[loss=0.06375, simple_loss=0.08393, pruned_loss=0.01191, audio_tagging_loss=0.00988, over 16074.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09258, pruned_loss=0.01398, audio_tagging_loss=0.009138, over 3043438.49 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:00:42,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2371600.0, ans=0.125
2023-11-23 12:00:46,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.95 vs. limit=15.0
2023-11-23 12:00:49,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355750
2023-11-23 12:01:06,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.44 vs. limit=12.0
2023-11-23 12:01:13,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2371733.3333333335, ans=0.125
2023-11-23 12:01:43,848 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7100, loss[loss=0.06925, simple_loss=0.09939, pruned_loss=0.01128, audio_tagging_loss=0.008275, over 14916.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09225, pruned_loss=0.01386, audio_tagging_loss=0.00921, over 3039625.72 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:01:53,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2371933.3333333335, ans=0.125
2023-11-23 12:01:54,855 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355800
2023-11-23 12:02:02,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2372000.0, ans=0.1
2023-11-23 12:02:25,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2372133.3333333335, ans=0.125
2023-11-23 12:02:27,603 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.12 vs. limit=15.0
2023-11-23 12:02:35,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2372200.0, ans=0.0
2023-11-23 12:02:43,344 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.520e+01 9.184e+01 9.861e+01 1.233e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 12:02:48,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2372266.6666666665, ans=0.125
2023-11-23 12:02:49,719 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7150, loss[loss=0.06836, simple_loss=0.09931, pruned_loss=0.01049, audio_tagging_loss=0.008216, over 15431.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09266, pruned_loss=0.01408, audio_tagging_loss=0.009354, over 3035596.11 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:02:57,628 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.25 vs. limit=22.5
2023-11-23 12:03:00,187 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355850
2023-11-23 12:03:00,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2372266.6666666665, ans=0.0
2023-11-23 12:03:27,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2372466.6666666665, ans=0.1
2023-11-23 12:03:31,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2372466.6666666665, ans=0.125
2023-11-23 12:03:32,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2372466.6666666665, ans=0.125
2023-11-23 12:03:33,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2372466.6666666665, ans=0.0
2023-11-23 12:03:49,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2372533.3333333335, ans=0.1
2023-11-23 12:03:53,737 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7200, loss[loss=0.08487, simple_loss=0.1143, pruned_loss=0.01928, audio_tagging_loss=0.008455, over 15868.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09288, pruned_loss=0.01421, audio_tagging_loss=0.009413, over 3048306.79 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:04:00,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2372600.0, ans=0.125
2023-11-23 12:04:01,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2372600.0, ans=0.1
2023-11-23 12:04:03,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355900
2023-11-23 12:04:34,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2372800.0, ans=0.125
2023-11-23 12:04:34,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2372800.0, ans=0.125
2023-11-23 12:04:35,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2372800.0, ans=0.1
2023-11-23 12:04:36,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2372800.0, ans=0.0
2023-11-23 12:04:36,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2372800.0, ans=0.1
2023-11-23 12:04:37,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2372800.0, ans=0.125
2023-11-23 12:04:52,179 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.220e+01 8.499e+01 9.077e+01 9.977e+01 1.549e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 12:04:57,115 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7250, loss[loss=0.0578, simple_loss=0.0744, pruned_loss=0.01157, audio_tagging_loss=0.009033, over 15043.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09271, pruned_loss=0.0141, audio_tagging_loss=0.009467, over 3050660.73 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:05:07,505 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 355950
2023-11-23 12:05:10,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2373000.0, ans=0.125
2023-11-23 12:05:39,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2373133.3333333335, ans=0.125
2023-11-23 12:05:45,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2373133.3333333335, ans=6.0
2023-11-23 12:05:56,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2373200.0, ans=0.125
2023-11-23 12:05:56,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2373200.0, ans=0.1
2023-11-23 12:05:59,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2373200.0, ans=0.125
2023-11-23 12:06:01,470 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7300, loss[loss=0.07404, simple_loss=0.1038, pruned_loss=0.01629, audio_tagging_loss=0.005836, over 16082.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09332, pruned_loss=0.01413, audio_tagging_loss=0.009309, over 3046804.04 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:06:12,732 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356000
2023-11-23 12:06:31,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.35 vs. limit=12.0
2023-11-23 12:06:51,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.44 vs. limit=15.0
2023-11-23 12:07:05,009 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.072e+01 8.323e+01 8.907e+01 9.427e+01 1.100e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 12:07:10,570 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7350, loss[loss=0.09525, simple_loss=0.1274, pruned_loss=0.02344, audio_tagging_loss=0.008107, over 14142.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09295, pruned_loss=0.01408, audio_tagging_loss=0.009157, over 3045597.56 frames. ], batch size: 52, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:07:10,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2373600.0, ans=0.125
2023-11-23 12:07:13,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2373600.0, ans=0.2
2023-11-23 12:07:14,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2373600.0, ans=0.125
2023-11-23 12:07:20,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356050
2023-11-23 12:07:24,823 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.33 vs. limit=10.0
2023-11-23 12:07:26,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2373666.6666666665, ans=0.0
2023-11-23 12:07:38,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2373733.3333333335, ans=0.2
2023-11-23 12:07:41,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2373733.3333333335, ans=0.09899494936611666
2023-11-23 12:07:53,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2373800.0, ans=0.125
2023-11-23 12:07:54,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2373800.0, ans=0.1
2023-11-23 12:07:57,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2373800.0, ans=0.1
2023-11-23 12:08:03,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2373866.6666666665, ans=0.0
2023-11-23 12:08:14,466 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7400, loss[loss=0.06651, simple_loss=0.08689, pruned_loss=0.01433, audio_tagging_loss=0.008735, over 14528.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09226, pruned_loss=0.01392, audio_tagging_loss=0.009087, over 3045628.57 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:08:24,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356100
2023-11-23 12:09:12,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2374200.0, ans=0.0
2023-11-23 12:09:13,007 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.433e+01 8.940e+01 9.689e+01 1.513e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 12:09:16,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2374200.0, ans=0.125
2023-11-23 12:09:18,588 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7450, loss[loss=0.07704, simple_loss=0.1085, pruned_loss=0.01328, audio_tagging_loss=0.009526, over 15653.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09258, pruned_loss=0.01381, audio_tagging_loss=0.008949, over 3056670.07 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:09:21,457 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:09:24,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2374266.6666666665, ans=0.1
2023-11-23 12:09:29,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356150
2023-11-23 12:09:54,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0
2023-11-23 12:09:57,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2374466.6666666665, ans=0.1
2023-11-23 12:10:12,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2374533.3333333335, ans=0.0
2023-11-23 12:10:19,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2374533.3333333335, ans=0.1
2023-11-23 12:10:21,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2374533.3333333335, ans=0.07
2023-11-23 12:10:21,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.04 vs. limit=15.0
2023-11-23 12:10:24,414 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7500, loss[loss=0.06723, simple_loss=0.08318, pruned_loss=0.01383, audio_tagging_loss=0.01181, over 15209.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09175, pruned_loss=0.01373, audio_tagging_loss=0.008991, over 3053084.52 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:10:34,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356200
2023-11-23 12:10:34,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2374600.0, ans=0.125
2023-11-23 12:10:38,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2374666.6666666665, ans=0.125
2023-11-23 12:10:41,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2374666.6666666665, ans=0.0
2023-11-23 12:10:43,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.65 vs. limit=15.0
2023-11-23 12:10:48,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2374733.3333333335, ans=0.0
2023-11-23 12:11:09,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2374800.0, ans=0.125
2023-11-23 12:11:18,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2374866.6666666665, ans=0.125
2023-11-23 12:11:18,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2374866.6666666665, ans=0.2
2023-11-23 12:11:23,846 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.224e+01 8.821e+01 9.416e+01 1.230e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 12:11:28,802 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7550, loss[loss=0.0624, simple_loss=0.07734, pruned_loss=0.015, audio_tagging_loss=0.008732, over 15171.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09086, pruned_loss=0.01371, audio_tagging_loss=0.008943, over 3054710.82 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:11:36,838 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.72 vs. limit=15.0
2023-11-23 12:11:38,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356250
2023-11-23 12:11:40,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2375000.0, ans=0.0
2023-11-23 12:11:48,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2375000.0, ans=0.0
2023-11-23 12:11:57,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2375066.6666666665, ans=0.125
2023-11-23 12:12:19,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.12 vs. limit=15.0
2023-11-23 12:12:34,256 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7600, loss[loss=0.04249, simple_loss=0.05365, pruned_loss=0.008266, audio_tagging_loss=0.007405, over 16685.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09062, pruned_loss=0.01369, audio_tagging_loss=0.008899, over 3061873.57 frames. ], batch size: 68, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:12:37,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2375266.6666666665, ans=0.2
2023-11-23 12:12:44,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356300
2023-11-23 12:12:59,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2375400.0, ans=0.1
2023-11-23 12:13:10,265 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.12 vs. limit=15.0
2023-11-23 12:13:12,678 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.69 vs. limit=15.0
2023-11-23 12:13:34,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2375533.3333333335, ans=0.1
2023-11-23 12:13:35,216 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.003e+01 8.205e+01 8.929e+01 9.831e+01 1.216e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 12:13:39,064 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7650, loss[loss=0.08381, simple_loss=0.1139, pruned_loss=0.01923, audio_tagging_loss=0.007651, over 15563.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09087, pruned_loss=0.01371, audio_tagging_loss=0.008901, over 3059189.55 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:13:49,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2375600.0, ans=0.0
2023-11-23 12:13:50,286 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356350
2023-11-23 12:13:52,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2375666.6666666665, ans=0.07
2023-11-23 12:14:02,979 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0
2023-11-23 12:14:21,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2375800.0, ans=0.1
2023-11-23 12:14:21,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2375800.0, ans=0.1
2023-11-23 12:14:32,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2375866.6666666665, ans=0.125
2023-11-23 12:14:39,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2375866.6666666665, ans=0.2
2023-11-23 12:14:41,895 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.97 vs. limit=15.0
2023-11-23 12:14:44,727 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7700, loss[loss=0.0647, simple_loss=0.07412, pruned_loss=0.01651, audio_tagging_loss=0.01112, over 15284.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09106, pruned_loss=0.01372, audio_tagging_loss=0.008993, over 3060015.92 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:14:45,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2375933.3333333335, ans=0.2
2023-11-23 12:14:54,584 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356400
2023-11-23 12:15:00,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2376000.0, ans=0.0
2023-11-23 12:15:05,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2376000.0, ans=0.025
2023-11-23 12:15:06,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2376000.0, ans=0.0
2023-11-23 12:15:13,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2376066.6666666665, ans=0.125
2023-11-23 12:15:17,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2376066.6666666665, ans=0.1
2023-11-23 12:15:17,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2376066.6666666665, ans=0.125
2023-11-23 12:15:45,412 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.176e+01 8.917e+01 9.503e+01 1.166e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 12:15:49,145 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7750, loss[loss=0.07748, simple_loss=0.1139, pruned_loss=0.01418, audio_tagging_loss=0.006333, over 15649.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.0917, pruned_loss=0.01388, audio_tagging_loss=0.009, over 3052355.81 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:16:00,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356450
2023-11-23 12:16:25,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2376400.0, ans=0.0
2023-11-23 12:16:32,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2376466.6666666665, ans=0.5
2023-11-23 12:16:54,247 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7800, loss[loss=0.07897, simple_loss=0.1137, pruned_loss=0.01627, audio_tagging_loss=0.005861, over 16055.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09127, pruned_loss=0.01382, audio_tagging_loss=0.009039, over 3050063.48 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:16:58,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2376600.0, ans=0.0
2023-11-23 12:17:04,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356500
2023-11-23 12:17:28,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2376733.3333333335, ans=0.0
2023-11-23 12:17:54,729 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.181e+01 8.920e+01 9.526e+01 1.528e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 12:17:56,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2376866.6666666665, ans=0.2
2023-11-23 12:17:57,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2376933.3333333335, ans=0.125
2023-11-23 12:17:58,453 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7850, loss[loss=0.05706, simple_loss=0.07004, pruned_loss=0.01041, audio_tagging_loss=0.01164, over 16016.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09125, pruned_loss=0.01386, audio_tagging_loss=0.00916, over 3053045.03 frames. ], batch size: 62, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:18:04,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2376933.3333333335, ans=0.1
2023-11-23 12:18:06,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2376933.3333333335, ans=0.0
2023-11-23 12:18:09,040 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356550
2023-11-23 12:18:53,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2377200.0, ans=0.95
2023-11-23 12:19:02,626 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7900, loss[loss=0.06531, simple_loss=0.08369, pruned_loss=0.01437, audio_tagging_loss=0.009093, over 14613.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.0922, pruned_loss=0.01402, audio_tagging_loss=0.009236, over 3047856.64 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:19:13,258 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356600
2023-11-23 12:19:16,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2377333.3333333335, ans=0.125
2023-11-23 12:19:18,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2377333.3333333335, ans=0.125
2023-11-23 12:19:19,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=13.90 vs. limit=15.0
2023-11-23 12:19:22,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2377333.3333333335, ans=0.125
2023-11-23 12:19:25,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2377333.3333333335, ans=0.5
2023-11-23 12:20:04,574 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.810e+01 8.633e+01 9.246e+01 1.018e+02 1.529e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-23 12:20:08,294 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 7950, loss[loss=0.0604, simple_loss=0.08039, pruned_loss=0.01101, audio_tagging_loss=0.00919, over 14424.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09249, pruned_loss=0.01413, audio_tagging_loss=0.009292, over 3043111.77 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:20:13,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2377600.0, ans=0.2
2023-11-23 12:20:18,745 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356650
2023-11-23 12:20:26,010 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:20:30,274 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.00 vs. limit=15.0
2023-11-23 12:20:45,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.14 vs. limit=15.0
2023-11-23 12:20:53,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2377800.0, ans=0.0
2023-11-23 12:21:12,859 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8000, loss[loss=0.09321, simple_loss=0.117, pruned_loss=0.02547, audio_tagging_loss=0.009241, over 15053.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09244, pruned_loss=0.01423, audio_tagging_loss=0.009341, over 3043631.91 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:21:23,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356700
2023-11-23 12:21:40,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.80 vs. limit=22.5
2023-11-23 12:21:55,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2378133.3333333335, ans=0.1
2023-11-23 12:21:59,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2378133.3333333335, ans=0.0
2023-11-23 12:21:59,933 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.48 vs. limit=15.0
2023-11-23 12:22:15,955 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.250e+01 8.793e+01 9.379e+01 1.192e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 12:22:18,452 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8050, loss[loss=0.07056, simple_loss=0.1017, pruned_loss=0.01319, audio_tagging_loss=0.006522, over 15375.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09208, pruned_loss=0.01406, audio_tagging_loss=0.009438, over 3046711.24 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:22:28,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356750
2023-11-23 12:22:57,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2378466.6666666665, ans=0.0
2023-11-23 12:22:57,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2378466.6666666665, ans=0.125
2023-11-23 12:23:07,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2378466.6666666665, ans=0.125
2023-11-23 12:23:14,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2378533.3333333335, ans=0.125
2023-11-23 12:23:24,485 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8100, loss[loss=0.04099, simple_loss=0.04611, pruned_loss=0.004793, audio_tagging_loss=0.01314, over 14995.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09105, pruned_loss=0.01385, audio_tagging_loss=0.009435, over 3045310.85 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:23:34,723 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356800
2023-11-23 12:24:12,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2378800.0, ans=0.1
2023-11-23 12:24:13,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2378800.0, ans=0.2
2023-11-23 12:24:15,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2378800.0, ans=0.2
2023-11-23 12:24:27,904 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.660e+01 9.350e+01 9.941e+01 1.281e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-23 12:24:30,479 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8150, loss[loss=0.06824, simple_loss=0.09272, pruned_loss=0.0144, audio_tagging_loss=0.007473, over 17108.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09195, pruned_loss=0.01394, audio_tagging_loss=0.009243, over 3046239.31 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:24:39,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.16 vs. limit=22.5
2023-11-23 12:24:40,547 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356850
2023-11-23 12:24:58,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2379066.6666666665, ans=0.1
2023-11-23 12:25:08,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2379133.3333333335, ans=0.0
2023-11-23 12:25:34,962 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8200, loss[loss=0.08415, simple_loss=0.1243, pruned_loss=0.01683, audio_tagging_loss=0.005159, over 16576.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09167, pruned_loss=0.0139, audio_tagging_loss=0.009094, over 3044353.77 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:25:37,462 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:25:46,482 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356900
2023-11-23 12:25:47,293 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.10 vs. limit=12.0
2023-11-23 12:26:13,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2379466.6666666665, ans=0.07
2023-11-23 12:26:17,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2379466.6666666665, ans=0.0
2023-11-23 12:26:29,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2379533.3333333335, ans=0.1
2023-11-23 12:26:38,254 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 8.430e+01 9.137e+01 9.990e+01 1.706e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 12:26:40,852 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8250, loss[loss=0.07641, simple_loss=0.1047, pruned_loss=0.01707, audio_tagging_loss=0.007004, over 15300.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09138, pruned_loss=0.01394, audio_tagging_loss=0.009163, over 3043629.59 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:26:51,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 356950
2023-11-23 12:26:59,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2379666.6666666665, ans=0.1
2023-11-23 12:27:14,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2379733.3333333335, ans=0.125
2023-11-23 12:27:45,809 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8300, loss[loss=0.06533, simple_loss=0.09176, pruned_loss=0.01191, audio_tagging_loss=0.007533, over 14709.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09118, pruned_loss=0.01385, audio_tagging_loss=0.009205, over 3044417.42 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:27:55,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357000
2023-11-23 12:27:56,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2379933.3333333335, ans=0.2
2023-11-23 12:27:57,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2380000.0, ans=10.0
2023-11-23 12:28:00,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2380000.0, ans=0.1
2023-11-23 12:28:02,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2380000.0, ans=0.125
2023-11-23 12:28:10,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2380066.6666666665, ans=0.025
2023-11-23 12:28:16,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2380066.6666666665, ans=0.2
2023-11-23 12:28:31,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2380133.3333333335, ans=0.125
2023-11-23 12:28:46,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2380200.0, ans=0.125
2023-11-23 12:28:47,623 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.282e+01 8.543e+01 9.003e+01 9.782e+01 1.205e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 12:28:50,041 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8350, loss[loss=0.05994, simple_loss=0.07536, pruned_loss=0.01291, audio_tagging_loss=0.00935, over 15047.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09223, pruned_loss=0.01403, audio_tagging_loss=0.009075, over 3042437.07 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:29:00,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357050
2023-11-23 12:29:05,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.64 vs. limit=15.0
2023-11-23 12:29:36,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2380466.6666666665, ans=0.1
2023-11-23 12:29:39,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2380466.6666666665, ans=0.95
2023-11-23 12:29:54,900 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8400, loss[loss=0.06063, simple_loss=0.07941, pruned_loss=0.01122, audio_tagging_loss=0.009711, over 14950.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.0916, pruned_loss=0.01405, audio_tagging_loss=0.009056, over 3041105.82 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:29:59,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.19 vs. limit=22.5
2023-11-23 12:30:05,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357100
2023-11-23 12:30:15,077 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.51 vs. limit=22.5
2023-11-23 12:30:24,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0
2023-11-23 12:30:29,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2380733.3333333335, ans=0.1
2023-11-23 12:30:45,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2380866.6666666665, ans=0.0
2023-11-23 12:30:57,936 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.179e+01 8.935e+01 9.716e+01 1.287e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 12:30:59,208 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8450, loss[loss=0.05141, simple_loss=0.05983, pruned_loss=0.01177, audio_tagging_loss=0.009731, over 14263.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09173, pruned_loss=0.01391, audio_tagging_loss=0.009053, over 3046882.66 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:30:59,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2380933.3333333335, ans=0.025
2023-11-23 12:31:06,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2380933.3333333335, ans=0.05
2023-11-23 12:31:09,725 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357150
2023-11-23 12:31:20,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2381000.0, ans=0.07
2023-11-23 12:31:24,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2381066.6666666665, ans=0.5
2023-11-23 12:31:39,863 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:31:39,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2381133.3333333335, ans=0.0
2023-11-23 12:31:45,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2381133.3333333335, ans=0.1
2023-11-23 12:31:50,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=15.0
2023-11-23 12:32:03,530 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8500, loss[loss=0.06021, simple_loss=0.08372, pruned_loss=0.009637, audio_tagging_loss=0.008707, over 15984.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09171, pruned_loss=0.01389, audio_tagging_loss=0.009089, over 3044884.67 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:32:06,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2381266.6666666665, ans=0.2
2023-11-23 12:32:10,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2381266.6666666665, ans=0.125
2023-11-23 12:32:13,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357200
2023-11-23 12:32:15,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.82 vs. limit=15.0
2023-11-23 12:32:21,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2381333.3333333335, ans=0.2
2023-11-23 12:32:28,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2381400.0, ans=0.1
2023-11-23 12:32:34,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2381400.0, ans=0.015
2023-11-23 12:32:47,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2381466.6666666665, ans=0.2
2023-11-23 12:32:54,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2381533.3333333335, ans=0.125
2023-11-23 12:33:06,894 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.293e+01 8.960e+01 9.755e+01 1.208e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 12:33:08,140 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8550, loss[loss=0.06503, simple_loss=0.08331, pruned_loss=0.0126, audio_tagging_loss=0.01077, over 14409.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09231, pruned_loss=0.01393, audio_tagging_loss=0.009067, over 3042261.14 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:33:13,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2381600.0, ans=0.125
2023-11-23 12:33:13,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2381600.0, ans=0.0
2023-11-23 12:33:19,357 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357250
2023-11-23 12:33:39,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2381733.3333333335, ans=0.1
2023-11-23 12:33:45,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2381733.3333333335, ans=0.2
2023-11-23 12:33:47,521 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:33:51,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2381800.0, ans=0.07
2023-11-23 12:34:03,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2381866.6666666665, ans=0.04949747468305833
2023-11-23 12:34:07,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2381866.6666666665, ans=0.0
2023-11-23 12:34:13,815 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8600, loss[loss=0.06431, simple_loss=0.08403, pruned_loss=0.01155, audio_tagging_loss=0.01074, over 15581.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09197, pruned_loss=0.01388, audio_tagging_loss=0.009137, over 3045648.88 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:34:24,338 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357300
2023-11-23 12:34:31,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2382000.0, ans=0.0
2023-11-23 12:34:35,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.74 vs. limit=15.0
2023-11-23 12:34:39,668 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=9.67 vs. limit=15.0
2023-11-23 12:34:41,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2382066.6666666665, ans=0.125
2023-11-23 12:34:42,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2382066.6666666665, ans=0.1
2023-11-23 12:34:47,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2382066.6666666665, ans=0.125
2023-11-23 12:35:17,573 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.105e+01 8.339e+01 9.203e+01 9.998e+01 1.297e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-23 12:35:18,847 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8650, loss[loss=0.08464, simple_loss=0.1137, pruned_loss=0.01956, audio_tagging_loss=0.008226, over 14792.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09225, pruned_loss=0.01394, audio_tagging_loss=0.00911, over 3038778.09 frames. ], batch size: 53, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:35:21,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2382266.6666666665, ans=0.125
2023-11-23 12:35:28,224 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.49 vs. limit=15.0
2023-11-23 12:35:28,757 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357350
2023-11-23 12:35:31,844 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.73 vs. limit=10.0
2023-11-23 12:35:41,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2382333.3333333335, ans=0.125
2023-11-23 12:35:54,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2382400.0, ans=0.125
2023-11-23 12:36:00,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.33 vs. limit=15.0
2023-11-23 12:36:22,798 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8700, loss[loss=0.06301, simple_loss=0.08652, pruned_loss=0.01115, audio_tagging_loss=0.008597, over 15083.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.0925, pruned_loss=0.01388, audio_tagging_loss=0.009219, over 3040549.14 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:36:23,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2382600.0, ans=0.125
2023-11-23 12:36:26,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2382600.0, ans=0.125
2023-11-23 12:36:32,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2382600.0, ans=0.125
2023-11-23 12:36:32,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2382600.0, ans=0.2
2023-11-23 12:36:33,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357400
2023-11-23 12:36:34,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2382600.0, ans=0.1
2023-11-23 12:36:54,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2382733.3333333335, ans=0.0
2023-11-23 12:37:21,801 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.81 vs. limit=15.0
2023-11-23 12:37:27,555 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.508e+01 9.323e+01 9.917e+01 1.440e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-23 12:37:28,907 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8750, loss[loss=0.09511, simple_loss=0.1382, pruned_loss=0.02042, audio_tagging_loss=0.005608, over 16013.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09272, pruned_loss=0.01385, audio_tagging_loss=0.009293, over 3039456.00 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:37:29,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2382933.3333333335, ans=0.125
2023-11-23 12:37:40,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357450
2023-11-23 12:38:11,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.73 vs. limit=15.0
2023-11-23 12:38:24,053 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.99 vs. limit=15.0
2023-11-23 12:38:26,920 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.99 vs. limit=15.0
2023-11-23 12:38:29,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2383200.0, ans=0.1
2023-11-23 12:38:29,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2383200.0, ans=0.125
2023-11-23 12:38:32,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2383200.0, ans=0.125
2023-11-23 12:38:35,182 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8800, loss[loss=0.066, simple_loss=0.0866, pruned_loss=0.01298, audio_tagging_loss=0.009713, over 15771.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09415, pruned_loss=0.01424, audio_tagging_loss=0.009317, over 3042128.16 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:38:44,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357500
2023-11-23 12:38:48,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2383333.3333333335, ans=0.125
2023-11-23 12:38:51,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2383333.3333333335, ans=0.125
2023-11-23 12:39:12,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2383466.6666666665, ans=0.125
2023-11-23 12:39:21,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2383466.6666666665, ans=0.2
2023-11-23 12:39:30,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2383533.3333333335, ans=0.125
2023-11-23 12:39:39,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2383600.0, ans=22.5
2023-11-23 12:39:39,747 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.582e+01 9.222e+01 9.925e+01 1.869e+02, threshold=1.844e+02, percent-clipped=1.0
2023-11-23 12:39:39,795 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8850, loss[loss=0.08404, simple_loss=0.1117, pruned_loss=0.02158, audio_tagging_loss=0.006643, over 14673.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09414, pruned_loss=0.01427, audio_tagging_loss=0.009312, over 3044012.68 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:39:42,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2383600.0, ans=0.0
2023-11-23 12:39:50,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357550
2023-11-23 12:39:54,660 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:39:58,934 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.09 vs. limit=22.5
2023-11-23 12:40:16,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2383733.3333333335, ans=0.125
2023-11-23 12:40:32,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2383866.6666666665, ans=0.0
2023-11-23 12:40:45,543 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8900, loss[loss=0.0774, simple_loss=0.1088, pruned_loss=0.01532, audio_tagging_loss=0.007689, over 15025.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09398, pruned_loss=0.01413, audio_tagging_loss=0.00917, over 3049937.06 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:40:55,939 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357600
2023-11-23 12:41:08,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2384000.0, ans=0.125
2023-11-23 12:41:09,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2384000.0, ans=0.1
2023-11-23 12:41:11,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-23 12:41:51,024 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.078e+01 8.629e+01 9.554e+01 1.140e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-23 12:41:51,066 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 8950, loss[loss=0.06445, simple_loss=0.08423, pruned_loss=0.01458, audio_tagging_loss=0.007755, over 15166.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09317, pruned_loss=0.01404, audio_tagging_loss=0.009043, over 3042151.17 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:41:58,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2384266.6666666665, ans=0.0
2023-11-23 12:42:00,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357650
2023-11-23 12:42:19,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2384400.0, ans=0.07
2023-11-23 12:42:40,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2384466.6666666665, ans=0.125
2023-11-23 12:42:48,991 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.00 vs. limit=15.0
2023-11-23 12:42:54,539 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9000, loss[loss=0.06913, simple_loss=0.09024, pruned_loss=0.01422, audio_tagging_loss=0.009794, over 13871.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09384, pruned_loss=0.01414, audio_tagging_loss=0.008952, over 3049854.45 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:42:54,540 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 12:43:26,025 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9692, 3.7659, 4.9149, 4.3576], device='cuda:2')
2023-11-23 12:43:36,351 INFO [train_asr.py:1253] (2/4) Epoch 30, validation: loss=0.05877, simple_loss=0.051, pruned_loss=0.005026, audio_tagging_loss=0.02824, over 4681554.00 frames. 
2023-11-23 12:43:36,351 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 12:43:44,569 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:43:46,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357700
2023-11-23 12:43:46,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2384600.0, ans=0.125
2023-11-23 12:43:53,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2384666.6666666665, ans=0.125
2023-11-23 12:44:03,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2384733.3333333335, ans=0.1
2023-11-23 12:44:07,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2384733.3333333335, ans=0.125
2023-11-23 12:44:19,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2384800.0, ans=0.5
2023-11-23 12:44:20,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2384800.0, ans=0.0
2023-11-23 12:44:30,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2384866.6666666665, ans=0.015
2023-11-23 12:44:41,113 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.551e+01 8.530e+01 8.986e+01 9.928e+01 1.282e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 12:44:41,163 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9050, loss[loss=0.07132, simple_loss=0.1042, pruned_loss=0.01232, audio_tagging_loss=0.006891, over 16146.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09477, pruned_loss=0.01441, audio_tagging_loss=0.008921, over 3051598.01 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:44:41,809 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.72 vs. limit=15.0
2023-11-23 12:44:51,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357750
2023-11-23 12:45:35,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2385200.0, ans=0.05
2023-11-23 12:45:36,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2385200.0, ans=0.1
2023-11-23 12:45:44,794 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9100, loss[loss=0.05525, simple_loss=0.07068, pruned_loss=0.01319, audio_tagging_loss=0.006725, over 14772.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09427, pruned_loss=0.01423, audio_tagging_loss=0.008844, over 3052604.52 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:45:55,227 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357800
2023-11-23 12:46:02,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2385333.3333333335, ans=0.125
2023-11-23 12:46:11,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2385400.0, ans=0.125
2023-11-23 12:46:41,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2385533.3333333335, ans=0.0
2023-11-23 12:46:49,805 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.972e+01 8.272e+01 8.936e+01 9.614e+01 1.250e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 12:46:49,851 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9150, loss[loss=0.06103, simple_loss=0.08506, pruned_loss=0.01173, audio_tagging_loss=0.006764, over 13998.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09329, pruned_loss=0.01405, audio_tagging_loss=0.008835, over 3050181.18 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:46:59,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2385600.0, ans=0.125
2023-11-23 12:47:00,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357850
2023-11-23 12:47:08,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2385666.6666666665, ans=0.0
2023-11-23 12:47:21,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2385733.3333333335, ans=0.125
2023-11-23 12:47:23,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2385733.3333333335, ans=0.0
2023-11-23 12:47:44,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2385866.6666666665, ans=0.1
2023-11-23 12:47:53,030 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9200, loss[loss=0.07705, simple_loss=0.09733, pruned_loss=0.01806, audio_tagging_loss=0.01033, over 13900.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09304, pruned_loss=0.01396, audio_tagging_loss=0.008864, over 3047384.16 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:47:53,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2385933.3333333335, ans=0.125
2023-11-23 12:47:53,757 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.82 vs. limit=15.0
2023-11-23 12:48:03,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357900
2023-11-23 12:48:06,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2386000.0, ans=0.125
2023-11-23 12:48:12,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.32 vs. limit=15.0
2023-11-23 12:48:39,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2386133.3333333335, ans=0.125
2023-11-23 12:48:55,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2386266.6666666665, ans=0.125
2023-11-23 12:48:56,514 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.180e+01 8.978e+01 9.494e+01 1.363e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 12:48:56,559 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9250, loss[loss=0.06988, simple_loss=0.1075, pruned_loss=0.01084, audio_tagging_loss=0.005269, over 15144.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09313, pruned_loss=0.01391, audio_tagging_loss=0.008869, over 3048960.54 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:49:06,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 357950
2023-11-23 12:49:18,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2386333.3333333335, ans=0.0
2023-11-23 12:49:31,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2386400.0, ans=0.1
2023-11-23 12:49:36,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.16 vs. limit=15.0
2023-11-23 12:49:41,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2386466.6666666665, ans=0.125
2023-11-23 12:50:00,437 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9300, loss[loss=0.08066, simple_loss=0.1096, pruned_loss=0.01593, audio_tagging_loss=0.009942, over 14753.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09309, pruned_loss=0.01391, audio_tagging_loss=0.008864, over 3055508.26 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:50:02,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.83 vs. limit=15.0
2023-11-23 12:50:10,751 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358000
2023-11-23 12:50:19,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2386666.6666666665, ans=0.125
2023-11-23 12:50:28,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.18 vs. limit=15.0
2023-11-23 12:50:30,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2386733.3333333335, ans=0.0
2023-11-23 12:50:31,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2386733.3333333335, ans=0.07
2023-11-23 12:50:37,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2386800.0, ans=0.04949747468305833
2023-11-23 12:50:38,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2386800.0, ans=0.1
2023-11-23 12:50:45,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2386800.0, ans=0.1
2023-11-23 12:50:50,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2386866.6666666665, ans=0.2
2023-11-23 12:50:54,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2386866.6666666665, ans=0.2
2023-11-23 12:51:04,206 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.283e+01 8.829e+01 9.752e+01 1.235e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 12:51:04,249 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9350, loss[loss=0.06185, simple_loss=0.08168, pruned_loss=0.01149, audio_tagging_loss=0.009517, over 13753.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09363, pruned_loss=0.01404, audio_tagging_loss=0.008816, over 3057213.28 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:51:05,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2386933.3333333335, ans=0.125
2023-11-23 12:51:05,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2386933.3333333335, ans=0.2
2023-11-23 12:51:13,920 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358050
2023-11-23 12:51:38,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.04 vs. limit=15.0
2023-11-23 12:52:00,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2387200.0, ans=0.125
2023-11-23 12:52:02,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2387200.0, ans=0.05
2023-11-23 12:52:07,399 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9400, loss[loss=0.04978, simple_loss=0.07005, pruned_loss=0.006965, audio_tagging_loss=0.007793, over 16080.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09277, pruned_loss=0.014, audio_tagging_loss=0.008973, over 3059171.14 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:52:14,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2387266.6666666665, ans=0.0
2023-11-23 12:52:15,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2387266.6666666665, ans=0.0
2023-11-23 12:52:17,967 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358100
2023-11-23 12:52:33,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2387400.0, ans=0.125
2023-11-23 12:52:56,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2387466.6666666665, ans=0.0
2023-11-23 12:53:04,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2387533.3333333335, ans=0.125
2023-11-23 12:53:10,712 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:53:11,875 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9450, loss[loss=0.0743, simple_loss=0.09992, pruned_loss=0.01456, audio_tagging_loss=0.009781, over 15398.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09305, pruned_loss=0.01411, audio_tagging_loss=0.009, over 3057787.79 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:53:14,839 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.523e+01 9.315e+01 1.049e+02 1.250e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-23 12:53:16,203 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:53:22,114 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358150
2023-11-23 12:53:23,836 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.91 vs. limit=15.0
2023-11-23 12:53:31,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.86 vs. limit=15.0
2023-11-23 12:53:36,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=15.0
2023-11-23 12:53:57,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2387800.0, ans=0.04949747468305833
2023-11-23 12:54:12,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2387866.6666666665, ans=0.125
2023-11-23 12:54:16,497 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9500, loss[loss=0.05602, simple_loss=0.06752, pruned_loss=0.01012, audio_tagging_loss=0.01214, over 14780.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09221, pruned_loss=0.01406, audio_tagging_loss=0.009128, over 3055406.85 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:54:25,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2387933.3333333335, ans=0.1
2023-11-23 12:54:26,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358200
2023-11-23 12:54:29,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2388000.0, ans=0.0
2023-11-23 12:54:35,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2388000.0, ans=0.125
2023-11-23 12:54:38,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2388000.0, ans=0.09899494936611666
2023-11-23 12:54:39,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2388000.0, ans=0.0
2023-11-23 12:54:54,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2388133.3333333335, ans=0.125
2023-11-23 12:55:19,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2388266.6666666665, ans=0.2
2023-11-23 12:55:19,851 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9550, loss[loss=0.07048, simple_loss=0.0941, pruned_loss=0.01426, audio_tagging_loss=0.009163, over 14572.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09248, pruned_loss=0.01408, audio_tagging_loss=0.00923, over 3047954.30 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:55:22,283 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.501e+01 9.046e+01 9.743e+01 1.461e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 12:55:22,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2388266.6666666665, ans=0.0
2023-11-23 12:55:23,774 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:55:29,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358250
2023-11-23 12:55:32,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.40 vs. limit=15.0
2023-11-23 12:55:48,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2388400.0, ans=0.0
2023-11-23 12:55:52,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2388400.0, ans=0.0
2023-11-23 12:56:04,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.21 vs. limit=15.0
2023-11-23 12:56:09,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2388533.3333333335, ans=0.07
2023-11-23 12:56:14,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2388533.3333333335, ans=0.0
2023-11-23 12:56:16,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2388533.3333333335, ans=0.0
2023-11-23 12:56:24,037 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9600, loss[loss=0.09546, simple_loss=0.1319, pruned_loss=0.019, audio_tagging_loss=0.01053, over 16387.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09268, pruned_loss=0.01397, audio_tagging_loss=0.009205, over 3051532.84 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:56:34,212 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358300
2023-11-23 12:56:41,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2388666.6666666665, ans=0.0
2023-11-23 12:56:47,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2388666.6666666665, ans=0.0
2023-11-23 12:57:01,115 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0
2023-11-23 12:57:17,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2388866.6666666665, ans=0.125
2023-11-23 12:57:18,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2388866.6666666665, ans=0.0
2023-11-23 12:57:27,867 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9650, loss[loss=0.04583, simple_loss=0.06043, pruned_loss=0.006409, audio_tagging_loss=0.009205, over 14137.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.0924, pruned_loss=0.01386, audio_tagging_loss=0.009151, over 3046604.95 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:57:30,912 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.376e+01 8.856e+01 9.540e+01 1.217e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 12:57:32,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2388933.3333333335, ans=0.125
2023-11-23 12:57:38,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358350
2023-11-23 12:57:48,554 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-23 12:58:16,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2389133.3333333335, ans=0.0
2023-11-23 12:58:21,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.48 vs. limit=6.0
2023-11-23 12:58:24,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2389200.0, ans=0.2
2023-11-23 12:58:31,814 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9700, loss[loss=0.06491, simple_loss=0.08387, pruned_loss=0.01443, audio_tagging_loss=0.008547, over 14686.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09259, pruned_loss=0.01393, audio_tagging_loss=0.009063, over 3050701.16 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:58:41,433 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.03 vs. limit=22.5
2023-11-23 12:58:41,886 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358400
2023-11-23 12:58:46,753 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.62 vs. limit=22.5
2023-11-23 12:58:47,512 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.55 vs. limit=15.0
2023-11-23 12:59:08,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2389400.0, ans=0.0
2023-11-23 12:59:36,168 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9750, loss[loss=0.06882, simple_loss=0.09567, pruned_loss=0.01341, audio_tagging_loss=0.007571, over 14888.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09315, pruned_loss=0.01402, audio_tagging_loss=0.008995, over 3053833.04 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:59:39,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.327e+01 8.730e+01 9.519e+01 2.872e+02, threshold=1.746e+02, percent-clipped=1.0
2023-11-23 12:59:43,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2389600.0, ans=0.1
2023-11-23 12:59:47,157 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358450
2023-11-23 12:59:58,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.93 vs. limit=10.0
2023-11-23 13:00:16,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2389800.0, ans=0.0
2023-11-23 13:00:40,789 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9800, loss[loss=0.06357, simple_loss=0.09139, pruned_loss=0.009726, audio_tagging_loss=0.008153, over 15277.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09284, pruned_loss=0.01408, audio_tagging_loss=0.008999, over 3044017.59 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:00:41,647 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.37 vs. limit=12.0
2023-11-23 13:00:51,359 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358500
2023-11-23 13:01:26,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2390133.3333333335, ans=0.07
2023-11-23 13:01:35,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2390200.0, ans=0.125
2023-11-23 13:01:38,908 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:01:45,031 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9850, loss[loss=0.07895, simple_loss=0.1192, pruned_loss=0.0138, audio_tagging_loss=0.005547, over 15908.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09332, pruned_loss=0.01415, audio_tagging_loss=0.008968, over 3040512.68 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:01:47,536 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.668e+01 8.339e+01 8.951e+01 9.575e+01 1.283e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 13:01:51,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2390266.6666666665, ans=0.125
2023-11-23 13:01:52,033 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.18 vs. limit=12.0
2023-11-23 13:01:52,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2390266.6666666665, ans=0.0
2023-11-23 13:01:55,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358550
2023-11-23 13:01:59,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2390333.3333333335, ans=0.04949747468305833
2023-11-23 13:02:14,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2390400.0, ans=0.1
2023-11-23 13:02:37,136 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-23 13:02:49,030 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9900, loss[loss=0.05854, simple_loss=0.08108, pruned_loss=0.009432, audio_tagging_loss=0.008572, over 14924.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09412, pruned_loss=0.01421, audio_tagging_loss=0.008879, over 3039197.36 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:02:59,923 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358600
2023-11-23 13:03:04,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2390666.6666666665, ans=0.0
2023-11-23 13:03:43,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2390866.6666666665, ans=0.125
2023-11-23 13:03:53,914 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 9950, loss[loss=0.04244, simple_loss=0.05002, pruned_loss=0.007923, audio_tagging_loss=0.009507, over 14576.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09344, pruned_loss=0.01416, audio_tagging_loss=0.008943, over 3042501.02 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:03:55,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2390933.3333333335, ans=0.0
2023-11-23 13:03:56,388 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.331e+01 8.358e+01 9.006e+01 9.900e+01 1.170e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 13:04:01,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2390933.3333333335, ans=0.125
2023-11-23 13:04:04,307 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358650
2023-11-23 13:04:12,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2391000.0, ans=0.0
2023-11-23 13:04:34,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2391133.3333333335, ans=0.125
2023-11-23 13:04:34,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2391133.3333333335, ans=0.125
2023-11-23 13:04:57,626 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10000, loss[loss=0.08546, simple_loss=0.1251, pruned_loss=0.01645, audio_tagging_loss=0.006454, over 15288.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09404, pruned_loss=0.01429, audio_tagging_loss=0.008881, over 3040496.84 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:05:01,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2391266.6666666665, ans=0.2
2023-11-23 13:05:07,298 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358700
2023-11-23 13:05:32,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2391400.0, ans=0.125
2023-11-23 13:06:01,419 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10050, loss[loss=0.07757, simple_loss=0.1036, pruned_loss=0.01722, audio_tagging_loss=0.008555, over 15098.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09309, pruned_loss=0.01418, audio_tagging_loss=0.008989, over 3037075.11 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:06:03,776 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.755e+01 8.322e+01 8.907e+01 9.751e+01 1.198e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 13:06:06,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.28 vs. limit=15.0
2023-11-23 13:06:08,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.36 vs. limit=22.5
2023-11-23 13:06:11,791 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358750
2023-11-23 13:06:13,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2391666.6666666665, ans=0.2
2023-11-23 13:06:49,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2391800.0, ans=0.2
2023-11-23 13:06:53,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2391866.6666666665, ans=0.1
2023-11-23 13:07:05,469 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10100, loss[loss=0.0666, simple_loss=0.09042, pruned_loss=0.01269, audio_tagging_loss=0.008701, over 16065.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09307, pruned_loss=0.0141, audio_tagging_loss=0.009011, over 3038872.74 frames. ], batch size: 61, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:07:16,139 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358800
2023-11-23 13:07:25,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2392000.0, ans=0.1
2023-11-23 13:07:42,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2392133.3333333335, ans=0.1
2023-11-23 13:07:57,877 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:08:08,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2392200.0, ans=0.05
2023-11-23 13:08:10,171 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10150, loss[loss=0.05157, simple_loss=0.06594, pruned_loss=0.007986, audio_tagging_loss=0.01061, over 15254.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09246, pruned_loss=0.01415, audio_tagging_loss=0.00919, over 3046158.66 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:08:13,706 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 8.548e+01 9.170e+01 9.737e+01 1.223e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 13:08:15,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2392266.6666666665, ans=0.0
2023-11-23 13:08:19,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358850
2023-11-23 13:08:40,895 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:08:48,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2392466.6666666665, ans=22.5
2023-11-23 13:08:55,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2392466.6666666665, ans=0.2
2023-11-23 13:09:06,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2392533.3333333335, ans=0.125
2023-11-23 13:09:13,950 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10200, loss[loss=0.06537, simple_loss=0.08905, pruned_loss=0.009745, audio_tagging_loss=0.0111, over 14533.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09295, pruned_loss=0.01414, audio_tagging_loss=0.009137, over 3052513.00 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:09:23,758 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358900
2023-11-23 13:09:39,617 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:10:11,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2392866.6666666665, ans=0.0
2023-11-23 13:10:18,484 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10250, loss[loss=0.07017, simple_loss=0.08455, pruned_loss=0.017, audio_tagging_loss=0.01089, over 14828.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09374, pruned_loss=0.0142, audio_tagging_loss=0.009205, over 3054578.98 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:10:22,069 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.314e+01 9.044e+01 9.616e+01 1.457e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 13:10:26,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=2392933.3333333335, ans=0.5
2023-11-23 13:10:28,398 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 358950
2023-11-23 13:10:28,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2392933.3333333335, ans=0.125
2023-11-23 13:11:23,019 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10300, loss[loss=0.08007, simple_loss=0.1066, pruned_loss=0.0162, audio_tagging_loss=0.01056, over 14348.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09407, pruned_loss=0.01429, audio_tagging_loss=0.009157, over 3054903.63 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:11:33,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359000
2023-11-23 13:11:37,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.81 vs. limit=22.5
2023-11-23 13:12:04,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2393466.6666666665, ans=0.0
2023-11-23 13:12:10,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2393466.6666666665, ans=0.1
2023-11-23 13:12:18,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2393533.3333333335, ans=0.1
2023-11-23 13:12:25,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2393600.0, ans=0.1
2023-11-23 13:12:26,648 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10350, loss[loss=0.0829, simple_loss=0.106, pruned_loss=0.02015, audio_tagging_loss=0.009742, over 15451.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09271, pruned_loss=0.01404, audio_tagging_loss=0.00935, over 3058316.23 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:12:30,154 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.344e+01 8.692e+01 9.326e+01 1.149e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-23 13:12:36,448 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359050
2023-11-23 13:12:54,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2393733.3333333335, ans=0.0
2023-11-23 13:12:56,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2393733.3333333335, ans=0.125
2023-11-23 13:13:09,559 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.64 vs. limit=15.0
2023-11-23 13:13:11,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2393800.0, ans=0.125
2023-11-23 13:13:17,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2393866.6666666665, ans=0.0
2023-11-23 13:13:17,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2393866.6666666665, ans=0.025
2023-11-23 13:13:28,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2393933.3333333335, ans=0.125
2023-11-23 13:13:30,221 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10400, loss[loss=0.06859, simple_loss=0.09006, pruned_loss=0.01485, audio_tagging_loss=0.008704, over 16083.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09214, pruned_loss=0.01405, audio_tagging_loss=0.009455, over 3059398.14 frames. ], batch size: 62, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:13:39,946 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359100
2023-11-23 13:13:42,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2394000.0, ans=0.125
2023-11-23 13:13:46,075 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.78 vs. limit=15.0
2023-11-23 13:13:59,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2394066.6666666665, ans=0.125
2023-11-23 13:14:05,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2394066.6666666665, ans=0.1
2023-11-23 13:14:33,790 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10450, loss[loss=0.05695, simple_loss=0.07278, pruned_loss=0.01048, audio_tagging_loss=0.01007, over 14793.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09193, pruned_loss=0.01401, audio_tagging_loss=0.009411, over 3058120.79 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:14:37,441 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.401e+01 9.132e+01 1.001e+02 1.338e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 13:14:43,718 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359150
2023-11-23 13:15:02,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2394400.0, ans=0.1
2023-11-23 13:15:29,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2394533.3333333335, ans=0.1
2023-11-23 13:15:37,339 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10500, loss[loss=0.0506, simple_loss=0.06642, pruned_loss=0.00651, audio_tagging_loss=0.01088, over 13912.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09227, pruned_loss=0.01405, audio_tagging_loss=0.009253, over 3056955.69 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:15:44,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2394600.0, ans=0.1
2023-11-23 13:15:47,176 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359200
2023-11-23 13:16:41,120 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10550, loss[loss=0.07521, simple_loss=0.1107, pruned_loss=0.01401, audio_tagging_loss=0.005861, over 14747.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09211, pruned_loss=0.014, audio_tagging_loss=0.009105, over 3044809.15 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:16:45,349 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.241e+01 8.844e+01 9.462e+01 1.240e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 13:16:47,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2394933.3333333335, ans=0.125
2023-11-23 13:16:49,376 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:16:51,723 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359250
2023-11-23 13:17:11,458 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.24 vs. limit=10.0
2023-11-23 13:17:15,172 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0
2023-11-23 13:17:45,207 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10600, loss[loss=0.0645, simple_loss=0.08288, pruned_loss=0.01471, audio_tagging_loss=0.008347, over 14730.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09118, pruned_loss=0.01388, audio_tagging_loss=0.0091, over 3038358.44 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:17:55,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-23 13:17:55,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359300
2023-11-23 13:18:01,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.84 vs. limit=6.0
2023-11-23 13:18:08,193 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.05 vs. limit=15.0
2023-11-23 13:18:16,576 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.31 vs. limit=15.0
2023-11-23 13:18:38,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2395533.3333333335, ans=0.0
2023-11-23 13:18:38,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=15.01 vs. limit=15.0
2023-11-23 13:18:49,242 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10650, loss[loss=0.05934, simple_loss=0.08184, pruned_loss=0.01138, audio_tagging_loss=0.007037, over 14706.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09156, pruned_loss=0.01391, audio_tagging_loss=0.009047, over 3034012.23 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:18:53,998 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.219e+01 8.921e+01 9.796e+01 1.166e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 13:18:59,593 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359350
2023-11-23 13:18:59,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2395600.0, ans=0.0
2023-11-23 13:19:35,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2395800.0, ans=0.2
2023-11-23 13:19:52,824 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10700, loss[loss=0.06034, simple_loss=0.08764, pruned_loss=0.008689, audio_tagging_loss=0.007831, over 15685.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09226, pruned_loss=0.01402, audio_tagging_loss=0.008971, over 3038068.96 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:19:58,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2395933.3333333335, ans=0.125
2023-11-23 13:20:03,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359400
2023-11-23 13:20:16,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2396000.0, ans=0.0
2023-11-23 13:20:18,242 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:20:20,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2396066.6666666665, ans=0.1
2023-11-23 13:20:21,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2396066.6666666665, ans=15.0
2023-11-23 13:20:25,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2396066.6666666665, ans=0.1
2023-11-23 13:20:30,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2396133.3333333335, ans=0.0
2023-11-23 13:20:54,927 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:20:57,064 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10750, loss[loss=0.06989, simple_loss=0.08831, pruned_loss=0.0152, audio_tagging_loss=0.01054, over 14194.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09181, pruned_loss=0.01394, audio_tagging_loss=0.009074, over 3037163.24 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:21:02,308 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.422e+01 8.871e+01 9.738e+01 1.299e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 13:21:02,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2396266.6666666665, ans=0.125
2023-11-23 13:21:07,225 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359450
2023-11-23 13:21:42,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.31 vs. limit=15.0
2023-11-23 13:21:45,044 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.26 vs. limit=15.0
2023-11-23 13:22:01,028 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10800, loss[loss=0.06449, simple_loss=0.08975, pruned_loss=0.01031, audio_tagging_loss=0.0093, over 15719.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09167, pruned_loss=0.01374, audio_tagging_loss=0.009072, over 3036742.93 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:22:11,229 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359500
2023-11-23 13:22:11,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2396600.0, ans=0.125
2023-11-23 13:22:11,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2396600.0, ans=0.125
2023-11-23 13:22:18,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2396666.6666666665, ans=0.0
2023-11-23 13:22:20,984 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.04 vs. limit=15.0
2023-11-23 13:22:41,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2396800.0, ans=0.125
2023-11-23 13:22:58,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2396866.6666666665, ans=0.1
2023-11-23 13:22:58,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.53 vs. limit=15.0
2023-11-23 13:23:04,155 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10850, loss[loss=0.02833, simple_loss=0.03262, pruned_loss=0.002133, audio_tagging_loss=0.009885, over 14596.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09166, pruned_loss=0.01385, audio_tagging_loss=0.009049, over 3041754.29 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:23:05,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2396933.3333333335, ans=0.125
2023-11-23 13:23:11,465 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.664e+01 8.074e+01 8.715e+01 9.516e+01 1.146e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-23 13:23:15,335 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359550
2023-11-23 13:23:17,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2397000.0, ans=6.0
2023-11-23 13:23:22,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2397000.0, ans=0.0
2023-11-23 13:23:23,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2397000.0, ans=15.0
2023-11-23 13:23:24,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2397000.0, ans=0.1
2023-11-23 13:23:46,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2397133.3333333335, ans=0.0
2023-11-23 13:23:47,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2397133.3333333335, ans=0.0
2023-11-23 13:23:48,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2397133.3333333335, ans=0.125
2023-11-23 13:24:05,396 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:24:07,852 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10900, loss[loss=0.05623, simple_loss=0.06689, pruned_loss=0.008522, audio_tagging_loss=0.01426, over 14527.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09228, pruned_loss=0.01384, audio_tagging_loss=0.009041, over 3041407.78 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:24:14,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.15 vs. limit=15.0
2023-11-23 13:24:18,506 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359600
2023-11-23 13:24:25,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2397333.3333333335, ans=0.125
2023-11-23 13:24:33,198 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.20 vs. limit=15.0
2023-11-23 13:24:41,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2397400.0, ans=0.0
2023-11-23 13:25:12,147 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 10950, loss[loss=0.08177, simple_loss=0.1037, pruned_loss=0.01738, audio_tagging_loss=0.01255, over 16332.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09143, pruned_loss=0.01383, audio_tagging_loss=0.009201, over 3042547.86 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:25:15,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2397600.0, ans=0.125
2023-11-23 13:25:18,224 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.360e+01 9.047e+01 9.764e+01 1.279e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 13:25:21,981 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359650
2023-11-23 13:25:24,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2397666.6666666665, ans=0.0
2023-11-23 13:25:24,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2397666.6666666665, ans=0.04949747468305833
2023-11-23 13:25:30,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2397666.6666666665, ans=0.0
2023-11-23 13:25:34,091 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.38 vs. limit=5.0
2023-11-23 13:25:41,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2397733.3333333335, ans=0.5
2023-11-23 13:25:42,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2397733.3333333335, ans=0.0
2023-11-23 13:25:50,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2397800.0, ans=0.125
2023-11-23 13:25:51,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=15.0
2023-11-23 13:26:03,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2397866.6666666665, ans=0.125
2023-11-23 13:26:11,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2397866.6666666665, ans=0.0
2023-11-23 13:26:16,464 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11000, loss[loss=0.06459, simple_loss=0.093, pruned_loss=0.009533, audio_tagging_loss=0.008553, over 14598.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09119, pruned_loss=0.01382, audio_tagging_loss=0.009226, over 3038560.29 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:26:19,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2397933.3333333335, ans=0.0
2023-11-23 13:26:19,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=15.0
2023-11-23 13:26:27,205 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359700
2023-11-23 13:26:28,856 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:26:57,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2398133.3333333335, ans=0.125
2023-11-23 13:27:22,163 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11050, loss[loss=0.04976, simple_loss=0.06088, pruned_loss=0.009037, audio_tagging_loss=0.01029, over 14390.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09206, pruned_loss=0.01409, audio_tagging_loss=0.009256, over 3036955.71 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:27:26,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2398266.6666666665, ans=0.125
2023-11-23 13:27:28,298 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.620e+01 9.222e+01 9.990e+01 1.274e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-23 13:27:28,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2398266.6666666665, ans=0.1
2023-11-23 13:27:32,678 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359750
2023-11-23 13:27:34,380 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.32 vs. limit=15.0
2023-11-23 13:27:50,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.89 vs. limit=15.0
2023-11-23 13:27:52,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2398400.0, ans=0.1
2023-11-23 13:28:06,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2398466.6666666665, ans=0.0
2023-11-23 13:28:13,861 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.71 vs. limit=6.0
2023-11-23 13:28:14,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2398533.3333333335, ans=0.125
2023-11-23 13:28:27,259 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11100, loss[loss=0.07068, simple_loss=0.08895, pruned_loss=0.01553, audio_tagging_loss=0.01067, over 12866.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09201, pruned_loss=0.01414, audio_tagging_loss=0.009348, over 3031007.47 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:28:37,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359800
2023-11-23 13:28:37,546 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:28:42,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2398666.6666666665, ans=0.0
2023-11-23 13:28:52,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0
2023-11-23 13:28:58,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2398733.3333333335, ans=0.0
2023-11-23 13:29:02,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2398733.3333333335, ans=0.125
2023-11-23 13:29:12,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2398800.0, ans=0.0
2023-11-23 13:29:13,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2398800.0, ans=0.1
2023-11-23 13:29:29,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2398866.6666666665, ans=0.2
2023-11-23 13:29:31,459 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11150, loss[loss=0.05464, simple_loss=0.06635, pruned_loss=0.009258, audio_tagging_loss=0.0122, over 14768.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09199, pruned_loss=0.01421, audio_tagging_loss=0.009467, over 3042807.65 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:29:32,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2398933.3333333335, ans=0.1
2023-11-23 13:29:37,476 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.369e+01 8.812e+01 9.424e+01 1.136e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 13:29:40,590 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.11 vs. limit=15.0
2023-11-23 13:29:41,379 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359850
2023-11-23 13:29:46,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2399000.0, ans=0.1
2023-11-23 13:29:47,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2399000.0, ans=0.125
2023-11-23 13:30:01,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2399066.6666666665, ans=0.1
2023-11-23 13:30:04,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2399066.6666666665, ans=0.0
2023-11-23 13:30:24,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2399200.0, ans=0.1
2023-11-23 13:30:27,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2399200.0, ans=0.125
2023-11-23 13:30:34,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2399266.6666666665, ans=0.125
2023-11-23 13:30:35,617 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11200, loss[loss=0.09062, simple_loss=0.1236, pruned_loss=0.02099, audio_tagging_loss=0.007808, over 15578.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09143, pruned_loss=0.01403, audio_tagging_loss=0.009548, over 3037995.15 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:30:46,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359900
2023-11-23 13:30:48,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2399333.3333333335, ans=0.1
2023-11-23 13:30:56,941 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.43 vs. limit=15.0
2023-11-23 13:31:21,375 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.36 vs. limit=15.0
2023-11-23 13:31:26,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2399533.3333333335, ans=0.125
2023-11-23 13:31:28,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2399533.3333333335, ans=0.0
2023-11-23 13:31:38,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2399533.3333333335, ans=10.0
2023-11-23 13:31:39,966 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11250, loss[loss=0.05118, simple_loss=0.06492, pruned_loss=0.009526, audio_tagging_loss=0.009196, over 15604.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09083, pruned_loss=0.01386, audio_tagging_loss=0.009519, over 3044040.18 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:31:42,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2399600.0, ans=0.1
2023-11-23 13:31:47,330 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.154e+01 8.449e+01 9.308e+01 1.008e+02 1.171e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-23 13:31:49,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 359950
2023-11-23 13:32:02,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2399666.6666666665, ans=0.2
2023-11-23 13:32:25,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2399800.0, ans=0.125
2023-11-23 13:32:43,510 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11300, loss[loss=0.08128, simple_loss=0.1142, pruned_loss=0.01874, audio_tagging_loss=0.005426, over 15570.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.0917, pruned_loss=0.01398, audio_tagging_loss=0.00931, over 3041001.25 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:32:43,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2399933.3333333335, ans=0.125
2023-11-23 13:32:53,473 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360000
2023-11-23 13:33:28,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2400133.3333333335, ans=0.125
2023-11-23 13:33:50,669 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11350, loss[loss=0.06437, simple_loss=0.08174, pruned_loss=0.01267, audio_tagging_loss=0.01083, over 16615.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09122, pruned_loss=0.01393, audio_tagging_loss=0.009204, over 3047630.31 frames. ], batch size: 61, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:33:52,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2400266.6666666665, ans=0.0
2023-11-23 13:33:58,000 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.206e+01 9.153e+01 9.834e+01 1.397e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 13:33:58,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2400266.6666666665, ans=0.125
2023-11-23 13:34:00,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360050
2023-11-23 13:34:20,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2400400.0, ans=0.125
2023-11-23 13:34:25,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2400400.0, ans=0.125
2023-11-23 13:34:44,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2400533.3333333335, ans=0.125
2023-11-23 13:34:54,324 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11400, loss[loss=0.06354, simple_loss=0.08718, pruned_loss=0.01312, audio_tagging_loss=0.006826, over 14835.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09203, pruned_loss=0.01403, audio_tagging_loss=0.009102, over 3049694.55 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:35:04,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360100
2023-11-23 13:35:15,560 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.33 vs. limit=15.0
2023-11-23 13:35:54,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2400866.6666666665, ans=0.0
2023-11-23 13:35:57,412 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11450, loss[loss=0.07043, simple_loss=0.1014, pruned_loss=0.01267, audio_tagging_loss=0.007041, over 15603.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09298, pruned_loss=0.01414, audio_tagging_loss=0.008974, over 3050187.80 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:35:58,033 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.46 vs. limit=15.0
2023-11-23 13:36:04,746 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.159e+01 8.794e+01 9.452e+01 1.261e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 13:36:07,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360150
2023-11-23 13:36:16,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2401000.0, ans=0.125
2023-11-23 13:36:16,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.99 vs. limit=15.0
2023-11-23 13:36:27,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2401066.6666666665, ans=0.025
2023-11-23 13:36:30,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2401066.6666666665, ans=0.2
2023-11-23 13:36:33,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.02 vs. limit=15.0
2023-11-23 13:36:35,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2401133.3333333335, ans=0.2
2023-11-23 13:36:44,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2401133.3333333335, ans=0.125
2023-11-23 13:36:54,745 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.51 vs. limit=10.0
2023-11-23 13:37:01,761 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11500, loss[loss=0.0678, simple_loss=0.09731, pruned_loss=0.009542, audio_tagging_loss=0.009608, over 16983.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09239, pruned_loss=0.01415, audio_tagging_loss=0.009027, over 3042475.20 frames. ], batch size: 63, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:37:12,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360200
2023-11-23 13:37:18,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2401333.3333333335, ans=0.2
2023-11-23 13:37:28,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2401400.0, ans=0.2
2023-11-23 13:37:38,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2401400.0, ans=0.125
2023-11-23 13:37:45,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2401466.6666666665, ans=0.125
2023-11-23 13:37:53,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2401533.3333333335, ans=0.0
2023-11-23 13:37:54,158 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.81 vs. limit=15.0
2023-11-23 13:37:54,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2401533.3333333335, ans=0.125
2023-11-23 13:37:58,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2401533.3333333335, ans=0.125
2023-11-23 13:37:59,196 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.56 vs. limit=12.0
2023-11-23 13:38:07,669 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11550, loss[loss=0.06163, simple_loss=0.07993, pruned_loss=0.01254, audio_tagging_loss=0.009122, over 15010.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09238, pruned_loss=0.01406, audio_tagging_loss=0.009004, over 3046243.48 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:38:09,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2401600.0, ans=6.0
2023-11-23 13:38:15,058 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.476e+01 9.086e+01 9.753e+01 1.372e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 13:38:18,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360250
2023-11-23 13:38:47,260 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:39:02,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2401866.6666666665, ans=0.125
2023-11-23 13:39:11,787 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11600, loss[loss=0.06116, simple_loss=0.08259, pruned_loss=0.01012, audio_tagging_loss=0.00975, over 14205.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09341, pruned_loss=0.01435, audio_tagging_loss=0.009056, over 3047273.21 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:39:11,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2401933.3333333335, ans=10.0
2023-11-23 13:39:21,680 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360300
2023-11-23 13:39:45,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2402066.6666666665, ans=0.0
2023-11-23 13:39:49,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2402133.3333333335, ans=0.1
2023-11-23 13:39:53,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2402133.3333333335, ans=0.0
2023-11-23 13:39:55,365 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.92 vs. limit=15.0
2023-11-23 13:40:07,471 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:40:07,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2402200.0, ans=0.125
2023-11-23 13:40:09,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2402200.0, ans=0.1
2023-11-23 13:40:14,612 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11650, loss[loss=0.05249, simple_loss=0.06987, pruned_loss=0.007948, audio_tagging_loss=0.009605, over 15085.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09257, pruned_loss=0.01403, audio_tagging_loss=0.009042, over 3045769.21 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:40:18,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.66 vs. limit=15.0
2023-11-23 13:40:21,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2402266.6666666665, ans=0.0
2023-11-23 13:40:22,548 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.635e+01 8.364e+01 9.300e+01 1.004e+02 1.361e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-23 13:40:25,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360350
2023-11-23 13:40:39,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2402400.0, ans=0.0
2023-11-23 13:40:45,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2402400.0, ans=0.125
2023-11-23 13:40:54,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2402466.6666666665, ans=0.2
2023-11-23 13:41:00,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.13 vs. limit=8.0
2023-11-23 13:41:18,472 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11700, loss[loss=0.06907, simple_loss=0.09224, pruned_loss=0.01253, audio_tagging_loss=0.01042, over 14869.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09203, pruned_loss=0.01398, audio_tagging_loss=0.0091, over 3047623.61 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:41:24,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2402600.0, ans=0.125
2023-11-23 13:41:29,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360400
2023-11-23 13:41:33,392 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:41:39,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2402666.6666666665, ans=0.0
2023-11-23 13:41:40,681 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.91 vs. limit=15.0
2023-11-23 13:41:41,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2402666.6666666665, ans=0.0
2023-11-23 13:41:47,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2402733.3333333335, ans=0.125
2023-11-23 13:41:57,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2402800.0, ans=0.125
2023-11-23 13:42:15,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2402866.6666666665, ans=0.125
2023-11-23 13:42:20,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2402866.6666666665, ans=0.05
2023-11-23 13:42:23,004 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11750, loss[loss=0.06511, simple_loss=0.08027, pruned_loss=0.01353, audio_tagging_loss=0.01145, over 14621.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09275, pruned_loss=0.01402, audio_tagging_loss=0.009136, over 3040738.73 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:42:24,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2402933.3333333335, ans=0.125
2023-11-23 13:42:25,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2402933.3333333335, ans=0.0
2023-11-23 13:42:25,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2402933.3333333335, ans=0.125
2023-11-23 13:42:27,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2402933.3333333335, ans=0.125
2023-11-23 13:42:32,127 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.747e+01 8.361e+01 8.957e+01 9.749e+01 1.339e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 13:42:33,411 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360450
2023-11-23 13:42:40,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2403000.0, ans=0.0
2023-11-23 13:42:45,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2403000.0, ans=0.125
2023-11-23 13:42:47,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2403066.6666666665, ans=0.125
2023-11-23 13:43:06,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2403133.3333333335, ans=0.2
2023-11-23 13:43:23,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2403200.0, ans=0.125
2023-11-23 13:43:24,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2403200.0, ans=0.0
2023-11-23 13:43:26,979 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11800, loss[loss=0.07192, simple_loss=0.09668, pruned_loss=0.014, audio_tagging_loss=0.009586, over 14978.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09174, pruned_loss=0.01385, audio_tagging_loss=0.009221, over 3040451.11 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:43:30,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5
2023-11-23 13:43:37,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360500
2023-11-23 13:43:50,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2403333.3333333335, ans=0.125
2023-11-23 13:43:55,926 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.58 vs. limit=12.0
2023-11-23 13:44:03,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2403466.6666666665, ans=0.0
2023-11-23 13:44:11,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2403466.6666666665, ans=0.125
2023-11-23 13:44:31,215 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11850, loss[loss=0.05387, simple_loss=0.07022, pruned_loss=0.008304, audio_tagging_loss=0.01045, over 13878.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09099, pruned_loss=0.01361, audio_tagging_loss=0.00935, over 3042479.35 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:44:37,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2403600.0, ans=0.125
2023-11-23 13:44:38,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2403600.0, ans=0.125
2023-11-23 13:44:40,399 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.325e+01 8.959e+01 9.729e+01 1.337e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 13:44:41,711 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360550
2023-11-23 13:44:44,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2403666.6666666665, ans=0.1
2023-11-23 13:44:50,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2403666.6666666665, ans=0.05
2023-11-23 13:45:08,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2403800.0, ans=0.0
2023-11-23 13:45:20,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.55 vs. limit=10.0
2023-11-23 13:45:29,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2403866.6666666665, ans=0.1
2023-11-23 13:45:30,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2403866.6666666665, ans=0.0
2023-11-23 13:45:33,423 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.73 vs. limit=15.0
2023-11-23 13:45:35,381 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11900, loss[loss=0.08577, simple_loss=0.1156, pruned_loss=0.01908, audio_tagging_loss=0.00888, over 13942.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.0916, pruned_loss=0.01364, audio_tagging_loss=0.009388, over 3049030.09 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:45:36,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2403933.3333333335, ans=0.0
2023-11-23 13:45:43,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2403933.3333333335, ans=0.125
2023-11-23 13:45:45,378 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360600
2023-11-23 13:45:49,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2404000.0, ans=0.0
2023-11-23 13:45:49,587 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.63 vs. limit=15.0
2023-11-23 13:45:56,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2404000.0, ans=0.2
2023-11-23 13:46:08,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.96 vs. limit=15.0
2023-11-23 13:46:13,280 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.50 vs. limit=15.0
2023-11-23 13:46:28,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2404200.0, ans=0.0
2023-11-23 13:46:33,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2404200.0, ans=0.1
2023-11-23 13:46:40,743 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 11950, loss[loss=0.06178, simple_loss=0.08337, pruned_loss=0.01069, audio_tagging_loss=0.00941, over 14762.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09171, pruned_loss=0.0137, audio_tagging_loss=0.009425, over 3051057.94 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:46:45,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.14 vs. limit=15.0
2023-11-23 13:46:50,082 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.271e+01 8.961e+01 9.651e+01 1.560e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 13:46:51,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360650
2023-11-23 13:46:53,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2404333.3333333335, ans=0.125
2023-11-23 13:47:01,237 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.87 vs. limit=10.0
2023-11-23 13:47:15,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2404400.0, ans=0.0
2023-11-23 13:47:37,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2404533.3333333335, ans=0.0
2023-11-23 13:47:42,984 INFO [train_asr.py:1221] (2/4) Epoch 30, batch 12000, loss[loss=0.04805, simple_loss=0.06042, pruned_loss=0.00752, audio_tagging_loss=0.01033, over 14812.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09221, pruned_loss=0.01384, audio_tagging_loss=0.009494, over 3048347.08 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:47:42,985 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 13:48:25,613 INFO [train_asr.py:1253] (2/4) Epoch 30, validation: loss=0.05798, simple_loss=0.05115, pruned_loss=0.00515, audio_tagging_loss=0.02725, over 4681554.00 frames. 
2023-11-23 13:48:25,614 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 13:48:29,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2404600.0, ans=0.125
2023-11-23 13:48:35,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360700
2023-11-23 13:48:37,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2404666.6666666665, ans=0.2
2023-11-23 13:48:51,113 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0
2023-11-23 13:49:30,367 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 0, loss[loss=0.07475, simple_loss=0.0797, pruned_loss=0.01283, audio_tagging_loss=0.02207, over 16172.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.0797, pruned_loss=0.01283, audio_tagging_loss=0.02207, over 16172.00 frames. ], batch size: 63, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:49:30,367 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 13:50:01,263 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9642, 3.7501, 4.8890, 4.4902], device='cuda:2')
2023-11-23 13:50:05,330 INFO [train_asr.py:1253] (2/4) Epoch 31, validation: loss=0.05797, simple_loss=0.05105, pruned_loss=0.005059, audio_tagging_loss=0.02738, over 4681554.00 frames. 
2023-11-23 13:50:05,331 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 13:50:15,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.78 vs. limit=15.0
2023-11-23 13:50:47,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.427e+01 8.784e+01 9.433e+01 1.048e+02 1.296e+02, threshold=1.887e+02, percent-clipped=0.0
2023-11-23 13:50:48,926 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360750
2023-11-23 13:50:53,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2404966.6666666665, ans=0.5
2023-11-23 13:50:56,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2405033.3333333335, ans=0.07
2023-11-23 13:51:10,461 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 50, loss[loss=0.06697, simple_loss=0.07678, pruned_loss=0.009714, audio_tagging_loss=0.01887, over 16426.00 frames. ], tot_loss[loss=0.07964, simple_loss=0.09568, pruned_loss=0.01398, audio_tagging_loss=0.01782, over 692398.87 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:51:27,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2405166.6666666665, ans=0.0
2023-11-23 13:51:29,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2405166.6666666665, ans=0.0
2023-11-23 13:51:30,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2405166.6666666665, ans=0.5
2023-11-23 13:51:35,084 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.08 vs. limit=22.5
2023-11-23 13:51:49,784 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.29 vs. limit=12.0
2023-11-23 13:51:53,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360800
2023-11-23 13:52:16,515 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 100, loss[loss=0.09768, simple_loss=0.1313, pruned_loss=0.02091, audio_tagging_loss=0.01112, over 16290.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09452, pruned_loss=0.0135, audio_tagging_loss=0.01662, over 1216243.78 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:52:21,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2405433.3333333335, ans=0.125
2023-11-23 13:52:48,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2405566.6666666665, ans=0.1
2023-11-23 13:52:57,687 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.885e+01 9.566e+01 1.092e+02 1.525e+02, threshold=1.913e+02, percent-clipped=0.0
2023-11-23 13:52:59,053 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360850
2023-11-23 13:53:04,234 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.41 vs. limit=12.0
2023-11-23 13:53:14,040 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.10 vs. limit=22.5
2023-11-23 13:53:19,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2405766.6666666665, ans=0.015
2023-11-23 13:53:20,664 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 150, loss[loss=0.06283, simple_loss=0.07144, pruned_loss=0.01245, audio_tagging_loss=0.01466, over 15361.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09234, pruned_loss=0.0134, audio_tagging_loss=0.0149, over 1619094.19 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:54:02,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2405966.6666666665, ans=0.0
2023-11-23 13:54:03,658 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360900
2023-11-23 13:54:07,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2405966.6666666665, ans=0.0
2023-11-23 13:54:25,023 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 200, loss[loss=0.0669, simple_loss=0.0841, pruned_loss=0.01556, audio_tagging_loss=0.009286, over 15830.00 frames. ], tot_loss[loss=0.07351, simple_loss=0.09354, pruned_loss=0.01364, audio_tagging_loss=0.01311, over 1940887.41 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:54:30,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2406100.0, ans=0.0
2023-11-23 13:54:35,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.80 vs. limit=22.5
2023-11-23 13:55:06,202 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.350e+01 9.246e+01 9.961e+01 1.409e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-23 13:55:07,601 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 360950
2023-11-23 13:55:11,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2406300.0, ans=0.0
2023-11-23 13:55:24,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2406366.6666666665, ans=0.125
2023-11-23 13:55:30,803 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 250, loss[loss=0.06172, simple_loss=0.08534, pruned_loss=0.009933, audio_tagging_loss=0.009117, over 15178.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09239, pruned_loss=0.01379, audio_tagging_loss=0.01198, over 2190986.43 frames. ], batch size: 57, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:55:34,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2406433.3333333335, ans=0.125
2023-11-23 13:55:41,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2406500.0, ans=0.125
2023-11-23 13:56:13,316 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361000
2023-11-23 13:56:16,461 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.57 vs. limit=15.0
2023-11-23 13:56:25,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2406700.0, ans=0.125
2023-11-23 13:56:35,023 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 300, loss[loss=0.05735, simple_loss=0.0674, pruned_loss=0.01192, audio_tagging_loss=0.01172, over 15151.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09208, pruned_loss=0.01396, audio_tagging_loss=0.01118, over 2375856.19 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:56:35,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2406766.6666666665, ans=0.05
2023-11-23 13:56:35,581 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.22 vs. limit=15.0
2023-11-23 13:56:46,451 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:56:48,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2406833.3333333335, ans=0.1
2023-11-23 13:57:01,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2406900.0, ans=0.2
2023-11-23 13:57:16,327 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.477e+01 9.124e+01 9.859e+01 1.340e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 13:57:17,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361050
2023-11-23 13:57:25,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2407033.3333333335, ans=0.5
2023-11-23 13:57:28,319 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.77 vs. limit=6.0
2023-11-23 13:57:29,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2407033.3333333335, ans=15.0
2023-11-23 13:57:32,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2407033.3333333335, ans=0.125
2023-11-23 13:57:38,746 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 350, loss[loss=0.09875, simple_loss=0.1467, pruned_loss=0.01938, audio_tagging_loss=0.006019, over 16998.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09111, pruned_loss=0.01361, audio_tagging_loss=0.01057, over 2530847.17 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:57:47,971 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=12.0
2023-11-23 13:58:02,657 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.11 vs. limit=22.5
2023-11-23 13:58:10,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2407233.3333333335, ans=0.125
2023-11-23 13:58:21,952 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361100
2023-11-23 13:58:44,372 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 400, loss[loss=0.0819, simple_loss=0.1103, pruned_loss=0.01884, audio_tagging_loss=0.007888, over 14509.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09242, pruned_loss=0.01366, audio_tagging_loss=0.01003, over 2647642.11 frames. ], batch size: 53, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:58:55,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2407433.3333333335, ans=0.2
2023-11-23 13:59:11,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2407566.6666666665, ans=0.1
2023-11-23 13:59:27,038 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.381e+01 8.890e+01 9.709e+01 1.192e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 13:59:27,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361150
2023-11-23 13:59:30,557 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:59:49,034 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 450, loss[loss=0.07698, simple_loss=0.1104, pruned_loss=0.01346, audio_tagging_loss=0.008331, over 16659.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09189, pruned_loss=0.01368, audio_tagging_loss=0.009794, over 2738170.95 frames. ], batch size: 63, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:59:54,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2407766.6666666665, ans=0.125
2023-11-23 14:00:05,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2407833.3333333335, ans=0.125
2023-11-23 14:00:14,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2407900.0, ans=0.125
2023-11-23 14:00:14,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2407900.0, ans=0.125
2023-11-23 14:00:25,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2407900.0, ans=0.2
2023-11-23 14:00:25,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2407900.0, ans=0.125
2023-11-23 14:00:31,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361200
2023-11-23 14:00:31,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2407966.6666666665, ans=0.1
2023-11-23 14:00:32,121 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.91 vs. limit=22.5
2023-11-23 14:00:40,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2408033.3333333335, ans=0.1
2023-11-23 14:00:52,577 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 500, loss[loss=0.06573, simple_loss=0.0824, pruned_loss=0.01452, audio_tagging_loss=0.01001, over 17068.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.0912, pruned_loss=0.01365, audio_tagging_loss=0.00957, over 2806229.84 frames. ], batch size: 62, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:00:52,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2408100.0, ans=0.1
2023-11-23 14:01:08,064 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.42 vs. limit=6.0
2023-11-23 14:01:16,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2408166.6666666665, ans=0.0
2023-11-23 14:01:22,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2408233.3333333335, ans=0.0
2023-11-23 14:01:35,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361250
2023-11-23 14:01:38,175 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.202e+01 8.738e+01 9.216e+01 1.113e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 14:01:41,437 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.57 vs. limit=15.0
2023-11-23 14:01:58,076 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 550, loss[loss=0.06205, simple_loss=0.07838, pruned_loss=0.01298, audio_tagging_loss=0.009883, over 14754.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09132, pruned_loss=0.01362, audio_tagging_loss=0.00951, over 2860177.72 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:02:03,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2408433.3333333335, ans=0.0
2023-11-23 14:02:19,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2408500.0, ans=0.125
2023-11-23 14:02:24,649 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.82 vs. limit=15.0
2023-11-23 14:02:25,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.18 vs. limit=15.0
2023-11-23 14:02:30,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2408566.6666666665, ans=0.0
2023-11-23 14:02:40,984 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361300
2023-11-23 14:02:45,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2408633.3333333335, ans=6.0
2023-11-23 14:03:02,870 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 600, loss[loss=0.05412, simple_loss=0.06002, pruned_loss=0.01508, audio_tagging_loss=0.009033, over 14542.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09064, pruned_loss=0.01371, audio_tagging_loss=0.009442, over 2900623.22 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:03:14,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2408833.3333333335, ans=0.0
2023-11-23 14:03:21,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2408833.3333333335, ans=0.025
2023-11-23 14:03:25,342 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.79 vs. limit=15.0
2023-11-23 14:03:31,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.50 vs. limit=15.0
2023-11-23 14:03:32,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2408900.0, ans=0.2
2023-11-23 14:03:45,717 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361350
2023-11-23 14:03:48,038 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.522e+01 9.044e+01 1.001e+02 1.721e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 14:03:53,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2409033.3333333335, ans=0.125
2023-11-23 14:04:06,502 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 650, loss[loss=0.05766, simple_loss=0.07734, pruned_loss=0.01019, audio_tagging_loss=0.008797, over 13424.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09102, pruned_loss=0.0137, audio_tagging_loss=0.009352, over 2937858.41 frames. ], batch size: 52, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:04:16,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2409100.0, ans=0.125
2023-11-23 14:04:40,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2409233.3333333335, ans=0.025
2023-11-23 14:04:46,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2409300.0, ans=0.0
2023-11-23 14:04:49,496 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361400
2023-11-23 14:04:50,750 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:05:04,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2409366.6666666665, ans=0.125
2023-11-23 14:05:12,024 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 700, loss[loss=0.06241, simple_loss=0.07909, pruned_loss=0.01359, audio_tagging_loss=0.009276, over 14639.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09124, pruned_loss=0.01358, audio_tagging_loss=0.009254, over 2965914.76 frames. ], batch size: 57, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:05:42,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2409566.6666666665, ans=0.025
2023-11-23 14:05:50,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2409633.3333333335, ans=0.0
2023-11-23 14:05:52,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2409633.3333333335, ans=0.125
2023-11-23 14:05:54,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361450
2023-11-23 14:05:57,071 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.383e+01 9.026e+01 9.852e+01 1.230e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 14:06:02,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2409700.0, ans=0.0
2023-11-23 14:06:09,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2409700.0, ans=0.125
2023-11-23 14:06:17,514 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 750, loss[loss=0.06289, simple_loss=0.07898, pruned_loss=0.01229, audio_tagging_loss=0.01111, over 16626.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09215, pruned_loss=0.01369, audio_tagging_loss=0.009289, over 2981929.82 frames. ], batch size: 61, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:06:23,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2409766.6666666665, ans=0.0
2023-11-23 14:06:24,423 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.67 vs. limit=15.0
2023-11-23 14:06:32,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2409833.3333333335, ans=0.015
2023-11-23 14:06:36,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2409833.3333333335, ans=0.1
2023-11-23 14:06:42,586 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.80 vs. limit=6.0
2023-11-23 14:06:49,446 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:06:59,679 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361500
2023-11-23 14:07:07,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2410033.3333333335, ans=0.0
2023-11-23 14:07:16,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2410033.3333333335, ans=0.0
2023-11-23 14:07:21,102 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 800, loss[loss=0.06872, simple_loss=0.09783, pruned_loss=0.01033, audio_tagging_loss=0.009471, over 14575.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09329, pruned_loss=0.014, audio_tagging_loss=0.00928, over 2998581.04 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:07:40,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2410166.6666666665, ans=0.04949747468305833
2023-11-23 14:07:41,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2410166.6666666665, ans=0.0
2023-11-23 14:08:04,595 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361550
2023-11-23 14:08:06,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.503e+01 9.179e+01 9.718e+01 1.363e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-23 14:08:08,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2410300.0, ans=0.2
2023-11-23 14:08:08,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2410300.0, ans=0.125
2023-11-23 14:08:26,936 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 850, loss[loss=0.05343, simple_loss=0.07124, pruned_loss=0.007493, audio_tagging_loss=0.01032, over 14472.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09263, pruned_loss=0.01381, audio_tagging_loss=0.0094, over 3008646.17 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:08:33,967 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.30 vs. limit=15.0
2023-11-23 14:09:05,129 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.80 vs. limit=15.0
2023-11-23 14:09:09,275 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361600
2023-11-23 14:09:10,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2410633.3333333335, ans=0.0
2023-11-23 14:09:18,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.25 vs. limit=15.0
2023-11-23 14:09:30,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2410700.0, ans=0.125
2023-11-23 14:09:32,511 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 900, loss[loss=0.04954, simple_loss=0.063, pruned_loss=0.008434, audio_tagging_loss=0.009604, over 15732.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09261, pruned_loss=0.0139, audio_tagging_loss=0.009462, over 3015304.66 frames. ], batch size: 61, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:09:45,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2410833.3333333335, ans=0.1
2023-11-23 14:09:45,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2410833.3333333335, ans=0.125
2023-11-23 14:10:09,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2410900.0, ans=0.125
2023-11-23 14:10:15,780 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361650
2023-11-23 14:10:18,171 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.422e+01 9.031e+01 9.667e+01 1.145e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 14:10:22,502 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.05 vs. limit=10.0
2023-11-23 14:10:37,452 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 950, loss[loss=0.07099, simple_loss=0.09475, pruned_loss=0.01647, audio_tagging_loss=0.007149, over 14726.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09201, pruned_loss=0.01369, audio_tagging_loss=0.009436, over 3026645.49 frames. ], batch size: 57, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:10:39,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2411100.0, ans=0.2
2023-11-23 14:10:42,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2411100.0, ans=0.1
2023-11-23 14:10:57,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2411166.6666666665, ans=0.0
2023-11-23 14:11:02,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2411233.3333333335, ans=0.1
2023-11-23 14:11:19,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361700
2023-11-23 14:11:20,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2411300.0, ans=0.0
2023-11-23 14:11:33,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2411366.6666666665, ans=0.0
2023-11-23 14:11:41,905 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1000, loss[loss=0.05742, simple_loss=0.08023, pruned_loss=0.01056, audio_tagging_loss=0.006738, over 15064.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09201, pruned_loss=0.01362, audio_tagging_loss=0.009295, over 3038393.21 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:11:55,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2411500.0, ans=0.0
2023-11-23 14:11:56,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2411500.0, ans=0.0
2023-11-23 14:11:57,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.93 vs. limit=6.0
2023-11-23 14:12:08,897 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:12:13,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.08 vs. limit=12.0
2023-11-23 14:12:19,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2411633.3333333335, ans=0.0
2023-11-23 14:12:23,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2411633.3333333335, ans=0.0
2023-11-23 14:12:24,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361750
2023-11-23 14:12:26,479 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.336e+01 9.096e+01 9.576e+01 1.309e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-23 14:12:44,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2411700.0, ans=0.125
2023-11-23 14:12:47,002 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1050, loss[loss=0.09744, simple_loss=0.1369, pruned_loss=0.02094, audio_tagging_loss=0.008034, over 16634.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09188, pruned_loss=0.01365, audio_tagging_loss=0.009197, over 3044954.84 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:12:52,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2411766.6666666665, ans=0.1
2023-11-23 14:12:53,774 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.68 vs. limit=15.0
2023-11-23 14:12:58,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2411833.3333333335, ans=0.0
2023-11-23 14:13:03,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2411833.3333333335, ans=0.125
2023-11-23 14:13:14,076 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:13:27,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2411966.6666666665, ans=0.0
2023-11-23 14:13:30,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361800
2023-11-23 14:13:41,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2412033.3333333335, ans=0.0
2023-11-23 14:13:47,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=12.0
2023-11-23 14:13:51,456 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1100, loss[loss=0.08234, simple_loss=0.1059, pruned_loss=0.0188, audio_tagging_loss=0.01059, over 15341.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09169, pruned_loss=0.01366, audio_tagging_loss=0.009103, over 3045089.61 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:13:53,961 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:14:14,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2412166.6666666665, ans=0.125
2023-11-23 14:14:34,873 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361850
2023-11-23 14:14:37,255 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.240e+01 8.744e+01 9.286e+01 1.135e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 14:14:46,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2412366.6666666665, ans=0.125
2023-11-23 14:14:56,290 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1150, loss[loss=0.06614, simple_loss=0.08431, pruned_loss=0.01614, audio_tagging_loss=0.007848, over 14438.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.0913, pruned_loss=0.01351, audio_tagging_loss=0.009102, over 3043707.62 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:14:57,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2412433.3333333335, ans=0.125
2023-11-23 14:15:15,134 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.19 vs. limit=8.0
2023-11-23 14:15:39,829 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361900
2023-11-23 14:16:02,437 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1200, loss[loss=0.05976, simple_loss=0.082, pruned_loss=0.008329, audio_tagging_loss=0.01043, over 14656.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09148, pruned_loss=0.01346, audio_tagging_loss=0.00903, over 3040292.10 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:16:06,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2412766.6666666665, ans=0.1
2023-11-23 14:16:44,764 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 361950
2023-11-23 14:16:48,313 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.465e+01 9.389e+01 1.006e+02 1.334e+02, threshold=1.878e+02, percent-clipped=0.0
2023-11-23 14:17:02,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2413033.3333333335, ans=0.2
2023-11-23 14:17:04,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2413033.3333333335, ans=0.0
2023-11-23 14:17:04,459 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.00 vs. limit=15.0
2023-11-23 14:17:06,319 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1250, loss[loss=0.05853, simple_loss=0.0734, pruned_loss=0.01197, audio_tagging_loss=0.009859, over 14359.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09143, pruned_loss=0.01354, audio_tagging_loss=0.008987, over 3038594.35 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:17:33,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2413233.3333333335, ans=0.125
2023-11-23 14:17:47,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2413300.0, ans=0.125
2023-11-23 14:17:50,016 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362000
2023-11-23 14:17:51,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2413300.0, ans=15.0
2023-11-23 14:17:56,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2413300.0, ans=0.09899494936611666
2023-11-23 14:17:57,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.37 vs. limit=22.5
2023-11-23 14:18:11,301 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1300, loss[loss=0.03894, simple_loss=0.04612, pruned_loss=0.005563, audio_tagging_loss=0.01032, over 14370.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09131, pruned_loss=0.01362, audio_tagging_loss=0.008983, over 3034081.21 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:18:14,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2413433.3333333335, ans=0.0
2023-11-23 14:18:21,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2413433.3333333335, ans=0.2
2023-11-23 14:18:25,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2413500.0, ans=0.0
2023-11-23 14:18:36,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2413500.0, ans=0.125
2023-11-23 14:18:42,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2413566.6666666665, ans=0.125
2023-11-23 14:18:45,324 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.90 vs. limit=10.0
2023-11-23 14:18:47,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2413566.6666666665, ans=0.125
2023-11-23 14:18:50,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.22 vs. limit=12.0
2023-11-23 14:18:54,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362050
2023-11-23 14:18:55,078 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:18:58,453 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.540e+01 8.227e+01 8.895e+01 9.502e+01 1.940e+02, threshold=1.779e+02, percent-clipped=1.0
2023-11-23 14:19:17,498 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1350, loss[loss=0.07942, simple_loss=0.1169, pruned_loss=0.01281, audio_tagging_loss=0.008142, over 16394.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09177, pruned_loss=0.01359, audio_tagging_loss=0.008952, over 3035447.09 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:19:19,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2413766.6666666665, ans=0.125
2023-11-23 14:19:50,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2413900.0, ans=0.0
2023-11-23 14:20:01,087 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362100
2023-11-23 14:20:01,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2413966.6666666665, ans=0.125
2023-11-23 14:20:04,683 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:20:18,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2414033.3333333335, ans=0.125
2023-11-23 14:20:22,792 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1400, loss[loss=0.07825, simple_loss=0.1031, pruned_loss=0.01593, audio_tagging_loss=0.01076, over 15524.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09201, pruned_loss=0.01359, audio_tagging_loss=0.008956, over 3039440.71 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:20:39,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2414166.6666666665, ans=0.2
2023-11-23 14:21:00,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten.whitening_limit, batch_count=2414233.3333333335, ans=15.0
2023-11-23 14:21:06,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362150
2023-11-23 14:21:06,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.13 vs. limit=22.5
2023-11-23 14:21:07,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2414300.0, ans=0.125
2023-11-23 14:21:09,852 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.379e+01 8.906e+01 9.568e+01 1.474e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 14:21:27,367 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1450, loss[loss=0.07524, simple_loss=0.09852, pruned_loss=0.01786, audio_tagging_loss=0.008111, over 14185.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09283, pruned_loss=0.01375, audio_tagging_loss=0.008907, over 3045963.12 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:21:27,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.39 vs. limit=10.0
2023-11-23 14:21:27,991 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.51 vs. limit=15.0
2023-11-23 14:21:29,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2414433.3333333335, ans=0.125
2023-11-23 14:21:43,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2414500.0, ans=0.125
2023-11-23 14:22:02,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-23 14:22:04,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2414566.6666666665, ans=0.125
2023-11-23 14:22:10,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362200
2023-11-23 14:22:34,351 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1500, loss[loss=0.0823, simple_loss=0.1167, pruned_loss=0.01517, audio_tagging_loss=0.008785, over 15101.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.0932, pruned_loss=0.01386, audio_tagging_loss=0.008956, over 3045051.95 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:22:48,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2414833.3333333335, ans=0.125
2023-11-23 14:22:50,570 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.58 vs. limit=22.5
2023-11-23 14:23:16,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362250
2023-11-23 14:23:20,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.572e+01 9.255e+01 1.007e+02 1.352e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-23 14:23:28,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2415033.3333333335, ans=0.125
2023-11-23 14:23:35,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2415033.3333333335, ans=0.1
2023-11-23 14:23:38,726 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1550, loss[loss=0.08477, simple_loss=0.1042, pruned_loss=0.02096, audio_tagging_loss=0.01169, over 14424.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09224, pruned_loss=0.01384, audio_tagging_loss=0.009099, over 3035700.23 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:23:44,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=13.56 vs. limit=22.5
2023-11-23 14:24:22,271 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362300
2023-11-23 14:24:26,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.25 vs. limit=15.0
2023-11-23 14:24:42,969 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1600, loss[loss=0.06474, simple_loss=0.08721, pruned_loss=0.009418, audio_tagging_loss=0.01171, over 14971.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09247, pruned_loss=0.01385, audio_tagging_loss=0.009264, over 3038692.49 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:24:44,514 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:24:48,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2415433.3333333335, ans=0.125
2023-11-23 14:24:53,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2415433.3333333335, ans=0.125
2023-11-23 14:25:05,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2415500.0, ans=0.0
2023-11-23 14:25:06,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2415500.0, ans=0.5
2023-11-23 14:25:14,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2415566.6666666665, ans=0.125
2023-11-23 14:25:15,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2415566.6666666665, ans=0.2
2023-11-23 14:25:22,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2415633.3333333335, ans=10.0
2023-11-23 14:25:26,564 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362350
2023-11-23 14:25:30,056 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.201e+01 8.953e+01 9.579e+01 1.488e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 14:25:44,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2415700.0, ans=0.0
2023-11-23 14:25:48,510 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1650, loss[loss=0.07114, simple_loss=0.1007, pruned_loss=0.01388, audio_tagging_loss=0.006902, over 14858.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09254, pruned_loss=0.01386, audio_tagging_loss=0.009349, over 3044410.74 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:26:31,322 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362400
2023-11-23 14:26:39,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2416033.3333333335, ans=0.125
2023-11-23 14:26:53,738 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1700, loss[loss=0.06488, simple_loss=0.08529, pruned_loss=0.01217, audio_tagging_loss=0.01007, over 16052.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09215, pruned_loss=0.01378, audio_tagging_loss=0.009354, over 3042385.26 frames. ], batch size: 62, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:26:55,671 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.72 vs. limit=12.0
2023-11-23 14:27:12,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2416166.6666666665, ans=0.125
2023-11-23 14:27:23,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2416233.3333333335, ans=0.0
2023-11-23 14:27:23,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.77 vs. limit=22.5
2023-11-23 14:27:31,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2416300.0, ans=0.125
2023-11-23 14:27:36,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362450
2023-11-23 14:27:36,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2416300.0, ans=0.1
2023-11-23 14:27:39,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2416300.0, ans=0.0
2023-11-23 14:27:40,064 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.211e+01 9.052e+01 9.581e+01 1.125e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 14:27:56,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2416433.3333333335, ans=0.025
2023-11-23 14:27:57,280 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1750, loss[loss=0.08255, simple_loss=0.1222, pruned_loss=0.01484, audio_tagging_loss=0.006596, over 14985.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09156, pruned_loss=0.01374, audio_tagging_loss=0.009276, over 3045343.99 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:28:24,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2416566.6666666665, ans=0.1
2023-11-23 14:28:27,433 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.79 vs. limit=22.5
2023-11-23 14:28:37,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2416633.3333333335, ans=0.05
2023-11-23 14:28:38,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2416633.3333333335, ans=0.125
2023-11-23 14:28:40,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362500
2023-11-23 14:28:56,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2416700.0, ans=0.1
2023-11-23 14:28:57,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0
2023-11-23 14:28:58,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.07 vs. limit=15.0
2023-11-23 14:29:02,040 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1800, loss[loss=0.05536, simple_loss=0.07833, pruned_loss=0.007209, audio_tagging_loss=0.008985, over 14832.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09178, pruned_loss=0.01375, audio_tagging_loss=0.009166, over 3038351.92 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:29:44,840 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362550
2023-11-23 14:29:48,998 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 8.278e+01 8.801e+01 9.362e+01 1.432e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-23 14:29:57,741 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.10 vs. limit=22.5
2023-11-23 14:29:59,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2417033.3333333335, ans=0.125
2023-11-23 14:30:02,009 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.43 vs. limit=15.0
2023-11-23 14:30:07,997 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1850, loss[loss=0.06849, simple_loss=0.09726, pruned_loss=0.01409, audio_tagging_loss=0.005777, over 15294.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09162, pruned_loss=0.01374, audio_tagging_loss=0.009157, over 3038263.65 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:30:35,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2417233.3333333335, ans=0.0
2023-11-23 14:30:50,691 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362600
2023-11-23 14:30:53,745 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:30:54,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2417300.0, ans=0.035
2023-11-23 14:31:12,189 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1900, loss[loss=0.08731, simple_loss=0.1209, pruned_loss=0.01982, audio_tagging_loss=0.007049, over 15473.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09195, pruned_loss=0.0138, audio_tagging_loss=0.009031, over 3044184.57 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:31:54,087 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.67 vs. limit=15.0
2023-11-23 14:31:54,855 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362650
2023-11-23 14:31:58,352 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.275e+01 8.858e+01 9.597e+01 1.368e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 14:32:04,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.50 vs. limit=22.5
2023-11-23 14:32:16,148 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 1950, loss[loss=0.07208, simple_loss=0.09737, pruned_loss=0.01469, audio_tagging_loss=0.008701, over 16414.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09158, pruned_loss=0.01369, audio_tagging_loss=0.009072, over 3045070.33 frames. ], batch size: 61, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:32:25,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2417766.6666666665, ans=0.025
2023-11-23 14:32:32,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2417833.3333333335, ans=0.125
2023-11-23 14:32:58,336 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362700
2023-11-23 14:33:09,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2418033.3333333335, ans=0.1
2023-11-23 14:33:13,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.43 vs. limit=15.0
2023-11-23 14:33:20,867 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2000, loss[loss=0.08429, simple_loss=0.1049, pruned_loss=0.02125, audio_tagging_loss=0.01057, over 15935.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09185, pruned_loss=0.01386, audio_tagging_loss=0.009047, over 3047954.44 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:33:23,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2418100.0, ans=0.125
2023-11-23 14:33:40,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2418166.6666666665, ans=22.5
2023-11-23 14:33:41,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2418166.6666666665, ans=0.125
2023-11-23 14:33:43,521 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.97 vs. limit=22.5
2023-11-23 14:33:49,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2418233.3333333335, ans=0.0
2023-11-23 14:33:53,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2418233.3333333335, ans=0.125
2023-11-23 14:34:01,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2418300.0, ans=0.125
2023-11-23 14:34:03,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362750
2023-11-23 14:34:10,256 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.308e+01 8.872e+01 9.432e+01 1.250e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 14:34:11,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2418366.6666666665, ans=0.125
2023-11-23 14:34:26,001 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2050, loss[loss=0.07977, simple_loss=0.09999, pruned_loss=0.01953, audio_tagging_loss=0.01025, over 15138.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09201, pruned_loss=0.014, audio_tagging_loss=0.009053, over 3043765.67 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:34:27,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.09 vs. limit=22.5
2023-11-23 14:34:35,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2418433.3333333335, ans=0.125
2023-11-23 14:34:40,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2418500.0, ans=0.125
2023-11-23 14:34:50,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2418500.0, ans=0.125
2023-11-23 14:34:53,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2418566.6666666665, ans=0.125
2023-11-23 14:34:56,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2418566.6666666665, ans=0.125
2023-11-23 14:34:58,238 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.88 vs. limit=15.0
2023-11-23 14:35:09,525 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362800
2023-11-23 14:35:26,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2418700.0, ans=0.1
2023-11-23 14:35:31,106 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2100, loss[loss=0.06645, simple_loss=0.08939, pruned_loss=0.0116, audio_tagging_loss=0.01016, over 14646.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09225, pruned_loss=0.01399, audio_tagging_loss=0.008983, over 3039727.90 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:35:49,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2418833.3333333335, ans=0.125
2023-11-23 14:35:54,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2418833.3333333335, ans=0.1
2023-11-23 14:35:57,078 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.01 vs. limit=15.0
2023-11-23 14:36:04,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2418900.0, ans=0.0
2023-11-23 14:36:08,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2418900.0, ans=0.0
2023-11-23 14:36:11,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2418966.6666666665, ans=0.125
2023-11-23 14:36:14,734 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362850
2023-11-23 14:36:18,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2418966.6666666665, ans=0.125
2023-11-23 14:36:20,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.76 vs. limit=22.5
2023-11-23 14:36:20,835 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 8.402e+01 8.985e+01 9.676e+01 1.115e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 14:36:22,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2419033.3333333335, ans=0.0
2023-11-23 14:36:37,086 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.20 vs. limit=10.0
2023-11-23 14:36:37,685 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2150, loss[loss=0.0623, simple_loss=0.0811, pruned_loss=0.01067, audio_tagging_loss=0.01108, over 14615.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09305, pruned_loss=0.01423, audio_tagging_loss=0.008943, over 3039822.43 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:36:42,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2419100.0, ans=0.2
2023-11-23 14:36:48,133 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.89 vs. limit=15.0
2023-11-23 14:36:50,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2419166.6666666665, ans=0.0
2023-11-23 14:36:57,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2419166.6666666665, ans=0.2
2023-11-23 14:37:14,946 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:37:20,589 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362900
2023-11-23 14:37:42,021 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2200, loss[loss=0.06844, simple_loss=0.08664, pruned_loss=0.01492, audio_tagging_loss=0.0102, over 15020.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09309, pruned_loss=0.01417, audio_tagging_loss=0.008961, over 3045820.35 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:37:56,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2419500.0, ans=10.0
2023-11-23 14:37:57,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2419500.0, ans=0.1
2023-11-23 14:38:23,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2419633.3333333335, ans=0.125
2023-11-23 14:38:25,474 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 362950
2023-11-23 14:38:31,612 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.445e+01 9.122e+01 9.777e+01 1.344e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 14:38:41,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2419700.0, ans=0.125
2023-11-23 14:38:47,357 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2250, loss[loss=0.05455, simple_loss=0.06224, pruned_loss=0.01066, audio_tagging_loss=0.01276, over 14246.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09297, pruned_loss=0.01402, audio_tagging_loss=0.008947, over 3042520.68 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:38:50,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.72 vs. limit=15.0
2023-11-23 14:39:02,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2419833.3333333335, ans=0.125
2023-11-23 14:39:07,434 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.51 vs. limit=6.0
2023-11-23 14:39:10,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2419833.3333333335, ans=0.0
2023-11-23 14:39:10,728 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:39:19,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2419900.0, ans=0.125
2023-11-23 14:39:31,106 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363000
2023-11-23 14:39:54,461 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2300, loss[loss=0.06741, simple_loss=0.08885, pruned_loss=0.01546, audio_tagging_loss=0.007526, over 15157.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09254, pruned_loss=0.014, audio_tagging_loss=0.008986, over 3042929.27 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:40:06,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2420166.6666666665, ans=0.125
2023-11-23 14:40:10,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2420166.6666666665, ans=0.125
2023-11-23 14:40:37,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363050
2023-11-23 14:40:43,594 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.427e+01 9.145e+01 9.809e+01 1.490e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 14:40:46,889 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.64 vs. limit=10.0
2023-11-23 14:40:49,896 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:40:58,424 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2350, loss[loss=0.06256, simple_loss=0.07795, pruned_loss=0.01306, audio_tagging_loss=0.01053, over 14533.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.0925, pruned_loss=0.01388, audio_tagging_loss=0.009103, over 3048273.37 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:41:07,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2420433.3333333335, ans=0.125
2023-11-23 14:41:14,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2420500.0, ans=0.0
2023-11-23 14:41:22,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2420500.0, ans=0.0
2023-11-23 14:41:30,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2420566.6666666665, ans=0.1
2023-11-23 14:41:30,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.65 vs. limit=15.0
2023-11-23 14:41:39,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2420633.3333333335, ans=0.0
2023-11-23 14:41:41,575 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363100
2023-11-23 14:41:51,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2420700.0, ans=0.125
2023-11-23 14:41:52,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2420700.0, ans=0.2
2023-11-23 14:42:02,802 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2400, loss[loss=0.06907, simple_loss=0.08827, pruned_loss=0.01552, audio_tagging_loss=0.009418, over 15245.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09255, pruned_loss=0.01392, audio_tagging_loss=0.009205, over 3050488.53 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:42:05,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2420766.6666666665, ans=0.125
2023-11-23 14:42:12,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2420766.6666666665, ans=0.1
2023-11-23 14:42:21,003 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.55 vs. limit=5.0
2023-11-23 14:42:29,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2420900.0, ans=0.125
2023-11-23 14:42:33,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2420900.0, ans=0.2
2023-11-23 14:42:33,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2420900.0, ans=0.2
2023-11-23 14:42:33,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2420900.0, ans=0.0
2023-11-23 14:42:36,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2420900.0, ans=0.1
2023-11-23 14:42:42,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2420966.6666666665, ans=0.125
2023-11-23 14:42:45,466 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363150
2023-11-23 14:42:51,916 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.436e+01 9.019e+01 9.718e+01 1.173e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 14:43:07,833 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2450, loss[loss=0.06279, simple_loss=0.07327, pruned_loss=0.01416, audio_tagging_loss=0.01199, over 14961.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.092, pruned_loss=0.01382, audio_tagging_loss=0.009213, over 3053278.56 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:43:16,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2421100.0, ans=0.0
2023-11-23 14:43:24,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2421166.6666666665, ans=0.0
2023-11-23 14:43:50,390 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363200
2023-11-23 14:44:05,973 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.63 vs. limit=15.0
2023-11-23 14:44:11,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2421433.3333333335, ans=0.125
2023-11-23 14:44:12,635 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2500, loss[loss=0.05965, simple_loss=0.07767, pruned_loss=0.01327, audio_tagging_loss=0.007548, over 14031.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09191, pruned_loss=0.01379, audio_tagging_loss=0.009229, over 3051715.26 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:44:14,053 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:44:16,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2421433.3333333335, ans=0.125
2023-11-23 14:44:22,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2421433.3333333335, ans=0.5
2023-11-23 14:44:26,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2421500.0, ans=0.125
2023-11-23 14:44:55,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2421633.3333333335, ans=0.0
2023-11-23 14:44:56,114 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363250
2023-11-23 14:45:01,988 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.211e+01 9.083e+01 9.843e+01 1.373e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 14:45:17,037 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2550, loss[loss=0.08305, simple_loss=0.1117, pruned_loss=0.01836, audio_tagging_loss=0.008818, over 15058.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09155, pruned_loss=0.01374, audio_tagging_loss=0.009215, over 3045897.65 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:45:53,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2421900.0, ans=0.1
2023-11-23 14:45:56,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2421966.6666666665, ans=0.125
2023-11-23 14:46:00,376 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363300
2023-11-23 14:46:05,741 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.87 vs. limit=15.0
2023-11-23 14:46:22,961 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2600, loss[loss=0.07283, simple_loss=0.1061, pruned_loss=0.01305, audio_tagging_loss=0.006738, over 16205.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09096, pruned_loss=0.01368, audio_tagging_loss=0.009099, over 3045860.85 frames. ], batch size: 61, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:46:24,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2422100.0, ans=0.0
2023-11-23 14:46:30,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2422100.0, ans=0.125
2023-11-23 14:46:30,857 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.68 vs. limit=15.0
2023-11-23 14:46:44,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2422166.6666666665, ans=0.0
2023-11-23 14:47:05,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363350
2023-11-23 14:47:12,904 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.370e+01 9.064e+01 9.837e+01 1.240e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 14:47:20,426 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.29 vs. limit=15.0
2023-11-23 14:47:28,332 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2650, loss[loss=0.0608, simple_loss=0.07671, pruned_loss=0.01195, audio_tagging_loss=0.0105, over 15082.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09082, pruned_loss=0.01384, audio_tagging_loss=0.009065, over 3042039.10 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:47:34,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2422433.3333333335, ans=0.2
2023-11-23 14:47:35,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2422433.3333333335, ans=0.125
2023-11-23 14:47:42,328 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:47:51,434 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.62 vs. limit=5.0
2023-11-23 14:47:53,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.00 vs. limit=15.0
2023-11-23 14:48:04,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2422566.6666666665, ans=0.125
2023-11-23 14:48:11,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363400
2023-11-23 14:48:32,429 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2700, loss[loss=0.0602, simple_loss=0.08269, pruned_loss=0.01081, audio_tagging_loss=0.00804, over 14769.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09118, pruned_loss=0.01396, audio_tagging_loss=0.009026, over 3047532.61 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:48:39,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2422766.6666666665, ans=0.125
2023-11-23 14:48:41,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2422766.6666666665, ans=0.1
2023-11-23 14:49:15,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363450
2023-11-23 14:49:23,351 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.399e+01 8.847e+01 9.605e+01 1.237e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 14:49:34,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2423033.3333333335, ans=0.125
2023-11-23 14:49:37,796 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2750, loss[loss=0.05932, simple_loss=0.06855, pruned_loss=0.009751, audio_tagging_loss=0.0153, over 15524.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09051, pruned_loss=0.01397, audio_tagging_loss=0.0091, over 3046595.83 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:49:57,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2423166.6666666665, ans=0.125
2023-11-23 14:50:13,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2423233.3333333335, ans=0.125
2023-11-23 14:50:20,837 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363500
2023-11-23 14:50:33,606 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:50:43,973 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2800, loss[loss=0.05611, simple_loss=0.07704, pruned_loss=0.00962, audio_tagging_loss=0.007969, over 14980.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.0909, pruned_loss=0.01388, audio_tagging_loss=0.009088, over 3040471.88 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:50:55,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2423500.0, ans=0.125
2023-11-23 14:51:00,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2423500.0, ans=0.125
2023-11-23 14:51:27,444 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363550
2023-11-23 14:51:32,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2423633.3333333335, ans=0.125
2023-11-23 14:51:36,068 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.140e+01 8.811e+01 9.422e+01 1.418e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 14:51:48,473 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2850, loss[loss=0.09244, simple_loss=0.1249, pruned_loss=0.02123, audio_tagging_loss=0.008764, over 15763.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09067, pruned_loss=0.01382, audio_tagging_loss=0.009063, over 3032587.75 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:52:05,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.81 vs. limit=15.0
2023-11-23 14:52:22,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.98 vs. limit=6.0
2023-11-23 14:52:31,479 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363600
2023-11-23 14:52:41,715 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:52:41,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2424033.3333333335, ans=0.125
2023-11-23 14:52:44,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2424033.3333333335, ans=0.125
2023-11-23 14:52:45,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2424033.3333333335, ans=0.0
2023-11-23 14:52:52,480 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2900, loss[loss=0.06973, simple_loss=0.08895, pruned_loss=0.01484, audio_tagging_loss=0.01041, over 15029.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09068, pruned_loss=0.0139, audio_tagging_loss=0.009046, over 3038411.58 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:53:11,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2424166.6666666665, ans=0.1
2023-11-23 14:53:35,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363650
2023-11-23 14:53:44,994 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.203e+01 8.514e+01 9.320e+01 9.888e+01 1.457e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-23 14:53:59,189 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 2950, loss[loss=0.07035, simple_loss=0.08977, pruned_loss=0.01678, audio_tagging_loss=0.00868, over 14546.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09105, pruned_loss=0.01387, audio_tagging_loss=0.009143, over 3034424.29 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:54:09,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2424433.3333333335, ans=0.1
2023-11-23 14:54:16,081 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.65 vs. limit=15.0
2023-11-23 14:54:22,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2424566.6666666665, ans=0.0
2023-11-23 14:54:35,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2424633.3333333335, ans=0.1
2023-11-23 14:54:36,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2424633.3333333335, ans=0.1
2023-11-23 14:54:41,424 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363700
2023-11-23 14:55:03,646 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3000, loss[loss=0.07442, simple_loss=0.09872, pruned_loss=0.01494, audio_tagging_loss=0.01011, over 15060.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09201, pruned_loss=0.01389, audio_tagging_loss=0.009152, over 3039238.33 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:55:03,647 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 14:55:45,748 INFO [train_asr.py:1253] (2/4) Epoch 31, validation: loss=0.0577, simple_loss=0.05103, pruned_loss=0.005016, audio_tagging_loss=0.02717, over 4681554.00 frames. 
2023-11-23 14:55:45,749 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 14:56:28,212 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363750
2023-11-23 14:56:30,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2424966.6666666665, ans=0.125
2023-11-23 14:56:31,121 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.51 vs. limit=15.0
2023-11-23 14:56:37,300 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.570e+01 9.368e+01 9.997e+01 1.561e+02, threshold=1.874e+02, percent-clipped=0.0
2023-11-23 14:56:50,885 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3050, loss[loss=0.07979, simple_loss=0.1122, pruned_loss=0.01561, audio_tagging_loss=0.008087, over 15799.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09271, pruned_loss=0.01401, audio_tagging_loss=0.009249, over 3040758.37 frames. ], batch size: 60, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:56:51,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2425100.0, ans=0.125
2023-11-23 14:57:16,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2425233.3333333335, ans=0.2
2023-11-23 14:57:16,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2425233.3333333335, ans=0.125
2023-11-23 14:57:20,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.38 vs. limit=12.0
2023-11-23 14:57:26,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.76 vs. limit=6.0
2023-11-23 14:57:28,683 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:57:34,107 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363800
2023-11-23 14:57:37,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2425300.0, ans=0.1
2023-11-23 14:57:40,319 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.05 vs. limit=10.0
2023-11-23 14:57:54,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=7.73 vs. limit=15.0
2023-11-23 14:57:56,138 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3100, loss[loss=0.07351, simple_loss=0.1047, pruned_loss=0.0153, audio_tagging_loss=0.005836, over 14917.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09237, pruned_loss=0.014, audio_tagging_loss=0.009324, over 3034168.36 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:58:00,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2425433.3333333335, ans=0.2
2023-11-23 14:58:20,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.42 vs. limit=15.0
2023-11-23 14:58:21,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2425566.6666666665, ans=0.0
2023-11-23 14:58:39,613 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363850
2023-11-23 14:58:46,368 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.16 vs. limit=15.0
2023-11-23 14:58:47,918 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.127e+01 8.546e+01 8.908e+01 9.738e+01 1.357e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 14:58:50,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2425700.0, ans=0.2
2023-11-23 14:58:56,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2425700.0, ans=0.0
2023-11-23 14:59:00,936 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3150, loss[loss=0.08271, simple_loss=0.1128, pruned_loss=0.01696, audio_tagging_loss=0.009361, over 15298.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09273, pruned_loss=0.01396, audio_tagging_loss=0.009268, over 3037553.73 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:59:17,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2425833.3333333335, ans=0.0
2023-11-23 14:59:32,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2425900.0, ans=0.125
2023-11-23 14:59:43,589 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363900
2023-11-23 14:59:56,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2426033.3333333335, ans=0.0
2023-11-23 15:00:06,087 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3200, loss[loss=0.06374, simple_loss=0.0799, pruned_loss=0.01158, audio_tagging_loss=0.01221, over 15161.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.0934, pruned_loss=0.01407, audio_tagging_loss=0.00927, over 3041867.64 frames. ], batch size: 60, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:00:17,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2426166.6666666665, ans=0.2
2023-11-23 15:00:29,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2426166.6666666665, ans=0.1
2023-11-23 15:00:31,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2426233.3333333335, ans=0.0
2023-11-23 15:00:31,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2426233.3333333335, ans=0.0
2023-11-23 15:00:33,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2426233.3333333335, ans=0.125
2023-11-23 15:00:41,076 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:00:49,081 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 363950
2023-11-23 15:00:54,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2426300.0, ans=0.0
2023-11-23 15:00:58,123 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.380e+01 9.015e+01 9.537e+01 1.251e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 15:01:11,015 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3250, loss[loss=0.06296, simple_loss=0.0727, pruned_loss=0.01033, audio_tagging_loss=0.01628, over 15111.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09328, pruned_loss=0.014, audio_tagging_loss=0.009404, over 3043524.30 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:01:16,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2426433.3333333335, ans=0.125
2023-11-23 15:01:19,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.52 vs. limit=10.0
2023-11-23 15:01:27,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2426500.0, ans=0.0
2023-11-23 15:01:35,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2426566.6666666665, ans=0.1
2023-11-23 15:01:51,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2426633.3333333335, ans=0.1
2023-11-23 15:01:53,572 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364000
2023-11-23 15:02:05,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2426700.0, ans=0.125
2023-11-23 15:02:18,601 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3300, loss[loss=0.04664, simple_loss=0.05759, pruned_loss=0.007082, audio_tagging_loss=0.01077, over 14946.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09295, pruned_loss=0.01401, audio_tagging_loss=0.009385, over 3053670.10 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:02:39,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2426833.3333333335, ans=0.1
2023-11-23 15:02:43,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2426900.0, ans=0.125
2023-11-23 15:02:44,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2426900.0, ans=0.125
2023-11-23 15:02:47,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2426900.0, ans=0.04949747468305833
2023-11-23 15:02:57,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2426966.6666666665, ans=0.04949747468305833
2023-11-23 15:03:01,209 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364050
2023-11-23 15:03:09,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.98 vs. limit=15.0
2023-11-23 15:03:10,311 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.350e+01 9.048e+01 9.771e+01 1.182e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 15:03:23,925 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3350, loss[loss=0.0607, simple_loss=0.07471, pruned_loss=0.0124, audio_tagging_loss=0.01095, over 14484.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09249, pruned_loss=0.01382, audio_tagging_loss=0.009272, over 3053432.50 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:03:25,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2427100.0, ans=0.125
2023-11-23 15:03:29,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2427100.0, ans=0.125
2023-11-23 15:03:39,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.69 vs. limit=15.0
2023-11-23 15:04:06,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364100
2023-11-23 15:04:19,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2427366.6666666665, ans=0.125
2023-11-23 15:04:28,236 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3400, loss[loss=0.07923, simple_loss=0.1048, pruned_loss=0.01892, audio_tagging_loss=0.007917, over 15073.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09301, pruned_loss=0.01395, audio_tagging_loss=0.009125, over 3059546.56 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:04:41,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2427500.0, ans=0.1
2023-11-23 15:05:07,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2427633.3333333335, ans=0.0
2023-11-23 15:05:11,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364150
2023-11-23 15:05:21,247 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.657e+01 9.005e+01 9.760e+01 1.355e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 15:05:21,667 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:05:22,151 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.55 vs. limit=6.0
2023-11-23 15:05:29,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.46 vs. limit=6.0
2023-11-23 15:05:30,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2427700.0, ans=0.0
2023-11-23 15:05:32,847 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3450, loss[loss=0.06019, simple_loss=0.07444, pruned_loss=0.01452, audio_tagging_loss=0.00845, over 14709.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09286, pruned_loss=0.01391, audio_tagging_loss=0.009021, over 3056977.74 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:06:16,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364200
2023-11-23 15:06:25,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=2428033.3333333335, ans=0.2
2023-11-23 15:06:29,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2428033.3333333335, ans=0.1
2023-11-23 15:06:35,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2428033.3333333335, ans=0.1
2023-11-23 15:06:39,840 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3500, loss[loss=0.09331, simple_loss=0.1295, pruned_loss=0.02139, audio_tagging_loss=0.007188, over 16130.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09261, pruned_loss=0.01388, audio_tagging_loss=0.008978, over 3048985.48 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:07:11,427 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:07:21,990 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364250
2023-11-23 15:07:22,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2428300.0, ans=0.1
2023-11-23 15:07:32,823 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.333e+01 8.735e+01 9.330e+01 1.206e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-23 15:07:43,888 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3550, loss[loss=0.0692, simple_loss=0.09641, pruned_loss=0.00958, audio_tagging_loss=0.01142, over 15521.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09218, pruned_loss=0.01371, audio_tagging_loss=0.008923, over 3048116.23 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:07:51,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2428433.3333333335, ans=0.1
2023-11-23 15:07:59,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2428500.0, ans=0.0
2023-11-23 15:08:09,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2428566.6666666665, ans=15.0
2023-11-23 15:08:10,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2428566.6666666665, ans=0.0
2023-11-23 15:08:11,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2428566.6666666665, ans=0.0
2023-11-23 15:08:12,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2428566.6666666665, ans=0.125
2023-11-23 15:08:14,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2428566.6666666665, ans=0.0
2023-11-23 15:08:27,278 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364300
2023-11-23 15:08:36,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2428700.0, ans=0.125
2023-11-23 15:08:49,124 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3600, loss[loss=0.04615, simple_loss=0.05847, pruned_loss=0.007703, audio_tagging_loss=0.009209, over 16150.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09077, pruned_loss=0.01345, audio_tagging_loss=0.008974, over 3047102.45 frames. ], batch size: 62, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:08:54,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2428766.6666666665, ans=0.0
2023-11-23 15:08:54,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2428766.6666666665, ans=0.1
2023-11-23 15:09:05,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.01 vs. limit=15.0
2023-11-23 15:09:16,132 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.31 vs. limit=22.5
2023-11-23 15:09:22,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.20 vs. limit=15.0
2023-11-23 15:09:30,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2428966.6666666665, ans=0.125
2023-11-23 15:09:31,798 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364350
2023-11-23 15:09:42,098 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.038e+01 8.312e+01 8.847e+01 9.562e+01 1.157e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 15:09:54,400 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3650, loss[loss=0.06567, simple_loss=0.08973, pruned_loss=0.01308, audio_tagging_loss=0.007725, over 15219.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.0912, pruned_loss=0.01372, audio_tagging_loss=0.009006, over 3046095.01 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:09:55,192 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.31 vs. limit=10.0
2023-11-23 15:10:36,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364400
2023-11-23 15:10:49,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2429366.6666666665, ans=0.125
2023-11-23 15:10:53,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2429366.6666666665, ans=0.0
2023-11-23 15:10:58,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2429433.3333333335, ans=0.125
2023-11-23 15:10:59,373 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3700, loss[loss=0.09136, simple_loss=0.126, pruned_loss=0.02266, audio_tagging_loss=0.005678, over 15881.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09211, pruned_loss=0.01383, audio_tagging_loss=0.008914, over 3051717.79 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:11:16,226 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.66 vs. limit=15.0
2023-11-23 15:11:19,835 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.64 vs. limit=15.0
2023-11-23 15:11:28,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2429566.6666666665, ans=0.2
2023-11-23 15:11:35,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2429566.6666666665, ans=0.07
2023-11-23 15:11:42,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364450
2023-11-23 15:11:52,584 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.697e+01 8.537e+01 9.041e+01 9.821e+01 1.313e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-23 15:12:03,712 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3750, loss[loss=0.08867, simple_loss=0.1164, pruned_loss=0.02231, audio_tagging_loss=0.008173, over 14278.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09252, pruned_loss=0.01408, audio_tagging_loss=0.008926, over 3056996.99 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:12:10,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2429766.6666666665, ans=0.125
2023-11-23 15:12:17,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2429833.3333333335, ans=0.125
2023-11-23 15:12:26,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.94 vs. limit=15.0
2023-11-23 15:12:31,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2429900.0, ans=0.125
2023-11-23 15:12:34,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.18 vs. limit=22.5
2023-11-23 15:12:37,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.96 vs. limit=15.0
2023-11-23 15:12:43,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2429966.6666666665, ans=0.125
2023-11-23 15:12:46,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364500
2023-11-23 15:12:46,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.38 vs. limit=15.0
2023-11-23 15:12:47,572 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:12:59,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.12 vs. limit=15.0
2023-11-23 15:13:08,472 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3800, loss[loss=0.04547, simple_loss=0.0573, pruned_loss=0.007276, audio_tagging_loss=0.009546, over 13813.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09175, pruned_loss=0.01393, audio_tagging_loss=0.008969, over 3051515.73 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:13:22,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2430166.6666666665, ans=0.125
2023-11-23 15:13:29,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2430166.6666666665, ans=0.125
2023-11-23 15:13:32,740 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.07 vs. limit=6.0
2023-11-23 15:13:37,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0
2023-11-23 15:13:44,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2430233.3333333335, ans=0.1
2023-11-23 15:13:46,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2430300.0, ans=0.2
2023-11-23 15:13:50,889 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364550
2023-11-23 15:13:50,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2430300.0, ans=10.0
2023-11-23 15:14:04,561 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.445e+01 8.986e+01 9.684e+01 1.334e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 15:14:11,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2430366.6666666665, ans=0.0
2023-11-23 15:14:14,489 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3850, loss[loss=0.06328, simple_loss=0.08341, pruned_loss=0.01156, audio_tagging_loss=0.01001, over 15890.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09297, pruned_loss=0.01412, audio_tagging_loss=0.008956, over 3055830.26 frames. ], batch size: 60, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:14:16,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2430433.3333333335, ans=10.0
2023-11-23 15:14:22,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2430433.3333333335, ans=0.05
2023-11-23 15:14:37,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2430500.0, ans=0.125
2023-11-23 15:14:56,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2430633.3333333335, ans=0.125
2023-11-23 15:14:57,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364600
2023-11-23 15:14:58,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2430633.3333333335, ans=0.125
2023-11-23 15:15:02,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2430633.3333333335, ans=0.0
2023-11-23 15:15:08,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.06 vs. limit=22.5
2023-11-23 15:15:18,987 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3900, loss[loss=0.06134, simple_loss=0.08361, pruned_loss=0.01082, audio_tagging_loss=0.008716, over 15227.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09212, pruned_loss=0.01415, audio_tagging_loss=0.008992, over 3045567.11 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:15:45,557 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.08 vs. limit=22.5
2023-11-23 15:15:46,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2430900.0, ans=0.1
2023-11-23 15:15:48,027 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.63 vs. limit=15.0
2023-11-23 15:15:52,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2430900.0, ans=0.035
2023-11-23 15:16:02,388 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364650
2023-11-23 15:16:09,410 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.22 vs. limit=5.0
2023-11-23 15:16:14,753 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.954e+01 8.382e+01 9.163e+01 9.741e+01 1.346e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 15:16:23,959 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 3950, loss[loss=0.0617, simple_loss=0.07634, pruned_loss=0.01212, audio_tagging_loss=0.01141, over 15792.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09296, pruned_loss=0.01418, audio_tagging_loss=0.009103, over 3042330.31 frames. ], batch size: 61, lr: 2.20e-03, grad_scale: 8.0
2023-11-23 15:16:32,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2431100.0, ans=0.1
2023-11-23 15:16:59,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2431233.3333333335, ans=0.0
2023-11-23 15:16:59,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2431233.3333333335, ans=0.125
2023-11-23 15:17:06,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364700
2023-11-23 15:17:29,994 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4000, loss[loss=0.06499, simple_loss=0.09348, pruned_loss=0.01132, audio_tagging_loss=0.006933, over 15645.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.0925, pruned_loss=0.01417, audio_tagging_loss=0.009206, over 3033216.97 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:18:04,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.11 vs. limit=6.0
2023-11-23 15:18:12,973 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364750
2023-11-23 15:18:17,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2431633.3333333335, ans=0.125
2023-11-23 15:18:25,120 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.345e+01 8.888e+01 9.598e+01 2.808e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-23 15:18:33,726 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4050, loss[loss=0.05475, simple_loss=0.06177, pruned_loss=0.01085, audio_tagging_loss=0.01301, over 14491.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09173, pruned_loss=0.01392, audio_tagging_loss=0.009314, over 3037664.64 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:18:36,195 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:18:37,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2431766.6666666665, ans=0.025
2023-11-23 15:18:38,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2431766.6666666665, ans=0.0
2023-11-23 15:18:40,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2431766.6666666665, ans=0.1
2023-11-23 15:18:45,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2431833.3333333335, ans=0.125
2023-11-23 15:19:16,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364800
2023-11-23 15:19:17,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2431966.6666666665, ans=0.2
2023-11-23 15:19:29,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2432033.3333333335, ans=0.025
2023-11-23 15:19:30,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2432033.3333333335, ans=0.125
2023-11-23 15:19:37,442 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4100, loss[loss=0.07237, simple_loss=0.0975, pruned_loss=0.01537, audio_tagging_loss=0.008248, over 15766.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09237, pruned_loss=0.01396, audio_tagging_loss=0.009222, over 3044380.65 frames. ], batch size: 62, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:20:06,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2432233.3333333335, ans=0.0
2023-11-23 15:20:19,942 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364850
2023-11-23 15:20:23,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2432300.0, ans=0.1
2023-11-23 15:20:28,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2432366.6666666665, ans=0.09899494936611666
2023-11-23 15:20:31,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2432366.6666666665, ans=0.1
2023-11-23 15:20:32,690 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.349e+01 8.595e+01 9.054e+01 9.904e+01 2.115e+02, threshold=1.811e+02, percent-clipped=1.0
2023-11-23 15:20:36,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.65 vs. limit=22.5
2023-11-23 15:20:43,285 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4150, loss[loss=0.0652, simple_loss=0.08708, pruned_loss=0.01392, audio_tagging_loss=0.00774, over 15491.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09197, pruned_loss=0.01396, audio_tagging_loss=0.009229, over 3042122.44 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:20:54,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2432500.0, ans=0.125
2023-11-23 15:21:01,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2432500.0, ans=0.2
2023-11-23 15:21:13,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2432566.6666666665, ans=0.0
2023-11-23 15:21:14,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2432566.6666666665, ans=0.0
2023-11-23 15:21:21,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2432633.3333333335, ans=15.0
2023-11-23 15:21:21,581 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.66 vs. limit=15.0
2023-11-23 15:21:25,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364900
2023-11-23 15:21:29,313 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:21:47,304 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4200, loss[loss=0.04887, simple_loss=0.06338, pruned_loss=0.009414, audio_tagging_loss=0.007766, over 16765.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09205, pruned_loss=0.01399, audio_tagging_loss=0.009111, over 3051594.22 frames. ], batch size: 64, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:22:30,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 364950
2023-11-23 15:22:42,430 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.314e+01 9.166e+01 9.730e+01 1.323e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 15:22:51,148 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4250, loss[loss=0.0791, simple_loss=0.116, pruned_loss=0.01406, audio_tagging_loss=0.007029, over 15463.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09184, pruned_loss=0.0138, audio_tagging_loss=0.009055, over 3052415.90 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:22:58,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2433100.0, ans=0.125
2023-11-23 15:23:10,689 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.16 vs. limit=15.0
2023-11-23 15:23:11,704 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.90 vs. limit=10.0
2023-11-23 15:23:15,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2433166.6666666665, ans=0.125
2023-11-23 15:23:19,707 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.31 vs. limit=15.0
2023-11-23 15:23:26,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2433233.3333333335, ans=0.0
2023-11-23 15:23:31,654 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.61 vs. limit=10.0
2023-11-23 15:23:33,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365000
2023-11-23 15:23:56,035 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4300, loss[loss=0.06805, simple_loss=0.09831, pruned_loss=0.00965, audio_tagging_loss=0.009238, over 15380.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.093, pruned_loss=0.01384, audio_tagging_loss=0.008963, over 3053410.65 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:23:57,882 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.19 vs. limit=22.5
2023-11-23 15:24:17,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2433500.0, ans=0.09899494936611666
2023-11-23 15:24:25,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2433566.6666666665, ans=0.0
2023-11-23 15:24:34,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2433633.3333333335, ans=0.125
2023-11-23 15:24:38,097 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365050
2023-11-23 15:24:38,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2433633.3333333335, ans=0.0
2023-11-23 15:24:50,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.539e+01 9.202e+01 9.861e+01 1.335e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 15:25:00,104 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4350, loss[loss=0.05517, simple_loss=0.07492, pruned_loss=0.009185, audio_tagging_loss=0.008525, over 15672.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.0935, pruned_loss=0.01403, audio_tagging_loss=0.008963, over 3050033.38 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:25:05,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2433766.6666666665, ans=0.2
2023-11-23 15:25:12,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2433833.3333333335, ans=0.125
2023-11-23 15:25:18,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2433833.3333333335, ans=0.125
2023-11-23 15:25:21,074 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.79 vs. limit=15.0
2023-11-23 15:25:27,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2433900.0, ans=0.0
2023-11-23 15:25:32,113 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.76 vs. limit=15.0
2023-11-23 15:25:42,407 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365100
2023-11-23 15:26:03,791 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4400, loss[loss=0.07266, simple_loss=0.09755, pruned_loss=0.01709, audio_tagging_loss=0.006794, over 13931.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.0926, pruned_loss=0.01377, audio_tagging_loss=0.00908, over 3043605.39 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:26:05,851 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.74 vs. limit=22.5
2023-11-23 15:26:20,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2434166.6666666665, ans=0.0
2023-11-23 15:26:46,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365150
2023-11-23 15:26:55,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2434366.6666666665, ans=0.125
2023-11-23 15:26:57,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2434366.6666666665, ans=0.1
2023-11-23 15:26:59,941 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.264e+01 9.016e+01 9.697e+01 1.171e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 15:27:02,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2434366.6666666665, ans=0.125
2023-11-23 15:27:03,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2434366.6666666665, ans=0.0
2023-11-23 15:27:09,451 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4450, loss[loss=0.0579, simple_loss=0.07655, pruned_loss=0.009588, audio_tagging_loss=0.01004, over 14626.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09308, pruned_loss=0.01365, audio_tagging_loss=0.008981, over 3044205.30 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:27:43,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.22 vs. limit=22.5
2023-11-23 15:27:44,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2434566.6666666665, ans=0.125
2023-11-23 15:27:48,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2434633.3333333335, ans=0.0
2023-11-23 15:27:51,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365200
2023-11-23 15:28:14,259 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4500, loss[loss=0.05375, simple_loss=0.07038, pruned_loss=0.007518, audio_tagging_loss=0.01104, over 14680.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09307, pruned_loss=0.01366, audio_tagging_loss=0.008921, over 3039361.90 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:28:16,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2434766.6666666665, ans=0.125
2023-11-23 15:28:30,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.34 vs. limit=22.5
2023-11-23 15:28:51,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2434900.0, ans=0.125
2023-11-23 15:28:57,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365250
2023-11-23 15:29:04,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2434966.6666666665, ans=0.0
2023-11-23 15:29:07,862 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:29:09,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2435033.3333333335, ans=0.125
2023-11-23 15:29:11,956 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.432e+01 8.316e+01 8.938e+01 9.967e+01 1.364e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 15:29:19,397 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4550, loss[loss=0.04998, simple_loss=0.06931, pruned_loss=0.009042, audio_tagging_loss=0.006287, over 16435.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09259, pruned_loss=0.01372, audio_tagging_loss=0.008962, over 3039490.28 frames. ], batch size: 62, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:29:40,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2435166.6666666665, ans=0.125
2023-11-23 15:30:03,110 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365300
2023-11-23 15:30:08,620 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.81 vs. limit=22.5
2023-11-23 15:30:09,131 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:30:15,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2435366.6666666665, ans=0.125
2023-11-23 15:30:15,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2435366.6666666665, ans=0.2
2023-11-23 15:30:24,792 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4600, loss[loss=0.06893, simple_loss=0.09248, pruned_loss=0.01331, audio_tagging_loss=0.009378, over 15382.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09221, pruned_loss=0.01373, audio_tagging_loss=0.009055, over 3035513.37 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:30:32,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2435433.3333333335, ans=0.125
2023-11-23 15:30:44,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2435500.0, ans=0.125
2023-11-23 15:31:04,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2435633.3333333335, ans=0.125
2023-11-23 15:31:08,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365350
2023-11-23 15:31:08,664 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.27 vs. limit=12.0
2023-11-23 15:31:22,439 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.583e+01 9.120e+01 9.726e+01 1.437e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 15:31:24,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2435700.0, ans=0.025
2023-11-23 15:31:29,875 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4650, loss[loss=0.09354, simple_loss=0.1248, pruned_loss=0.022, audio_tagging_loss=0.009157, over 15393.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09228, pruned_loss=0.01385, audio_tagging_loss=0.009171, over 3032589.93 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:31:58,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2435900.0, ans=0.125
2023-11-23 15:32:03,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2435900.0, ans=0.0
2023-11-23 15:32:12,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365400
2023-11-23 15:32:33,930 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4700, loss[loss=0.07963, simple_loss=0.1038, pruned_loss=0.01808, audio_tagging_loss=0.009661, over 14549.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09224, pruned_loss=0.01378, audio_tagging_loss=0.009194, over 3030430.89 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:32:49,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2436166.6666666665, ans=0.125
2023-11-23 15:32:51,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2436166.6666666665, ans=0.2
2023-11-23 15:32:52,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2436166.6666666665, ans=0.125
2023-11-23 15:32:58,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2436166.6666666665, ans=0.125
2023-11-23 15:33:06,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2436233.3333333335, ans=0.125
2023-11-23 15:33:17,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365450
2023-11-23 15:33:19,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2436300.0, ans=0.1
2023-11-23 15:33:31,144 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.245e+01 8.834e+01 9.507e+01 1.184e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 15:33:39,241 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4750, loss[loss=0.06829, simple_loss=0.08564, pruned_loss=0.01554, audio_tagging_loss=0.009929, over 13541.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09126, pruned_loss=0.01369, audio_tagging_loss=0.009356, over 3033794.78 frames. ], batch size: 52, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:33:48,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2436433.3333333335, ans=0.125
2023-11-23 15:33:57,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.30 vs. limit=22.5
2023-11-23 15:34:22,114 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365500
2023-11-23 15:34:38,046 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.65 vs. limit=15.0
2023-11-23 15:34:42,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2436700.0, ans=0.2
2023-11-23 15:34:44,603 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4800, loss[loss=0.06318, simple_loss=0.08395, pruned_loss=0.01145, audio_tagging_loss=0.009753, over 14288.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09177, pruned_loss=0.01375, audio_tagging_loss=0.009431, over 3038219.63 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:34:47,599 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.71 vs. limit=15.0
2023-11-23 15:34:58,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2436833.3333333335, ans=0.125
2023-11-23 15:34:58,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2436833.3333333335, ans=0.0
2023-11-23 15:35:02,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.16 vs. limit=22.5
2023-11-23 15:35:15,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.83 vs. limit=22.5
2023-11-23 15:35:27,548 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365550
2023-11-23 15:35:35,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2437033.3333333335, ans=0.1
2023-11-23 15:35:42,119 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.333e+01 8.912e+01 9.620e+01 1.211e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 15:35:48,314 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4850, loss[loss=0.07196, simple_loss=0.09633, pruned_loss=0.01338, audio_tagging_loss=0.01041, over 15285.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09157, pruned_loss=0.01369, audio_tagging_loss=0.009505, over 3037530.24 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:35:48,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2437100.0, ans=0.1
2023-11-23 15:36:30,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2437300.0, ans=0.125
2023-11-23 15:36:31,603 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365600
2023-11-23 15:36:53,496 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4900, loss[loss=0.06287, simple_loss=0.07831, pruned_loss=0.01253, audio_tagging_loss=0.01118, over 13738.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09184, pruned_loss=0.01369, audio_tagging_loss=0.009385, over 3042685.63 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:36:59,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.61 vs. limit=22.5
2023-11-23 15:37:08,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2437500.0, ans=0.0
2023-11-23 15:37:14,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2437500.0, ans=0.05
2023-11-23 15:37:14,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2437500.0, ans=10.0
2023-11-23 15:37:15,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2437500.0, ans=0.2
2023-11-23 15:37:35,639 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365650
2023-11-23 15:37:53,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.240e+01 8.743e+01 9.594e+01 1.319e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 15:37:58,803 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 4950, loss[loss=0.05972, simple_loss=0.07293, pruned_loss=0.01196, audio_tagging_loss=0.0113, over 15528.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09173, pruned_loss=0.01368, audio_tagging_loss=0.009272, over 3045419.97 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:38:18,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2437833.3333333335, ans=10.0
2023-11-23 15:38:24,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2437900.0, ans=0.125
2023-11-23 15:38:35,386 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0
2023-11-23 15:38:41,995 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365700
2023-11-23 15:38:49,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2438033.3333333335, ans=0.125
2023-11-23 15:39:02,598 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5000, loss[loss=0.06989, simple_loss=0.08425, pruned_loss=0.01629, audio_tagging_loss=0.01147, over 15506.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09131, pruned_loss=0.01359, audio_tagging_loss=0.009234, over 3048943.43 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:39:05,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2438100.0, ans=0.0
2023-11-23 15:39:10,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2438100.0, ans=0.125
2023-11-23 15:39:23,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.57 vs. limit=22.5
2023-11-23 15:39:32,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2438233.3333333335, ans=0.2
2023-11-23 15:39:45,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365750
2023-11-23 15:39:52,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2438300.0, ans=0.2
2023-11-23 15:40:01,803 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.229e+01 8.801e+01 9.337e+01 1.229e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-23 15:40:07,582 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5050, loss[loss=0.08194, simple_loss=0.1183, pruned_loss=0.01557, audio_tagging_loss=0.007223, over 16273.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09203, pruned_loss=0.01385, audio_tagging_loss=0.00906, over 3043376.51 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:40:12,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2438433.3333333335, ans=0.2
2023-11-23 15:40:18,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2438433.3333333335, ans=0.125
2023-11-23 15:40:30,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2438500.0, ans=0.0
2023-11-23 15:40:35,508 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.94 vs. limit=15.0
2023-11-23 15:40:49,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365800
2023-11-23 15:40:50,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.91 vs. limit=15.0
2023-11-23 15:40:57,154 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.77 vs. limit=15.0
2023-11-23 15:40:57,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.14 vs. limit=15.0
2023-11-23 15:40:59,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2438700.0, ans=0.125
2023-11-23 15:40:59,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0
2023-11-23 15:41:04,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2438700.0, ans=0.125
2023-11-23 15:41:07,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2438700.0, ans=0.125
2023-11-23 15:41:09,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2438700.0, ans=0.1
2023-11-23 15:41:12,962 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5100, loss[loss=0.052, simple_loss=0.06452, pruned_loss=0.00835, audio_tagging_loss=0.0114, over 15590.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09224, pruned_loss=0.01391, audio_tagging_loss=0.009035, over 3040264.71 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:41:49,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2438966.6666666665, ans=0.125
2023-11-23 15:41:49,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2438966.6666666665, ans=0.0
2023-11-23 15:41:53,816 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365850
2023-11-23 15:41:59,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2438966.6666666665, ans=0.1
2023-11-23 15:42:06,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2439033.3333333335, ans=0.025
2023-11-23 15:42:10,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2439033.3333333335, ans=0.2
2023-11-23 15:42:11,230 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.471e+01 8.600e+01 9.200e+01 9.982e+01 4.087e+02, threshold=1.840e+02, percent-clipped=1.0
2023-11-23 15:42:11,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2439033.3333333335, ans=0.0
2023-11-23 15:42:16,215 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5150, loss[loss=0.07584, simple_loss=0.1038, pruned_loss=0.01413, audio_tagging_loss=0.009784, over 15073.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.0925, pruned_loss=0.01388, audio_tagging_loss=0.008983, over 3038251.37 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:42:16,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2439100.0, ans=10.0
2023-11-23 15:42:16,801 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.18 vs. limit=15.0
2023-11-23 15:42:29,150 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0
2023-11-23 15:42:38,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2439166.6666666665, ans=0.2
2023-11-23 15:42:42,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2439233.3333333335, ans=0.125
2023-11-23 15:42:53,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2439233.3333333335, ans=0.125
2023-11-23 15:42:55,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2439300.0, ans=0.125
2023-11-23 15:42:59,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365900
2023-11-23 15:43:05,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2439300.0, ans=0.2
2023-11-23 15:43:20,546 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5200, loss[loss=0.06605, simple_loss=0.09359, pruned_loss=0.01125, audio_tagging_loss=0.008002, over 15804.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09331, pruned_loss=0.01386, audio_tagging_loss=0.008851, over 3038120.01 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:43:56,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2439566.6666666665, ans=0.125
2023-11-23 15:44:04,174 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 365950
2023-11-23 15:44:21,105 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.221e+01 8.985e+01 9.483e+01 1.522e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 15:44:27,289 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5250, loss[loss=0.06165, simple_loss=0.0897, pruned_loss=0.009045, audio_tagging_loss=0.007758, over 15532.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09269, pruned_loss=0.01389, audio_tagging_loss=0.008861, over 3045010.38 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:44:53,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.45 vs. limit=22.5
2023-11-23 15:45:05,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2439966.6666666665, ans=0.1
2023-11-23 15:45:09,962 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366000
2023-11-23 15:45:13,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2439966.6666666665, ans=0.1
2023-11-23 15:45:33,376 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5300, loss[loss=0.06021, simple_loss=0.07715, pruned_loss=0.00968, audio_tagging_loss=0.01196, over 14872.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09353, pruned_loss=0.014, audio_tagging_loss=0.00879, over 3048020.87 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:45:52,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2440166.6666666665, ans=0.2
2023-11-23 15:46:01,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2440233.3333333335, ans=0.125
2023-11-23 15:46:17,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366050
2023-11-23 15:46:22,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2440300.0, ans=0.125
2023-11-23 15:46:32,938 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.490e+01 8.834e+01 9.644e+01 1.833e+02, threshold=1.767e+02, percent-clipped=1.0
2023-11-23 15:46:37,972 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5350, loss[loss=0.05509, simple_loss=0.07437, pruned_loss=0.009685, audio_tagging_loss=0.008218, over 14866.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09399, pruned_loss=0.01407, audio_tagging_loss=0.008732, over 3049288.15 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:47:06,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2440566.6666666665, ans=0.1
2023-11-23 15:47:10,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2440566.6666666665, ans=0.2
2023-11-23 15:47:21,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366100
2023-11-23 15:47:28,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2440700.0, ans=0.125
2023-11-23 15:47:30,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2440700.0, ans=0.125
2023-11-23 15:47:42,606 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5400, loss[loss=0.09316, simple_loss=0.1294, pruned_loss=0.0223, audio_tagging_loss=0.006152, over 14949.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09337, pruned_loss=0.01397, audio_tagging_loss=0.008914, over 3048325.07 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:47:55,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2440833.3333333335, ans=0.0
2023-11-23 15:48:13,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2440900.0, ans=0.125
2023-11-23 15:48:26,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366150
2023-11-23 15:48:31,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2440966.6666666665, ans=0.1
2023-11-23 15:48:33,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.62 vs. limit=5.0
2023-11-23 15:48:42,966 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.475e+01 9.130e+01 9.785e+01 1.452e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 15:48:48,680 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5450, loss[loss=0.06097, simple_loss=0.07859, pruned_loss=0.01346, audio_tagging_loss=0.008218, over 14996.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09393, pruned_loss=0.0141, audio_tagging_loss=0.008976, over 3057374.83 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:48:48,853 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:48:52,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2441100.0, ans=0.1
2023-11-23 15:48:56,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2441100.0, ans=0.2
2023-11-23 15:48:58,789 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:49:26,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2441300.0, ans=0.2
2023-11-23 15:49:31,105 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366200
2023-11-23 15:49:51,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2441433.3333333335, ans=0.1
2023-11-23 15:49:52,718 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5500, loss[loss=0.05191, simple_loss=0.06126, pruned_loss=0.01194, audio_tagging_loss=0.00934, over 13923.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09421, pruned_loss=0.01408, audio_tagging_loss=0.008969, over 3056379.47 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:50:04,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2441500.0, ans=0.0
2023-11-23 15:50:18,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2441566.6666666665, ans=0.125
2023-11-23 15:50:21,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2441566.6666666665, ans=0.125
2023-11-23 15:50:35,849 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366250
2023-11-23 15:50:48,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2441700.0, ans=0.125
2023-11-23 15:50:52,283 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.602e+01 8.303e+01 8.937e+01 9.609e+01 1.241e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 15:50:57,363 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5550, loss[loss=0.1001, simple_loss=0.1271, pruned_loss=0.02744, audio_tagging_loss=0.009088, over 15513.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09461, pruned_loss=0.01429, audio_tagging_loss=0.009027, over 3049150.74 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:51:07,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2441766.6666666665, ans=0.125
2023-11-23 15:51:11,054 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.41 vs. limit=15.0
2023-11-23 15:51:25,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2441900.0, ans=0.125
2023-11-23 15:51:29,043 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.41 vs. limit=15.0
2023-11-23 15:51:40,001 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366300
2023-11-23 15:51:40,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2441966.6666666665, ans=0.2
2023-11-23 15:51:56,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2442033.3333333335, ans=0.125
2023-11-23 15:52:01,531 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5600, loss[loss=0.0564, simple_loss=0.07744, pruned_loss=0.00785, audio_tagging_loss=0.009832, over 15268.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09455, pruned_loss=0.01416, audio_tagging_loss=0.009161, over 3047853.43 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:52:20,410 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.34 vs. limit=15.0
2023-11-23 15:52:43,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2442300.0, ans=0.1
2023-11-23 15:52:44,393 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366350
2023-11-23 15:52:48,047 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:52:54,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2442366.6666666665, ans=0.125
2023-11-23 15:53:00,690 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.372e+01 8.291e+01 8.935e+01 9.655e+01 1.271e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 15:53:05,723 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5650, loss[loss=0.08721, simple_loss=0.1187, pruned_loss=0.0194, audio_tagging_loss=0.008465, over 15330.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09552, pruned_loss=0.01442, audio_tagging_loss=0.009205, over 3055048.70 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:53:23,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2442500.0, ans=0.125
2023-11-23 15:53:47,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366400
2023-11-23 15:53:51,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2442633.3333333335, ans=0.2
2023-11-23 15:53:54,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.35 vs. limit=15.0
2023-11-23 15:54:09,825 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5700, loss[loss=0.09035, simple_loss=0.127, pruned_loss=0.019, audio_tagging_loss=0.007863, over 14896.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09396, pruned_loss=0.01413, audio_tagging_loss=0.009248, over 3048885.74 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:54:33,977 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:54:51,490 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366450
2023-11-23 15:55:08,818 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.337e+01 8.162e+01 8.678e+01 9.471e+01 1.109e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 15:55:13,708 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5750, loss[loss=0.06868, simple_loss=0.08698, pruned_loss=0.01573, audio_tagging_loss=0.009459, over 14694.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09308, pruned_loss=0.01408, audio_tagging_loss=0.009146, over 3049494.97 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:55:29,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2443166.6666666665, ans=0.1
2023-11-23 15:55:36,767 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:55:40,981 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.54 vs. limit=6.0
2023-11-23 15:55:43,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2443233.3333333335, ans=0.125
2023-11-23 15:55:56,275 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366500
2023-11-23 15:56:11,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2443366.6666666665, ans=0.0
2023-11-23 15:56:13,227 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=6.70 vs. limit=15.0
2023-11-23 15:56:17,462 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5800, loss[loss=0.06083, simple_loss=0.08655, pruned_loss=0.01055, audio_tagging_loss=0.007011, over 15932.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09313, pruned_loss=0.01406, audio_tagging_loss=0.008988, over 3048983.44 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:56:20,252 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.72 vs. limit=12.0
2023-11-23 15:56:32,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.29 vs. limit=10.0
2023-11-23 15:56:51,455 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.09 vs. limit=22.5
2023-11-23 15:56:58,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2443633.3333333335, ans=0.09899494936611666
2023-11-23 15:57:00,807 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366550
2023-11-23 15:57:18,850 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.421e+01 8.535e+01 9.028e+01 9.591e+01 1.185e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 15:57:22,554 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:57:23,532 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5850, loss[loss=0.08602, simple_loss=0.1119, pruned_loss=0.02169, audio_tagging_loss=0.008393, over 15799.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09291, pruned_loss=0.01401, audio_tagging_loss=0.008926, over 3046703.82 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:57:44,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2443833.3333333335, ans=0.125
2023-11-23 15:57:51,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.53 vs. limit=15.0
2023-11-23 15:57:55,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2443900.0, ans=0.1
2023-11-23 15:58:07,226 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366600
2023-11-23 15:58:17,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2444033.3333333335, ans=0.125
2023-11-23 15:58:21,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2444033.3333333335, ans=0.125
2023-11-23 15:58:22,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2444033.3333333335, ans=0.2
2023-11-23 15:58:29,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2444100.0, ans=0.1
2023-11-23 15:58:30,441 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5900, loss[loss=0.06744, simple_loss=0.09348, pruned_loss=0.01208, audio_tagging_loss=0.008621, over 16931.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.0922, pruned_loss=0.01386, audio_tagging_loss=0.008897, over 3039362.79 frames. ], batch size: 62, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:58:36,105 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.08 vs. limit=15.0
2023-11-23 15:58:46,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2444166.6666666665, ans=0.125
2023-11-23 15:58:51,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2444166.6666666665, ans=0.1
2023-11-23 15:59:05,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2444233.3333333335, ans=0.125
2023-11-23 15:59:08,461 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.56 vs. limit=15.0
2023-11-23 15:59:14,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366650
2023-11-23 15:59:21,111 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.12 vs. limit=15.0
2023-11-23 15:59:27,418 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.90 vs. limit=22.5
2023-11-23 15:59:31,530 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.501e+01 9.185e+01 9.833e+01 1.392e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 15:59:35,332 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 5950, loss[loss=0.0614, simple_loss=0.07289, pruned_loss=0.01439, audio_tagging_loss=0.01057, over 14513.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09197, pruned_loss=0.01381, audio_tagging_loss=0.008909, over 3041739.45 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:59:53,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2444500.0, ans=0.0
2023-11-23 15:59:55,203 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.21 vs. limit=12.0
2023-11-23 16:00:08,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0
2023-11-23 16:00:10,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.87 vs. limit=15.0
2023-11-23 16:00:19,505 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366700
2023-11-23 16:00:33,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.94 vs. limit=22.5
2023-11-23 16:00:41,890 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6000, loss[loss=0.07082, simple_loss=0.0971, pruned_loss=0.01255, audio_tagging_loss=0.009721, over 15208.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09167, pruned_loss=0.0138, audio_tagging_loss=0.008914, over 3036833.54 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:00:41,891 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 16:01:05,794 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7321, 5.4508, 4.9375, 5.4168], device='cuda:2')
2023-11-23 16:01:26,559 INFO [train_asr.py:1253] (2/4) Epoch 31, validation: loss=0.05852, simple_loss=0.05109, pruned_loss=0.00511, audio_tagging_loss=0.02786, over 4681554.00 frames. 
2023-11-23 16:01:26,559 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 16:01:38,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2444833.3333333335, ans=0.125
2023-11-23 16:01:41,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2444833.3333333335, ans=0.125
2023-11-23 16:01:43,449 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.97 vs. limit=15.0
2023-11-23 16:01:46,996 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.95 vs. limit=15.0
2023-11-23 16:02:00,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2444900.0, ans=0.125
2023-11-23 16:02:03,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2444966.6666666665, ans=0.125
2023-11-23 16:02:08,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366750
2023-11-23 16:02:13,135 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:02:14,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2444966.6666666665, ans=0.125
2023-11-23 16:02:16,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.69 vs. limit=22.5
2023-11-23 16:02:26,660 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.104e+01 8.225e+01 8.734e+01 9.674e+01 1.175e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-23 16:02:28,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2445033.3333333335, ans=0.2
2023-11-23 16:02:30,504 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6050, loss[loss=0.07399, simple_loss=0.08854, pruned_loss=0.01863, audio_tagging_loss=0.01109, over 15178.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09201, pruned_loss=0.01371, audio_tagging_loss=0.008876, over 3039133.39 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:02:37,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2445100.0, ans=0.0
2023-11-23 16:02:38,601 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.84 vs. limit=10.0
2023-11-23 16:03:05,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2445233.3333333335, ans=0.0
2023-11-23 16:03:14,274 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366800
2023-11-23 16:03:17,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2445300.0, ans=0.0
2023-11-23 16:03:29,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2445366.6666666665, ans=0.0
2023-11-23 16:03:30,883 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:03:36,438 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6100, loss[loss=0.06623, simple_loss=0.07877, pruned_loss=0.0166, audio_tagging_loss=0.01024, over 15884.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09217, pruned_loss=0.01393, audio_tagging_loss=0.008999, over 3034285.26 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:03:39,160 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.09 vs. limit=12.0
2023-11-23 16:03:45,485 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:03:45,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2445433.3333333335, ans=0.125
2023-11-23 16:03:57,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2445500.0, ans=0.125
2023-11-23 16:04:15,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2445633.3333333335, ans=10.0
2023-11-23 16:04:19,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366850
2023-11-23 16:04:26,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2445633.3333333335, ans=0.0
2023-11-23 16:04:27,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2445700.0, ans=0.125
2023-11-23 16:04:38,278 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.426e+01 8.983e+01 9.647e+01 1.287e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 16:04:42,612 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6150, loss[loss=0.06432, simple_loss=0.08492, pruned_loss=0.01189, audio_tagging_loss=0.009974, over 15554.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09203, pruned_loss=0.01386, audio_tagging_loss=0.008995, over 3035215.10 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:04:44,491 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.31 vs. limit=12.0
2023-11-23 16:04:45,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2445766.6666666665, ans=0.1
2023-11-23 16:04:49,401 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0
2023-11-23 16:04:51,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2445766.6666666665, ans=0.2
2023-11-23 16:04:54,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2445833.3333333335, ans=0.2
2023-11-23 16:05:01,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2445833.3333333335, ans=0.125
2023-11-23 16:05:01,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2445833.3333333335, ans=0.0
2023-11-23 16:05:13,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.67 vs. limit=22.5
2023-11-23 16:05:18,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2445900.0, ans=0.0
2023-11-23 16:05:25,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366900
2023-11-23 16:05:29,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2445966.6666666665, ans=0.125
2023-11-23 16:05:47,266 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6200, loss[loss=0.06228, simple_loss=0.07481, pruned_loss=0.01427, audio_tagging_loss=0.01061, over 15279.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09138, pruned_loss=0.01375, audio_tagging_loss=0.009043, over 3032651.35 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:06:01,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2446166.6666666665, ans=0.125
2023-11-23 16:06:23,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2446233.3333333335, ans=0.1
2023-11-23 16:06:31,184 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 366950
2023-11-23 16:06:41,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2446366.6666666665, ans=0.125
2023-11-23 16:06:48,425 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.382e+01 9.012e+01 9.950e+01 2.061e+02, threshold=1.802e+02, percent-clipped=1.0
2023-11-23 16:06:48,694 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:06:52,762 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6250, loss[loss=0.0456, simple_loss=0.06117, pruned_loss=0.008038, audio_tagging_loss=0.006981, over 14574.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09116, pruned_loss=0.01379, audio_tagging_loss=0.009098, over 3032613.92 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:07:05,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2446500.0, ans=0.125
2023-11-23 16:07:27,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2446566.6666666665, ans=0.1
2023-11-23 16:07:35,889 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367000
2023-11-23 16:07:55,133 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:07:58,524 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6300, loss[loss=0.06209, simple_loss=0.08378, pruned_loss=0.01192, audio_tagging_loss=0.008285, over 15306.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.092, pruned_loss=0.01382, audio_tagging_loss=0.009105, over 3041410.83 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:08:03,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2446766.6666666665, ans=0.09899494936611666
2023-11-23 16:08:40,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367050
2023-11-23 16:08:59,068 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.957e+01 8.355e+01 9.008e+01 9.768e+01 1.209e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 16:09:00,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2447033.3333333335, ans=0.1
2023-11-23 16:09:02,794 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6350, loss[loss=0.0587, simple_loss=0.07367, pruned_loss=0.01025, audio_tagging_loss=0.01162, over 15541.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09275, pruned_loss=0.01383, audio_tagging_loss=0.009104, over 3046832.82 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:09:11,874 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.47 vs. limit=15.0
2023-11-23 16:09:17,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2447166.6666666665, ans=0.125
2023-11-23 16:09:21,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2447166.6666666665, ans=0.125
2023-11-23 16:09:45,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367100
2023-11-23 16:10:00,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2447366.6666666665, ans=0.2
2023-11-23 16:10:06,302 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6400, loss[loss=0.05802, simple_loss=0.07409, pruned_loss=0.01072, audio_tagging_loss=0.01025, over 14762.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09314, pruned_loss=0.01402, audio_tagging_loss=0.009305, over 3039419.10 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:10:28,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2447500.0, ans=0.125
2023-11-23 16:10:32,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2447566.6666666665, ans=0.125
2023-11-23 16:10:49,441 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367150
2023-11-23 16:10:49,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2447633.3333333335, ans=0.025
2023-11-23 16:10:54,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2447633.3333333335, ans=0.125
2023-11-23 16:11:02,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2447700.0, ans=0.125
2023-11-23 16:11:08,617 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.384e+01 9.024e+01 9.742e+01 1.267e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 16:11:11,828 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6450, loss[loss=0.09283, simple_loss=0.1411, pruned_loss=0.01726, audio_tagging_loss=0.005006, over 15074.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09261, pruned_loss=0.0138, audio_tagging_loss=0.009298, over 3032097.36 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:11:34,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2447833.3333333335, ans=0.0
2023-11-23 16:11:34,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2447833.3333333335, ans=0.125
2023-11-23 16:11:53,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367200
2023-11-23 16:12:05,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2448033.3333333335, ans=0.0
2023-11-23 16:12:14,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2448033.3333333335, ans=0.125
2023-11-23 16:12:17,092 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6500, loss[loss=0.07992, simple_loss=0.1103, pruned_loss=0.01721, audio_tagging_loss=0.007565, over 15114.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09181, pruned_loss=0.01373, audio_tagging_loss=0.009206, over 3039974.76 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:13:00,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367250
2023-11-23 16:13:01,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2448300.0, ans=0.125
2023-11-23 16:13:12,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=15.0
2023-11-23 16:13:17,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2448366.6666666665, ans=0.125
2023-11-23 16:13:18,765 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.647e+01 8.406e+01 9.076e+01 9.815e+01 1.354e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 16:13:21,305 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6550, loss[loss=0.07893, simple_loss=0.1089, pruned_loss=0.01643, audio_tagging_loss=0.008069, over 15987.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09235, pruned_loss=0.01386, audio_tagging_loss=0.009128, over 3043993.87 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:13:35,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2448500.0, ans=0.125
2023-11-23 16:14:03,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2448633.3333333335, ans=0.125
2023-11-23 16:14:04,629 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367300
2023-11-23 16:14:18,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2448700.0, ans=0.0
2023-11-23 16:14:25,930 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6600, loss[loss=0.1036, simple_loss=0.1304, pruned_loss=0.03011, audio_tagging_loss=0.008296, over 16093.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09237, pruned_loss=0.01397, audio_tagging_loss=0.008947, over 3042379.33 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:14:42,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2448833.3333333335, ans=0.0
2023-11-23 16:15:07,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2448966.6666666665, ans=0.125
2023-11-23 16:15:08,972 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367350
2023-11-23 16:15:10,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2448966.6666666665, ans=0.0
2023-11-23 16:15:11,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2448966.6666666665, ans=0.04949747468305833
2023-11-23 16:15:14,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2448966.6666666665, ans=0.0
2023-11-23 16:15:31,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.404e+01 9.056e+01 9.809e+01 1.686e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 16:15:31,502 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6650, loss[loss=0.06555, simple_loss=0.09873, pruned_loss=0.007601, audio_tagging_loss=0.008579, over 16136.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09225, pruned_loss=0.01392, audio_tagging_loss=0.008911, over 3033622.40 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:15:39,473 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.05 vs. limit=15.0
2023-11-23 16:15:51,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2449166.6666666665, ans=0.125
2023-11-23 16:15:57,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.25 vs. limit=22.5
2023-11-23 16:15:58,741 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.43 vs. limit=6.0
2023-11-23 16:16:11,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2449300.0, ans=10.0
2023-11-23 16:16:14,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367400
2023-11-23 16:16:18,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2449300.0, ans=0.125
2023-11-23 16:16:26,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.57 vs. limit=15.0
2023-11-23 16:16:35,756 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6700, loss[loss=0.0732, simple_loss=0.09498, pruned_loss=0.01645, audio_tagging_loss=0.009264, over 15435.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09149, pruned_loss=0.01376, audio_tagging_loss=0.008966, over 3033854.67 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:16:36,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.89 vs. limit=6.0
2023-11-23 16:16:38,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.32 vs. limit=15.0
2023-11-23 16:16:45,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2449433.3333333335, ans=0.125
2023-11-23 16:17:06,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2449566.6666666665, ans=0.0
2023-11-23 16:17:19,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367450
2023-11-23 16:17:19,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2449633.3333333335, ans=0.0
2023-11-23 16:17:25,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2449633.3333333335, ans=0.125
2023-11-23 16:17:27,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2449700.0, ans=0.125
2023-11-23 16:17:29,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2449700.0, ans=0.2
2023-11-23 16:17:37,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=2449700.0, ans=15.0
2023-11-23 16:17:40,240 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.536e+01 8.542e+01 9.005e+01 9.931e+01 1.410e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 16:17:40,297 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6750, loss[loss=0.06577, simple_loss=0.08997, pruned_loss=0.01053, audio_tagging_loss=0.01025, over 15902.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09114, pruned_loss=0.01365, audio_tagging_loss=0.008951, over 3033015.26 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:17:51,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2449766.6666666665, ans=0.125
2023-11-23 16:18:22,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.93 vs. limit=6.0
2023-11-23 16:18:23,586 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367500
2023-11-23 16:18:31,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.75 vs. limit=10.0
2023-11-23 16:18:35,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2450033.3333333335, ans=0.0
2023-11-23 16:18:42,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2450033.3333333335, ans=0.0
2023-11-23 16:18:42,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2450033.3333333335, ans=0.125
2023-11-23 16:18:45,574 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6800, loss[loss=0.08347, simple_loss=0.1207, pruned_loss=0.01713, audio_tagging_loss=0.005964, over 15110.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.0919, pruned_loss=0.01371, audio_tagging_loss=0.008896, over 3038125.65 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:18:45,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2450100.0, ans=0.125
2023-11-23 16:18:57,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2450166.6666666665, ans=0.125
2023-11-23 16:19:17,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.83 vs. limit=22.5
2023-11-23 16:19:28,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367550
2023-11-23 16:19:28,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2450300.0, ans=0.0
2023-11-23 16:19:35,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2450300.0, ans=0.125
2023-11-23 16:19:38,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2450366.6666666665, ans=0.05
2023-11-23 16:19:44,427 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.59 vs. limit=22.5
2023-11-23 16:19:50,466 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6850, loss[loss=0.08388, simple_loss=0.1081, pruned_loss=0.02035, audio_tagging_loss=0.009478, over 14580.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09171, pruned_loss=0.01358, audio_tagging_loss=0.008818, over 3039415.05 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:19:51,628 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.211e+01 8.952e+01 9.815e+01 1.222e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 16:20:03,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2450500.0, ans=0.125
2023-11-23 16:20:16,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2450566.6666666665, ans=0.0
2023-11-23 16:20:16,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2450566.6666666665, ans=0.07
2023-11-23 16:20:19,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2450566.6666666665, ans=0.1
2023-11-23 16:20:22,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2450566.6666666665, ans=0.125
2023-11-23 16:20:29,813 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:20:33,199 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367600
2023-11-23 16:20:34,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2450633.3333333335, ans=0.125
2023-11-23 16:20:55,376 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.33 vs. limit=15.0
2023-11-23 16:20:55,992 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6900, loss[loss=0.0709, simple_loss=0.1027, pruned_loss=0.01373, audio_tagging_loss=0.005814, over 15373.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09245, pruned_loss=0.0136, audio_tagging_loss=0.008779, over 3040612.20 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:21:14,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2450833.3333333335, ans=0.125
2023-11-23 16:21:18,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2450833.3333333335, ans=0.1
2023-11-23 16:21:19,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2450833.3333333335, ans=0.125
2023-11-23 16:21:37,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2450966.6666666665, ans=15.0
2023-11-23 16:21:38,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367650
2023-11-23 16:21:46,591 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:21:51,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2451033.3333333335, ans=0.1
2023-11-23 16:21:58,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2451033.3333333335, ans=0.125
2023-11-23 16:22:01,290 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 6950, loss[loss=0.05571, simple_loss=0.07133, pruned_loss=0.009923, audio_tagging_loss=0.01012, over 15701.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09213, pruned_loss=0.01363, audio_tagging_loss=0.00887, over 3042327.05 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:22:02,443 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.751e+01 8.116e+01 8.969e+01 9.790e+01 1.259e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 16:22:38,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2451300.0, ans=0.0
2023-11-23 16:22:42,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2451300.0, ans=0.125
2023-11-23 16:22:44,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367700
2023-11-23 16:22:46,892 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.29 vs. limit=22.5
2023-11-23 16:23:05,239 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7000, loss[loss=0.06244, simple_loss=0.06447, pruned_loss=0.01409, audio_tagging_loss=0.01612, over 16182.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09264, pruned_loss=0.01379, audio_tagging_loss=0.008923, over 3052073.32 frames. ], batch size: 63, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:23:08,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2451433.3333333335, ans=0.125
2023-11-23 16:23:26,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2451500.0, ans=0.0
2023-11-23 16:23:36,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2451566.6666666665, ans=0.0
2023-11-23 16:23:39,161 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.74 vs. limit=22.5
2023-11-23 16:23:48,398 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367750
2023-11-23 16:23:54,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2451633.3333333335, ans=0.125
2023-11-23 16:24:10,635 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7050, loss[loss=0.06309, simple_loss=0.08188, pruned_loss=0.012, audio_tagging_loss=0.01014, over 14448.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09205, pruned_loss=0.01371, audio_tagging_loss=0.008959, over 3042928.33 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:24:11,787 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.295e+01 8.920e+01 9.938e+01 1.396e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 16:24:21,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2451766.6666666665, ans=10.0
2023-11-23 16:24:25,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.67 vs. limit=10.0
2023-11-23 16:24:27,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2451833.3333333335, ans=0.2
2023-11-23 16:24:29,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2451833.3333333335, ans=0.125
2023-11-23 16:24:31,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2451833.3333333335, ans=0.0
2023-11-23 16:24:46,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2451900.0, ans=0.1
2023-11-23 16:24:53,010 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367800
2023-11-23 16:25:00,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2451966.6666666665, ans=0.125
2023-11-23 16:25:10,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2452033.3333333335, ans=0.2
2023-11-23 16:25:15,568 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7100, loss[loss=0.05809, simple_loss=0.08032, pruned_loss=0.01309, audio_tagging_loss=0.004842, over 14925.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.0918, pruned_loss=0.01367, audio_tagging_loss=0.009068, over 3038125.02 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:25:21,612 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.77 vs. limit=22.5
2023-11-23 16:25:32,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2452166.6666666665, ans=0.1
2023-11-23 16:25:41,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.14 vs. limit=15.0
2023-11-23 16:25:47,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2452233.3333333335, ans=0.125
2023-11-23 16:25:58,062 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367850
2023-11-23 16:26:01,934 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0
2023-11-23 16:26:16,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2452366.6666666665, ans=0.125
2023-11-23 16:26:20,281 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7150, loss[loss=0.03942, simple_loss=0.0475, pruned_loss=0.004877, audio_tagging_loss=0.01079, over 17194.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09248, pruned_loss=0.01389, audio_tagging_loss=0.009115, over 3045477.30 frames. ], batch size: 69, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:26:21,451 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.184e+01 8.537e+01 9.022e+01 9.919e+01 1.379e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 16:26:43,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2452500.0, ans=0.125
2023-11-23 16:26:58,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2452633.3333333335, ans=0.2
2023-11-23 16:27:03,255 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367900
2023-11-23 16:27:04,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2452633.3333333335, ans=0.125
2023-11-23 16:27:12,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2452700.0, ans=0.125
2023-11-23 16:27:15,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2452700.0, ans=0.125
2023-11-23 16:27:24,435 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7200, loss[loss=0.05258, simple_loss=0.06652, pruned_loss=0.00825, audio_tagging_loss=0.01107, over 14281.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09125, pruned_loss=0.01365, audio_tagging_loss=0.009262, over 3038046.91 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:27:31,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2452766.6666666665, ans=0.0
2023-11-23 16:27:36,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2452833.3333333335, ans=0.0
2023-11-23 16:27:47,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2452833.3333333335, ans=0.1
2023-11-23 16:27:58,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2452900.0, ans=0.0
2023-11-23 16:28:06,771 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 367950
2023-11-23 16:28:10,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2452966.6666666665, ans=0.1
2023-11-23 16:28:15,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2453033.3333333335, ans=0.1
2023-11-23 16:28:29,992 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7250, loss[loss=0.07133, simple_loss=0.102, pruned_loss=0.0131, audio_tagging_loss=0.007229, over 15235.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09221, pruned_loss=0.01381, audio_tagging_loss=0.009202, over 3044483.08 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:28:31,186 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.513e+01 9.044e+01 9.572e+01 1.281e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 16:28:34,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2453100.0, ans=0.0
2023-11-23 16:28:47,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2453166.6666666665, ans=0.0
2023-11-23 16:29:12,965 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368000
2023-11-23 16:29:25,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2453366.6666666665, ans=0.125
2023-11-23 16:29:34,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2453366.6666666665, ans=0.125
2023-11-23 16:29:38,265 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7300, loss[loss=0.07081, simple_loss=0.08921, pruned_loss=0.0179, audio_tagging_loss=0.008309, over 14759.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09257, pruned_loss=0.0139, audio_tagging_loss=0.009155, over 3039395.98 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:30:22,129 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368050
2023-11-23 16:30:31,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2453700.0, ans=0.125
2023-11-23 16:30:42,832 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7350, loss[loss=0.0507, simple_loss=0.06506, pruned_loss=0.008574, audio_tagging_loss=0.009598, over 15784.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09262, pruned_loss=0.01395, audio_tagging_loss=0.009054, over 3042704.85 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:30:44,145 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.269e+01 8.908e+01 9.511e+01 1.304e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 16:31:09,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2453900.0, ans=0.0
2023-11-23 16:31:10,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2453900.0, ans=0.1
2023-11-23 16:31:11,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.43 vs. limit=15.0
2023-11-23 16:31:19,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2453900.0, ans=0.0
2023-11-23 16:31:24,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2453966.6666666665, ans=0.125
2023-11-23 16:31:26,589 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368100
2023-11-23 16:31:26,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2453966.6666666665, ans=0.07
2023-11-23 16:31:27,271 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.09 vs. limit=6.0
2023-11-23 16:31:38,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2454033.3333333335, ans=0.2
2023-11-23 16:31:39,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2454033.3333333335, ans=0.1
2023-11-23 16:31:49,063 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7400, loss[loss=0.0631, simple_loss=0.08577, pruned_loss=0.01431, audio_tagging_loss=0.005903, over 14406.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.0917, pruned_loss=0.01382, audio_tagging_loss=0.008962, over 3041284.16 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:31:55,457 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.79 vs. limit=15.0
2023-11-23 16:32:26,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2454233.3333333335, ans=0.2
2023-11-23 16:32:32,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368150
2023-11-23 16:32:55,331 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7450, loss[loss=0.0767, simple_loss=0.1043, pruned_loss=0.01569, audio_tagging_loss=0.008834, over 14828.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09211, pruned_loss=0.01396, audio_tagging_loss=0.008943, over 3041073.07 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:32:56,567 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.753e+01 8.234e+01 8.919e+01 9.567e+01 1.256e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 16:33:03,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.16 vs. limit=12.0
2023-11-23 16:33:10,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2454500.0, ans=0.015
2023-11-23 16:33:38,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368200
2023-11-23 16:33:59,413 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7500, loss[loss=0.06266, simple_loss=0.08153, pruned_loss=0.01202, audio_tagging_loss=0.009874, over 14427.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09227, pruned_loss=0.01408, audio_tagging_loss=0.008946, over 3043661.14 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:34:00,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2454766.6666666665, ans=0.125
2023-11-23 16:34:03,824 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.75 vs. limit=15.0
2023-11-23 16:34:29,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=2454900.0, ans=0.2
2023-11-23 16:34:42,469 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368250
2023-11-23 16:35:03,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=12.0
2023-11-23 16:35:04,298 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7550, loss[loss=0.04402, simple_loss=0.05537, pruned_loss=0.007423, audio_tagging_loss=0.008916, over 14480.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09241, pruned_loss=0.0141, audio_tagging_loss=0.008837, over 3052096.42 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:35:05,438 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.880e+01 8.421e+01 8.983e+01 9.712e+01 1.318e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 16:35:07,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.27 vs. limit=15.0
2023-11-23 16:35:17,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2455166.6666666665, ans=0.125
2023-11-23 16:35:31,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2455233.3333333335, ans=0.2
2023-11-23 16:35:48,354 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368300
2023-11-23 16:35:49,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2455300.0, ans=0.125
2023-11-23 16:35:54,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2455300.0, ans=0.125
2023-11-23 16:36:06,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2455366.6666666665, ans=0.1
2023-11-23 16:36:10,441 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7600, loss[loss=0.06198, simple_loss=0.08399, pruned_loss=0.01095, audio_tagging_loss=0.00904, over 15021.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09289, pruned_loss=0.01407, audio_tagging_loss=0.008871, over 3056115.37 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:36:29,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.74 vs. limit=6.0
2023-11-23 16:36:42,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2455566.6666666665, ans=0.2
2023-11-23 16:36:45,885 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:36:49,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2455633.3333333335, ans=0.125
2023-11-23 16:36:53,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368350
2023-11-23 16:37:15,058 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7650, loss[loss=0.06347, simple_loss=0.08723, pruned_loss=0.0109, audio_tagging_loss=0.00896, over 15541.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09276, pruned_loss=0.0141, audio_tagging_loss=0.008921, over 3054422.27 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:37:15,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2455766.6666666665, ans=0.125
2023-11-23 16:37:16,298 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 8.296e+01 8.999e+01 9.815e+01 1.332e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 16:37:17,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2455766.6666666665, ans=0.0
2023-11-23 16:37:19,372 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.54 vs. limit=15.0
2023-11-23 16:37:28,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2455833.3333333335, ans=0.125
2023-11-23 16:37:40,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2455900.0, ans=0.1
2023-11-23 16:37:58,752 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368400
2023-11-23 16:38:01,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2455966.6666666665, ans=0.035
2023-11-23 16:38:01,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2455966.6666666665, ans=0.09899494936611666
2023-11-23 16:38:11,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2456033.3333333335, ans=0.2
2023-11-23 16:38:20,823 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7700, loss[loss=0.07276, simple_loss=0.1016, pruned_loss=0.01429, audio_tagging_loss=0.007657, over 15646.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09199, pruned_loss=0.01387, audio_tagging_loss=0.008995, over 3050788.08 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:38:21,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2456100.0, ans=10.0
2023-11-23 16:38:22,722 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0
2023-11-23 16:38:24,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2456100.0, ans=0.09899494936611666
2023-11-23 16:38:41,910 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.33 vs. limit=15.0
2023-11-23 16:39:03,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2456300.0, ans=0.2
2023-11-23 16:39:03,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.16 vs. limit=10.0
2023-11-23 16:39:04,001 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368450
2023-11-23 16:39:19,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-23 16:39:20,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2456366.6666666665, ans=0.125
2023-11-23 16:39:21,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2456366.6666666665, ans=0.125
2023-11-23 16:39:26,199 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7750, loss[loss=0.06236, simple_loss=0.0795, pruned_loss=0.01102, audio_tagging_loss=0.01159, over 16418.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09106, pruned_loss=0.01379, audio_tagging_loss=0.009164, over 3050130.06 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:39:29,814 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.166e+01 8.983e+01 1.017e+02 1.300e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 16:39:32,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2456433.3333333335, ans=0.125
2023-11-23 16:39:38,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2456500.0, ans=0.125
2023-11-23 16:39:40,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2456500.0, ans=0.125
2023-11-23 16:39:41,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2456500.0, ans=0.125
2023-11-23 16:39:47,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2456500.0, ans=0.05
2023-11-23 16:39:49,931 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:39:59,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2456566.6666666665, ans=0.125
2023-11-23 16:40:08,802 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368500
2023-11-23 16:40:10,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2456633.3333333335, ans=0.125
2023-11-23 16:40:11,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2456633.3333333335, ans=0.125
2023-11-23 16:40:17,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2456700.0, ans=10.0
2023-11-23 16:40:25,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2456700.0, ans=0.125
2023-11-23 16:40:30,631 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7800, loss[loss=0.08541, simple_loss=0.1184, pruned_loss=0.01969, audio_tagging_loss=0.006537, over 15161.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09282, pruned_loss=0.01408, audio_tagging_loss=0.009081, over 3049944.86 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:40:34,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2456766.6666666665, ans=0.5
2023-11-23 16:40:48,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2456833.3333333335, ans=0.125
2023-11-23 16:40:53,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2456833.3333333335, ans=0.125
2023-11-23 16:41:09,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2456966.6666666665, ans=0.125
2023-11-23 16:41:13,762 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368550
2023-11-23 16:41:17,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2456966.6666666665, ans=0.125
2023-11-23 16:41:35,463 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7850, loss[loss=0.0806, simple_loss=0.1073, pruned_loss=0.01923, audio_tagging_loss=0.007722, over 15107.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09309, pruned_loss=0.0141, audio_tagging_loss=0.009195, over 3049313.67 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:41:39,078 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 8.382e+01 9.207e+01 1.008e+02 1.649e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-23 16:41:48,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2457166.6666666665, ans=0.2
2023-11-23 16:42:02,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2457233.3333333335, ans=0.015
2023-11-23 16:42:15,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2457300.0, ans=0.0
2023-11-23 16:42:16,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.01 vs. limit=15.0
2023-11-23 16:42:17,997 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368600
2023-11-23 16:42:32,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2457366.6666666665, ans=0.0
2023-11-23 16:42:40,593 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7900, loss[loss=0.05803, simple_loss=0.07138, pruned_loss=0.01085, audio_tagging_loss=0.01149, over 14543.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09304, pruned_loss=0.01409, audio_tagging_loss=0.009273, over 3044986.48 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:42:42,100 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:42:48,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2457433.3333333335, ans=0.125
2023-11-23 16:43:13,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2457566.6666666665, ans=0.0
2023-11-23 16:43:20,288 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.75 vs. limit=22.5
2023-11-23 16:43:23,354 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368650
2023-11-23 16:43:34,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2457700.0, ans=0.0
2023-11-23 16:43:45,884 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 7950, loss[loss=0.05954, simple_loss=0.08495, pruned_loss=0.008872, audio_tagging_loss=0.008192, over 16365.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09201, pruned_loss=0.01378, audio_tagging_loss=0.009397, over 3044340.67 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:43:49,525 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.234e+01 8.856e+01 9.622e+01 1.359e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 16:43:54,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2457766.6666666665, ans=0.125
2023-11-23 16:44:01,287 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:44:11,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2457900.0, ans=0.0
2023-11-23 16:44:14,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=2457900.0, ans=0.02
2023-11-23 16:44:22,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2457900.0, ans=0.0
2023-11-23 16:44:23,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2457966.6666666665, ans=22.5
2023-11-23 16:44:23,894 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.89 vs. limit=22.5
2023-11-23 16:44:28,249 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368700
2023-11-23 16:44:50,928 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8000, loss[loss=0.07356, simple_loss=0.09731, pruned_loss=0.01383, audio_tagging_loss=0.01108, over 15701.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.0924, pruned_loss=0.01381, audio_tagging_loss=0.009476, over 3045824.28 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:45:22,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2458233.3333333335, ans=0.2
2023-11-23 16:45:33,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368750
2023-11-23 16:45:36,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2458300.0, ans=0.2
2023-11-23 16:45:55,495 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8050, loss[loss=0.06887, simple_loss=0.09439, pruned_loss=0.01168, audio_tagging_loss=0.009991, over 15212.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09233, pruned_loss=0.01379, audio_tagging_loss=0.009446, over 3042053.25 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:46:00,364 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.362e+01 8.796e+01 9.330e+01 1.321e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 16:46:09,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2458500.0, ans=0.0
2023-11-23 16:46:14,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.88 vs. limit=15.0
2023-11-23 16:46:28,363 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.21 vs. limit=15.0
2023-11-23 16:46:28,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.85 vs. limit=15.0
2023-11-23 16:46:38,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368800
2023-11-23 16:46:50,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2458700.0, ans=0.125
2023-11-23 16:47:00,256 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8100, loss[loss=0.06342, simple_loss=0.09679, pruned_loss=0.01074, audio_tagging_loss=0.004292, over 16753.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09113, pruned_loss=0.01353, audio_tagging_loss=0.009389, over 3041283.83 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:47:14,801 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.89 vs. limit=15.0
2023-11-23 16:47:15,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2458833.3333333335, ans=0.125
2023-11-23 16:47:18,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2458833.3333333335, ans=0.0
2023-11-23 16:47:43,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368850
2023-11-23 16:47:51,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2459033.3333333335, ans=0.07
2023-11-23 16:48:04,225 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8150, loss[loss=0.08576, simple_loss=0.1146, pruned_loss=0.02048, audio_tagging_loss=0.007986, over 15312.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09246, pruned_loss=0.01386, audio_tagging_loss=0.00917, over 3051036.95 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:48:09,575 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.557e+01 9.149e+01 9.819e+01 1.226e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 16:48:20,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2459166.6666666665, ans=0.0
2023-11-23 16:48:32,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.07 vs. limit=10.0
2023-11-23 16:48:46,908 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368900
2023-11-23 16:48:56,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2459366.6666666665, ans=0.125
2023-11-23 16:48:59,259 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0
2023-11-23 16:49:08,932 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8200, loss[loss=0.04486, simple_loss=0.06085, pruned_loss=0.006287, audio_tagging_loss=0.008154, over 15063.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09241, pruned_loss=0.01377, audio_tagging_loss=0.009061, over 3047386.21 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:49:10,214 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:49:29,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2459500.0, ans=0.125
2023-11-23 16:49:48,216 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.03 vs. limit=15.0
2023-11-23 16:49:51,297 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 368950
2023-11-23 16:50:07,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2459700.0, ans=0.1
2023-11-23 16:50:07,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2459700.0, ans=0.1
2023-11-23 16:50:13,325 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8250, loss[loss=0.07836, simple_loss=0.1055, pruned_loss=0.01708, audio_tagging_loss=0.008502, over 15350.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09199, pruned_loss=0.01376, audio_tagging_loss=0.009121, over 3050297.92 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:50:18,095 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.120e+01 8.966e+01 9.710e+01 1.268e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 16:50:31,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2459833.3333333335, ans=0.0
2023-11-23 16:50:39,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2459900.0, ans=0.125
2023-11-23 16:50:56,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369000
2023-11-23 16:51:18,621 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8300, loss[loss=0.07985, simple_loss=0.1066, pruned_loss=0.01916, audio_tagging_loss=0.007395, over 15502.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09074, pruned_loss=0.01361, audio_tagging_loss=0.009126, over 3052613.94 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:51:36,619 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.93 vs. limit=22.5
2023-11-23 16:52:01,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369050
2023-11-23 16:52:01,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2460300.0, ans=0.0
2023-11-23 16:52:04,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2460300.0, ans=0.125
2023-11-23 16:52:17,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2460366.6666666665, ans=0.1
2023-11-23 16:52:23,790 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8350, loss[loss=0.06733, simple_loss=0.09401, pruned_loss=0.01057, audio_tagging_loss=0.009755, over 15074.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09163, pruned_loss=0.01381, audio_tagging_loss=0.009047, over 3048135.07 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:52:29,174 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.409e+01 8.427e+01 9.150e+01 9.760e+01 1.195e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 16:52:50,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2460566.6666666665, ans=0.125
2023-11-23 16:53:06,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369100
2023-11-23 16:53:28,827 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8400, loss[loss=0.06055, simple_loss=0.08263, pruned_loss=0.009353, audio_tagging_loss=0.009884, over 15604.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09145, pruned_loss=0.01378, audio_tagging_loss=0.008987, over 3046762.62 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:53:29,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2460766.6666666665, ans=0.125
2023-11-23 16:53:30,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2460766.6666666665, ans=0.0
2023-11-23 16:54:10,936 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369150
2023-11-23 16:54:31,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2461100.0, ans=0.125
2023-11-23 16:54:32,568 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8450, loss[loss=0.08553, simple_loss=0.1175, pruned_loss=0.01828, audio_tagging_loss=0.00853, over 16388.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09137, pruned_loss=0.01377, audio_tagging_loss=0.009071, over 3050440.93 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:54:37,475 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.445e+01 8.837e+01 9.573e+01 1.326e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 16:54:37,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2461100.0, ans=0.2
2023-11-23 16:54:50,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2461166.6666666665, ans=0.125
2023-11-23 16:55:04,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2461233.3333333335, ans=10.0
2023-11-23 16:55:16,207 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369200
2023-11-23 16:55:26,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2461366.6666666665, ans=0.0
2023-11-23 16:55:38,829 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8500, loss[loss=0.05892, simple_loss=0.07208, pruned_loss=0.01253, audio_tagging_loss=0.01034, over 16579.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09088, pruned_loss=0.0136, audio_tagging_loss=0.009113, over 3057613.09 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:55:48,491 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.57 vs. limit=22.5
2023-11-23 16:56:04,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2461566.6666666665, ans=0.1
2023-11-23 16:56:21,316 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369250
2023-11-23 16:56:28,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2461633.3333333335, ans=0.1
2023-11-23 16:56:33,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.63 vs. limit=12.0
2023-11-23 16:56:44,050 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8550, loss[loss=0.07752, simple_loss=0.1036, pruned_loss=0.01799, audio_tagging_loss=0.007745, over 14800.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09152, pruned_loss=0.01376, audio_tagging_loss=0.009093, over 3050356.57 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:56:49,050 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.318e+01 8.758e+01 9.491e+01 1.282e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 16:57:05,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2461833.3333333335, ans=0.125
2023-11-23 16:57:21,622 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:57:26,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369300
2023-11-23 16:57:48,089 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8600, loss[loss=0.08277, simple_loss=0.1107, pruned_loss=0.01878, audio_tagging_loss=0.008629, over 15072.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09227, pruned_loss=0.01393, audio_tagging_loss=0.00908, over 3048252.76 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:57:52,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.81 vs. limit=15.0
2023-11-23 16:58:19,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2462233.3333333335, ans=0.125
2023-11-23 16:58:31,378 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369350
2023-11-23 16:58:36,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2462300.0, ans=0.125
2023-11-23 16:58:37,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2462300.0, ans=0.1
2023-11-23 16:58:49,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2462366.6666666665, ans=0.1
2023-11-23 16:58:52,580 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8650, loss[loss=0.06192, simple_loss=0.08984, pruned_loss=0.00836, audio_tagging_loss=0.008643, over 14290.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09219, pruned_loss=0.01372, audio_tagging_loss=0.009127, over 3047622.26 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:58:57,967 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.075e+01 8.582e+01 9.201e+01 9.858e+01 1.215e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 16:59:11,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2462500.0, ans=0.2
2023-11-23 16:59:15,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.58 vs. limit=10.0
2023-11-23 16:59:35,331 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369400
2023-11-23 16:59:50,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2462700.0, ans=0.125
2023-11-23 16:59:57,836 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8700, loss[loss=0.05586, simple_loss=0.07537, pruned_loss=0.00969, audio_tagging_loss=0.008491, over 15099.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09255, pruned_loss=0.01393, audio_tagging_loss=0.009252, over 3050325.08 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:00:07,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2462766.6666666665, ans=0.125
2023-11-23 17:00:17,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2462833.3333333335, ans=0.0
2023-11-23 17:00:20,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2462833.3333333335, ans=0.035
2023-11-23 17:00:20,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2462833.3333333335, ans=0.1
2023-11-23 17:00:41,052 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369450
2023-11-23 17:01:02,631 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8750, loss[loss=0.0792, simple_loss=0.1081, pruned_loss=0.01646, audio_tagging_loss=0.008709, over 15770.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09374, pruned_loss=0.01412, audio_tagging_loss=0.009219, over 3053923.15 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:01:07,433 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.270e+01 8.973e+01 9.908e+01 1.550e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 17:01:16,621 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.10 vs. limit=15.0
2023-11-23 17:01:30,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2463233.3333333335, ans=0.2
2023-11-23 17:01:44,905 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.60 vs. limit=22.5
2023-11-23 17:01:45,509 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369500
2023-11-23 17:01:46,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2463300.0, ans=6.0
2023-11-23 17:02:06,819 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8800, loss[loss=0.06516, simple_loss=0.0799, pruned_loss=0.01444, audio_tagging_loss=0.01077, over 15473.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09347, pruned_loss=0.014, audio_tagging_loss=0.009251, over 3053184.89 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:02:07,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.43 vs. limit=15.0
2023-11-23 17:02:08,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2463433.3333333335, ans=0.2
2023-11-23 17:02:10,789 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:02:27,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2463500.0, ans=0.0
2023-11-23 17:02:49,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369550
2023-11-23 17:03:05,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2463700.0, ans=0.0
2023-11-23 17:03:12,312 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8850, loss[loss=0.05602, simple_loss=0.07887, pruned_loss=0.01003, audio_tagging_loss=0.006553, over 14715.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.0933, pruned_loss=0.01407, audio_tagging_loss=0.009239, over 3052941.21 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:03:17,844 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.515e+01 9.142e+01 9.954e+01 1.396e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 17:03:24,149 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:03:24,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2463833.3333333335, ans=0.0
2023-11-23 17:03:28,546 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.38 vs. limit=15.0
2023-11-23 17:03:53,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2463966.6666666665, ans=0.2
2023-11-23 17:03:54,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369600
2023-11-23 17:04:09,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2464033.3333333335, ans=0.0
2023-11-23 17:04:16,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2464100.0, ans=0.125
2023-11-23 17:04:17,295 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8900, loss[loss=0.05271, simple_loss=0.06725, pruned_loss=0.009209, audio_tagging_loss=0.009871, over 15988.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09335, pruned_loss=0.01407, audio_tagging_loss=0.00909, over 3055765.18 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:04:18,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2464100.0, ans=0.2
2023-11-23 17:04:31,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2464166.6666666665, ans=0.0
2023-11-23 17:04:39,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2464166.6666666665, ans=0.0
2023-11-23 17:05:00,779 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369650
2023-11-23 17:05:22,409 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 8950, loss[loss=0.07973, simple_loss=0.09855, pruned_loss=0.01966, audio_tagging_loss=0.0108, over 15124.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09333, pruned_loss=0.01397, audio_tagging_loss=0.008978, over 3053685.77 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:05:27,243 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.154e+01 8.323e+01 9.092e+01 9.814e+01 1.259e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 17:05:30,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2464433.3333333335, ans=0.5
2023-11-23 17:06:00,332 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:06:02,002 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.57 vs. limit=6.0
2023-11-23 17:06:04,930 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369700
2023-11-23 17:06:05,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2464633.3333333335, ans=0.0
2023-11-23 17:06:08,274 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.64 vs. limit=15.0
2023-11-23 17:06:12,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2464633.3333333335, ans=0.2
2023-11-23 17:06:25,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2464700.0, ans=0.2
2023-11-23 17:06:27,170 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9000, loss[loss=0.06566, simple_loss=0.09266, pruned_loss=0.01187, audio_tagging_loss=0.007457, over 14552.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09332, pruned_loss=0.01402, audio_tagging_loss=0.008961, over 3056967.40 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:06:27,170 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 17:06:53,444 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.8846, 5.7914, 5.6476, 5.5120], device='cuda:2')
2023-11-23 17:06:55,806 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.0005, 5.8629, 5.6292, 5.5490], device='cuda:2')
2023-11-23 17:07:11,623 INFO [train_asr.py:1253] (2/4) Epoch 31, validation: loss=0.05916, simple_loss=0.05108, pruned_loss=0.005166, audio_tagging_loss=0.02845, over 4681554.00 frames. 
2023-11-23 17:07:11,624 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 17:07:12,401 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=15.0
2023-11-23 17:07:17,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2464766.6666666665, ans=0.125
2023-11-23 17:07:23,868 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.61 vs. limit=15.0
2023-11-23 17:07:33,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2464833.3333333335, ans=0.0
2023-11-23 17:07:50,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2464966.6666666665, ans=0.125
2023-11-23 17:07:54,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369750
2023-11-23 17:08:01,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2464966.6666666665, ans=0.125
2023-11-23 17:08:16,240 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9050, loss[loss=0.04885, simple_loss=0.06238, pruned_loss=0.007828, audio_tagging_loss=0.009831, over 15761.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09302, pruned_loss=0.0139, audio_tagging_loss=0.00895, over 3057392.94 frames. ], batch size: 63, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:08:16,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2465100.0, ans=0.0
2023-11-23 17:08:20,600 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.25 vs. limit=22.5
2023-11-23 17:08:21,068 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.342e+01 8.456e+01 8.970e+01 9.727e+01 1.359e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 17:08:26,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2465100.0, ans=0.1
2023-11-23 17:08:31,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2465166.6666666665, ans=0.125
2023-11-23 17:08:45,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2465233.3333333335, ans=0.1
2023-11-23 17:08:54,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2465300.0, ans=0.125
2023-11-23 17:08:58,831 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369800
2023-11-23 17:09:20,909 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9100, loss[loss=0.0744, simple_loss=0.1049, pruned_loss=0.01236, audio_tagging_loss=0.009579, over 14974.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09292, pruned_loss=0.01386, audio_tagging_loss=0.008895, over 3064626.76 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:09:23,932 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.82 vs. limit=15.0
2023-11-23 17:09:35,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2465500.0, ans=0.07
2023-11-23 17:09:53,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2465566.6666666665, ans=0.125
2023-11-23 17:09:58,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2465633.3333333335, ans=0.0
2023-11-23 17:10:03,287 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369850
2023-11-23 17:10:06,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2465633.3333333335, ans=0.125
2023-11-23 17:10:10,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2465633.3333333335, ans=0.125
2023-11-23 17:10:12,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2465700.0, ans=0.2
2023-11-23 17:10:18,513 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-23 17:10:25,948 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9150, loss[loss=0.05702, simple_loss=0.06848, pruned_loss=0.01216, audio_tagging_loss=0.01063, over 15882.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09266, pruned_loss=0.01392, audio_tagging_loss=0.008915, over 3060349.68 frames. ], batch size: 61, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:10:30,941 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.474e+01 8.198e+01 8.578e+01 9.347e+01 1.177e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-23 17:10:53,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2465900.0, ans=0.0
2023-11-23 17:10:55,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2465900.0, ans=0.125
2023-11-23 17:11:08,787 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369900
2023-11-23 17:11:08,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2465966.6666666665, ans=0.2
2023-11-23 17:11:17,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2466033.3333333335, ans=0.125
2023-11-23 17:11:27,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2466033.3333333335, ans=0.125
2023-11-23 17:11:30,960 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9200, loss[loss=0.06577, simple_loss=0.09131, pruned_loss=0.01114, audio_tagging_loss=0.008972, over 15689.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09207, pruned_loss=0.01381, audio_tagging_loss=0.008937, over 3051987.23 frames. ], batch size: 61, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:11:51,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2466166.6666666665, ans=0.0
2023-11-23 17:11:53,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.99 vs. limit=15.0
2023-11-23 17:11:54,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2466166.6666666665, ans=0.0
2023-11-23 17:12:13,762 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 369950
2023-11-23 17:12:29,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2466366.6666666665, ans=0.125
2023-11-23 17:12:35,946 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9250, loss[loss=0.06653, simple_loss=0.0891, pruned_loss=0.01139, audio_tagging_loss=0.01058, over 15261.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09229, pruned_loss=0.01378, audio_tagging_loss=0.008943, over 3058469.09 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:12:40,854 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.301e+01 8.925e+01 9.795e+01 1.226e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 17:12:41,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2466433.3333333335, ans=0.5
2023-11-23 17:13:04,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2466566.6666666665, ans=0.125
2023-11-23 17:13:18,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370000
2023-11-23 17:13:39,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2466700.0, ans=0.125
2023-11-23 17:13:40,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2466766.6666666665, ans=0.125
2023-11-23 17:13:42,040 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9300, loss[loss=0.0527, simple_loss=0.06959, pruned_loss=0.006466, audio_tagging_loss=0.01144, over 14735.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09159, pruned_loss=0.01369, audio_tagging_loss=0.00905, over 3054866.50 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:14:08,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2466900.0, ans=0.125
2023-11-23 17:14:20,559 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.15 vs. limit=22.5
2023-11-23 17:14:25,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370050
2023-11-23 17:14:30,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2466966.6666666665, ans=0.125
2023-11-23 17:14:38,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2467033.3333333335, ans=0.5
2023-11-23 17:14:45,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2467033.3333333335, ans=0.2
2023-11-23 17:14:47,284 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9350, loss[loss=0.05701, simple_loss=0.06769, pruned_loss=0.01043, audio_tagging_loss=0.01273, over 17212.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09233, pruned_loss=0.01374, audio_tagging_loss=0.009167, over 3058204.66 frames. ], batch size: 65, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:14:52,709 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.132e+01 8.275e+01 9.065e+01 9.778e+01 1.188e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 17:15:29,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370100
2023-11-23 17:15:50,780 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9400, loss[loss=0.0746, simple_loss=0.1064, pruned_loss=0.01111, audio_tagging_loss=0.01027, over 16220.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09259, pruned_loss=0.01374, audio_tagging_loss=0.009097, over 3056105.86 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:15:57,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2467433.3333333335, ans=0.125
2023-11-23 17:16:06,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2467500.0, ans=0.0
2023-11-23 17:16:21,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2467566.6666666665, ans=0.125
2023-11-23 17:16:33,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370150
2023-11-23 17:16:39,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2467633.3333333335, ans=0.1
2023-11-23 17:16:41,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2467700.0, ans=0.09899494936611666
2023-11-23 17:16:52,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2467700.0, ans=0.0
2023-11-23 17:16:53,387 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:16:54,586 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9450, loss[loss=0.06903, simple_loss=0.09357, pruned_loss=0.0136, audio_tagging_loss=0.008644, over 14133.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09332, pruned_loss=0.01395, audio_tagging_loss=0.00916, over 3049722.59 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:16:59,340 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.446e+01 8.380e+01 9.034e+01 1.002e+02 1.394e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 17:17:09,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2467833.3333333335, ans=0.0
2023-11-23 17:17:12,502 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:17:19,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2467900.0, ans=0.0
2023-11-23 17:17:36,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370200
2023-11-23 17:17:58,801 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9500, loss[loss=0.0579, simple_loss=0.07597, pruned_loss=0.01065, audio_tagging_loss=0.009264, over 15862.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09314, pruned_loss=0.014, audio_tagging_loss=0.009204, over 3047336.40 frames. ], batch size: 63, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:18:09,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2468100.0, ans=0.0
2023-11-23 17:18:28,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2468233.3333333335, ans=0.0
2023-11-23 17:18:40,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370250
2023-11-23 17:18:43,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.86 vs. limit=12.0
2023-11-23 17:19:01,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2468433.3333333335, ans=0.0
2023-11-23 17:19:02,066 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9550, loss[loss=0.08003, simple_loss=0.1033, pruned_loss=0.02095, audio_tagging_loss=0.007434, over 16004.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09291, pruned_loss=0.0139, audio_tagging_loss=0.009293, over 3043634.32 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:19:07,291 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.559e+01 9.097e+01 9.877e+01 1.537e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-23 17:19:12,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2468433.3333333335, ans=0.125
2023-11-23 17:19:13,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2468500.0, ans=0.2
2023-11-23 17:19:22,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2468500.0, ans=0.05
2023-11-23 17:19:40,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2468633.3333333335, ans=0.2
2023-11-23 17:19:43,672 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370300
2023-11-23 17:19:48,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2468633.3333333335, ans=10.0
2023-11-23 17:20:05,140 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9600, loss[loss=0.06376, simple_loss=0.07978, pruned_loss=0.01171, audio_tagging_loss=0.01216, over 15066.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09166, pruned_loss=0.01372, audio_tagging_loss=0.009432, over 3034265.02 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:20:09,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2468766.6666666665, ans=0.0
2023-11-23 17:20:46,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370350
2023-11-23 17:21:05,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2469100.0, ans=0.5
2023-11-23 17:21:06,828 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9650, loss[loss=0.06778, simple_loss=0.09143, pruned_loss=0.01434, audio_tagging_loss=0.007724, over 15556.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09175, pruned_loss=0.01389, audio_tagging_loss=0.00938, over 3037212.92 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:21:08,731 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0
2023-11-23 17:21:11,539 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.236e+01 8.814e+01 9.557e+01 1.553e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 17:21:22,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2469166.6666666665, ans=0.1
2023-11-23 17:21:42,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2469233.3333333335, ans=0.2
2023-11-23 17:21:44,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2469300.0, ans=0.125
2023-11-23 17:21:48,755 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370400
2023-11-23 17:22:10,462 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9700, loss[loss=0.05718, simple_loss=0.06537, pruned_loss=0.01402, audio_tagging_loss=0.01047, over 15442.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.0911, pruned_loss=0.0137, audio_tagging_loss=0.009296, over 3041508.52 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:22:12,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2469433.3333333335, ans=0.125
2023-11-23 17:22:22,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2469500.0, ans=0.05
2023-11-23 17:22:29,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2469500.0, ans=0.035
2023-11-23 17:22:40,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2469566.6666666665, ans=0.0
2023-11-23 17:22:46,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2469633.3333333335, ans=0.1
2023-11-23 17:22:52,016 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370450
2023-11-23 17:22:55,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2469633.3333333335, ans=0.0
2023-11-23 17:23:13,587 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9750, loss[loss=0.08348, simple_loss=0.1141, pruned_loss=0.01785, audio_tagging_loss=0.008599, over 15220.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09069, pruned_loss=0.01369, audio_tagging_loss=0.009184, over 3032630.35 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:23:20,158 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.276e+01 8.363e+01 8.980e+01 9.912e+01 1.548e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 17:23:29,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2469833.3333333335, ans=0.1
2023-11-23 17:23:32,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2469833.3333333335, ans=10.0
2023-11-23 17:23:43,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.99 vs. limit=22.5
2023-11-23 17:23:44,769 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.28 vs. limit=15.0
2023-11-23 17:23:55,031 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370500
2023-11-23 17:24:16,006 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9800, loss[loss=0.05492, simple_loss=0.0757, pruned_loss=0.008129, audio_tagging_loss=0.008938, over 15749.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09133, pruned_loss=0.0138, audio_tagging_loss=0.009209, over 3039338.92 frames. ], batch size: 61, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:24:30,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2470166.6666666665, ans=0.2
2023-11-23 17:24:30,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2470166.6666666665, ans=0.0
2023-11-23 17:24:51,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2470233.3333333335, ans=0.125
2023-11-23 17:24:57,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370550
2023-11-23 17:24:58,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2470300.0, ans=0.1
2023-11-23 17:25:09,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2470366.6666666665, ans=0.0
2023-11-23 17:25:10,333 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:25:18,005 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9850, loss[loss=0.07465, simple_loss=0.08971, pruned_loss=0.01864, audio_tagging_loss=0.01115, over 15896.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09084, pruned_loss=0.01355, audio_tagging_loss=0.009207, over 3039003.40 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:25:24,367 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.399e+01 8.482e+01 9.119e+01 9.855e+01 1.403e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 17:25:42,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.41 vs. limit=15.0
2023-11-23 17:25:48,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2470566.6666666665, ans=0.1
2023-11-23 17:25:57,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2470633.3333333335, ans=0.2
2023-11-23 17:25:58,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370600
2023-11-23 17:25:58,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2470633.3333333335, ans=0.125
2023-11-23 17:26:08,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2470700.0, ans=0.09899494936611666
2023-11-23 17:26:20,135 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9900, loss[loss=0.05308, simple_loss=0.06681, pruned_loss=0.009114, audio_tagging_loss=0.01057, over 15534.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.0921, pruned_loss=0.01361, audio_tagging_loss=0.009043, over 3048073.09 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:26:23,282 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.21 vs. limit=15.0
2023-11-23 17:26:38,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2470833.3333333335, ans=0.2
2023-11-23 17:26:40,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.59 vs. limit=15.0
2023-11-23 17:26:46,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.84 vs. limit=15.0
2023-11-23 17:27:00,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370650
2023-11-23 17:27:05,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2470966.6666666665, ans=0.125
2023-11-23 17:27:22,246 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 9950, loss[loss=0.06019, simple_loss=0.0832, pruned_loss=0.01115, audio_tagging_loss=0.007433, over 14226.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09186, pruned_loss=0.0136, audio_tagging_loss=0.008992, over 3049558.41 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:27:28,089 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.973e+01 8.205e+01 8.691e+01 9.411e+01 1.200e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-23 17:28:03,487 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370700
2023-11-23 17:28:04,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.48 vs. limit=6.0
2023-11-23 17:28:15,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2471366.6666666665, ans=0.0
2023-11-23 17:28:24,109 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10000, loss[loss=0.07183, simple_loss=0.102, pruned_loss=0.01183, audio_tagging_loss=0.008998, over 14718.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09199, pruned_loss=0.01365, audio_tagging_loss=0.008969, over 3050829.66 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:28:57,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2471566.6666666665, ans=0.0
2023-11-23 17:28:59,264 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:29:04,992 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370750
2023-11-23 17:29:09,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2471633.3333333335, ans=0.0
2023-11-23 17:29:26,663 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10050, loss[loss=0.06413, simple_loss=0.08259, pruned_loss=0.01174, audio_tagging_loss=0.0111, over 15830.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09202, pruned_loss=0.01367, audio_tagging_loss=0.008956, over 3050396.51 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:29:33,087 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.306e+01 9.076e+01 9.554e+01 1.175e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 17:29:33,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2471766.6666666665, ans=0.1
2023-11-23 17:29:36,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2471766.6666666665, ans=0.09899494936611666
2023-11-23 17:30:06,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370800
2023-11-23 17:30:25,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2472033.3333333335, ans=0.0
2023-11-23 17:30:28,774 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10100, loss[loss=0.05401, simple_loss=0.07333, pruned_loss=0.006682, audio_tagging_loss=0.01067, over 15147.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09209, pruned_loss=0.01373, audio_tagging_loss=0.008994, over 3051108.59 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:30:55,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2472233.3333333335, ans=0.2
2023-11-23 17:31:08,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2472300.0, ans=0.125
2023-11-23 17:31:09,979 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370850
2023-11-23 17:31:18,152 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:31:20,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2472366.6666666665, ans=0.125
2023-11-23 17:31:29,904 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10150, loss[loss=0.04772, simple_loss=0.05988, pruned_loss=0.006887, audio_tagging_loss=0.01089, over 15065.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09163, pruned_loss=0.01366, audio_tagging_loss=0.009035, over 3051621.42 frames. ], batch size: 61, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:31:30,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2472433.3333333335, ans=0.1
2023-11-23 17:31:36,348 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.103e+01 8.204e+01 9.033e+01 9.611e+01 1.160e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 17:31:37,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2472433.3333333335, ans=0.0
2023-11-23 17:31:44,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2472500.0, ans=0.2
2023-11-23 17:31:47,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2472500.0, ans=0.125
2023-11-23 17:31:57,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2472566.6666666665, ans=0.125
2023-11-23 17:31:59,351 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:32:05,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2472566.6666666665, ans=0.2
2023-11-23 17:32:11,326 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370900
2023-11-23 17:32:31,947 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10200, loss[loss=0.05986, simple_loss=0.07536, pruned_loss=0.009818, audio_tagging_loss=0.01236, over 14185.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09084, pruned_loss=0.01357, audio_tagging_loss=0.009097, over 3049012.39 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:32:32,766 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:32:55,233 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:33:05,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2472900.0, ans=0.0
2023-11-23 17:33:13,524 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 370950
2023-11-23 17:33:35,228 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10250, loss[loss=0.06103, simple_loss=0.08369, pruned_loss=0.01116, audio_tagging_loss=0.00803, over 15376.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09189, pruned_loss=0.01386, audio_tagging_loss=0.009059, over 3043816.46 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:33:42,407 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.633e+01 9.319e+01 1.005e+02 1.656e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-23 17:33:43,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2473100.0, ans=0.2
2023-11-23 17:33:57,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2473166.6666666665, ans=0.0
2023-11-23 17:34:03,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2473233.3333333335, ans=0.0
2023-11-23 17:34:10,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2473300.0, ans=0.1
2023-11-23 17:34:14,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2473300.0, ans=0.0
2023-11-23 17:34:16,111 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371000
2023-11-23 17:34:31,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0
2023-11-23 17:34:34,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2473366.6666666665, ans=0.125
2023-11-23 17:34:34,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2473366.6666666665, ans=0.125
2023-11-23 17:34:36,430 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10300, loss[loss=0.08346, simple_loss=0.1137, pruned_loss=0.01834, audio_tagging_loss=0.008273, over 14964.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09241, pruned_loss=0.01395, audio_tagging_loss=0.009149, over 3048511.50 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:35:00,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.23 vs. limit=10.0
2023-11-23 17:35:03,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2473566.6666666665, ans=0.015
2023-11-23 17:35:11,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2473566.6666666665, ans=0.2
2023-11-23 17:35:17,966 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371050
2023-11-23 17:35:19,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2473633.3333333335, ans=0.1
2023-11-23 17:35:19,728 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0
2023-11-23 17:35:31,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2473700.0, ans=0.125
2023-11-23 17:35:38,933 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10350, loss[loss=0.08267, simple_loss=0.1108, pruned_loss=0.01988, audio_tagging_loss=0.007392, over 16431.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09362, pruned_loss=0.0142, audio_tagging_loss=0.009198, over 3046844.35 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:35:46,712 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.382e+01 9.122e+01 9.955e+01 1.360e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 17:36:07,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2473900.0, ans=0.2
2023-11-23 17:36:19,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2473966.6666666665, ans=0.0
2023-11-23 17:36:20,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371100
2023-11-23 17:36:26,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2473966.6666666665, ans=0.125
2023-11-23 17:36:28,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2474033.3333333335, ans=0.125
2023-11-23 17:36:33,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2474033.3333333335, ans=0.1
2023-11-23 17:36:35,378 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2474033.3333333335, ans=0.0
2023-11-23 17:36:40,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2474100.0, ans=0.0
2023-11-23 17:36:41,598 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10400, loss[loss=0.07845, simple_loss=0.1091, pruned_loss=0.01586, audio_tagging_loss=0.00807, over 16099.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09283, pruned_loss=0.01394, audio_tagging_loss=0.009327, over 3044492.96 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:36:42,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2474100.0, ans=0.1
2023-11-23 17:37:07,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2474233.3333333335, ans=0.0
2023-11-23 17:37:10,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2474233.3333333335, ans=0.125
2023-11-23 17:37:15,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0
2023-11-23 17:37:18,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2474300.0, ans=0.1
2023-11-23 17:37:22,496 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371150
2023-11-23 17:37:27,183 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.93 vs. limit=6.0
2023-11-23 17:37:41,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2474366.6666666665, ans=0.0
2023-11-23 17:37:43,959 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10450, loss[loss=0.08606, simple_loss=0.1096, pruned_loss=0.02288, audio_tagging_loss=0.008358, over 15949.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09294, pruned_loss=0.01398, audio_tagging_loss=0.009256, over 3047765.55 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:37:51,533 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.218e+01 8.367e+01 9.085e+01 9.738e+01 1.469e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 17:38:13,481 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.69 vs. limit=15.0
2023-11-23 17:38:25,399 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371200
2023-11-23 17:38:46,217 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10500, loss[loss=0.06868, simple_loss=0.09441, pruned_loss=0.01189, audio_tagging_loss=0.009582, over 15494.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09239, pruned_loss=0.01376, audio_tagging_loss=0.00914, over 3053679.84 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:38:53,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2474766.6666666665, ans=0.2
2023-11-23 17:38:53,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2474766.6666666665, ans=0.0
2023-11-23 17:39:02,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2474833.3333333335, ans=0.0
2023-11-23 17:39:27,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371250
2023-11-23 17:39:31,831 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.32 vs. limit=10.0
2023-11-23 17:39:38,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2475033.3333333335, ans=0.125
2023-11-23 17:39:42,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2475033.3333333335, ans=0.04949747468305833
2023-11-23 17:39:48,782 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10550, loss[loss=0.06747, simple_loss=0.09408, pruned_loss=0.01354, audio_tagging_loss=0.006886, over 15275.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09162, pruned_loss=0.01362, audio_tagging_loss=0.009036, over 3055136.10 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:39:55,925 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.253e+01 8.407e+01 8.928e+01 9.553e+01 1.582e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 17:40:07,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2475166.6666666665, ans=0.125
2023-11-23 17:40:09,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.96 vs. limit=12.0
2023-11-23 17:40:22,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2475233.3333333335, ans=0.125
2023-11-23 17:40:29,990 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371300
2023-11-23 17:40:33,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2475300.0, ans=0.0
2023-11-23 17:40:38,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2475366.6666666665, ans=0.0
2023-11-23 17:40:50,995 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10600, loss[loss=0.04486, simple_loss=0.05652, pruned_loss=0.00824, audio_tagging_loss=0.00836, over 14332.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.0917, pruned_loss=0.01361, audio_tagging_loss=0.009012, over 3050176.73 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:40:57,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2475433.3333333335, ans=0.0
2023-11-23 17:41:01,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2475433.3333333335, ans=0.125
2023-11-23 17:41:07,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2475500.0, ans=0.0
2023-11-23 17:41:13,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2475500.0, ans=0.2
2023-11-23 17:41:31,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371350
2023-11-23 17:41:42,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.29 vs. limit=15.0
2023-11-23 17:41:51,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2475700.0, ans=0.05
2023-11-23 17:41:53,219 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10650, loss[loss=0.07504, simple_loss=0.1031, pruned_loss=0.0166, audio_tagging_loss=0.006913, over 15320.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09173, pruned_loss=0.01357, audio_tagging_loss=0.009019, over 3043183.14 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:42:00,379 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.195e+01 8.706e+01 9.399e+01 1.158e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-23 17:42:03,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2475766.6666666665, ans=0.95
2023-11-23 17:42:11,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2475833.3333333335, ans=0.0
2023-11-23 17:42:23,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2475900.0, ans=0.2
2023-11-23 17:42:33,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2475966.6666666665, ans=0.125
2023-11-23 17:42:34,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371400
2023-11-23 17:42:55,575 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10700, loss[loss=0.0815, simple_loss=0.1163, pruned_loss=0.01561, audio_tagging_loss=0.007731, over 16173.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09236, pruned_loss=0.01349, audio_tagging_loss=0.008997, over 3045815.42 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:43:01,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2476100.0, ans=0.125
2023-11-23 17:43:06,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.68 vs. limit=15.0
2023-11-23 17:43:36,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371450
2023-11-23 17:43:47,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2476366.6666666665, ans=0.1
2023-11-23 17:43:56,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2476366.6666666665, ans=0.07
2023-11-23 17:43:58,058 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10750, loss[loss=0.06314, simple_loss=0.09135, pruned_loss=0.009667, audio_tagging_loss=0.007799, over 14752.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09257, pruned_loss=0.01364, audio_tagging_loss=0.00903, over 3051731.19 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:44:02,406 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.00 vs. limit=15.0
2023-11-23 17:44:05,717 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.362e+01 8.891e+01 9.533e+01 1.093e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 17:44:39,558 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.94 vs. limit=10.0
2023-11-23 17:44:39,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371500
2023-11-23 17:44:55,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2476700.0, ans=0.1
2023-11-23 17:45:01,414 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10800, loss[loss=0.06706, simple_loss=0.08791, pruned_loss=0.01283, audio_tagging_loss=0.01027, over 14620.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09343, pruned_loss=0.0138, audio_tagging_loss=0.00886, over 3052441.66 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:45:07,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2476766.6666666665, ans=0.0
2023-11-23 17:45:29,168 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=15.0
2023-11-23 17:45:31,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2476900.0, ans=0.1
2023-11-23 17:45:41,923 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371550
2023-11-23 17:46:03,192 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10850, loss[loss=0.06862, simple_loss=0.08382, pruned_loss=0.01657, audio_tagging_loss=0.01014, over 13967.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09197, pruned_loss=0.01364, audio_tagging_loss=0.009057, over 3048822.78 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:46:03,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2477100.0, ans=0.125
2023-11-23 17:46:14,350 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.439e+01 8.913e+01 9.439e+01 1.225e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 17:46:26,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2477233.3333333335, ans=0.125
2023-11-23 17:46:27,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2477233.3333333335, ans=0.5
2023-11-23 17:46:28,738 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:46:43,565 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371600
2023-11-23 17:47:00,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2477366.6666666665, ans=0.1
2023-11-23 17:47:01,411 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:47:04,939 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10900, loss[loss=0.05948, simple_loss=0.0881, pruned_loss=0.008037, audio_tagging_loss=0.007392, over 13884.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09273, pruned_loss=0.01369, audio_tagging_loss=0.008987, over 3051074.83 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:47:22,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2477500.0, ans=0.1
2023-11-23 17:47:29,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2477566.6666666665, ans=0.1
2023-11-23 17:47:38,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.85 vs. limit=15.0
2023-11-23 17:47:46,036 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371650
2023-11-23 17:47:54,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2477700.0, ans=0.025
2023-11-23 17:48:00,570 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.25 vs. limit=15.0
2023-11-23 17:48:06,478 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 10950, loss[loss=0.07063, simple_loss=0.1006, pruned_loss=0.01329, audio_tagging_loss=0.007027, over 15377.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09331, pruned_loss=0.01387, audio_tagging_loss=0.008981, over 3056508.40 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:48:07,177 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.56 vs. limit=15.0
2023-11-23 17:48:17,609 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.169e+01 8.798e+01 9.612e+01 4.443e+02, threshold=1.760e+02, percent-clipped=1.0
2023-11-23 17:48:21,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0
2023-11-23 17:48:47,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371700
2023-11-23 17:48:56,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2478033.3333333335, ans=0.2
2023-11-23 17:49:02,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=15.0
2023-11-23 17:49:08,788 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11000, loss[loss=0.06385, simple_loss=0.08556, pruned_loss=0.01386, audio_tagging_loss=0.007204, over 14246.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09319, pruned_loss=0.01384, audio_tagging_loss=0.009082, over 3055303.64 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:49:13,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2478100.0, ans=0.125
2023-11-23 17:49:18,879 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:49:42,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2478233.3333333335, ans=0.125
2023-11-23 17:49:49,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371750
2023-11-23 17:49:51,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.91 vs. limit=15.0
2023-11-23 17:49:52,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2478300.0, ans=0.2
2023-11-23 17:50:05,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.20 vs. limit=15.0
2023-11-23 17:50:06,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.33 vs. limit=12.0
2023-11-23 17:50:11,123 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11050, loss[loss=0.06649, simple_loss=0.09033, pruned_loss=0.0123, audio_tagging_loss=0.00903, over 15500.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09325, pruned_loss=0.01374, audio_tagging_loss=0.009242, over 3051062.87 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:50:21,702 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.456e+01 8.354e+01 9.034e+01 9.901e+01 1.113e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 17:50:45,998 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:50:52,545 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371800
2023-11-23 17:50:53,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2478633.3333333335, ans=0.125
2023-11-23 17:51:10,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2478700.0, ans=0.0
2023-11-23 17:51:13,274 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11100, loss[loss=0.0572, simple_loss=0.06733, pruned_loss=0.01406, audio_tagging_loss=0.009478, over 13713.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09248, pruned_loss=0.01363, audio_tagging_loss=0.009305, over 3049974.11 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:51:42,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2478900.0, ans=0.125
2023-11-23 17:51:54,205 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371850
2023-11-23 17:52:04,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2479033.3333333335, ans=0.0
2023-11-23 17:52:15,284 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11150, loss[loss=0.06531, simple_loss=0.08399, pruned_loss=0.01128, audio_tagging_loss=0.01204, over 14996.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09222, pruned_loss=0.01356, audio_tagging_loss=0.009387, over 3042763.42 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:52:26,735 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 8.417e+01 9.306e+01 9.835e+01 1.557e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-23 17:52:28,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2479166.6666666665, ans=0.025
2023-11-23 17:52:29,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.63 vs. limit=15.0
2023-11-23 17:52:37,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2479166.6666666665, ans=0.125
2023-11-23 17:52:41,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2479233.3333333335, ans=0.0
2023-11-23 17:52:55,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371900
2023-11-23 17:53:00,428 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:53:05,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2479366.6666666665, ans=0.0
2023-11-23 17:53:17,766 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11200, loss[loss=0.0695, simple_loss=0.09004, pruned_loss=0.01449, audio_tagging_loss=0.009994, over 14430.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.0918, pruned_loss=0.01352, audio_tagging_loss=0.009415, over 3043785.22 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:53:58,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 371950
2023-11-23 17:54:10,045 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=15.0
2023-11-23 17:54:13,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2479700.0, ans=0.0
2023-11-23 17:54:16,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2479700.0, ans=0.04949747468305833
2023-11-23 17:54:19,189 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11250, loss[loss=0.07925, simple_loss=0.1065, pruned_loss=0.0202, audio_tagging_loss=0.005792, over 15259.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09214, pruned_loss=0.01369, audio_tagging_loss=0.009449, over 3035343.38 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 17:54:19,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2479766.6666666665, ans=0.07
2023-11-23 17:54:30,510 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.457e+01 8.293e+01 8.818e+01 9.873e+01 1.266e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 17:54:39,094 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2479833.3333333335, ans=0.125
2023-11-23 17:54:42,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.77 vs. limit=15.0
2023-11-23 17:54:46,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2479900.0, ans=0.2
2023-11-23 17:54:48,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2479900.0, ans=0.5
2023-11-23 17:55:00,513 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372000
2023-11-23 17:55:02,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.48 vs. limit=15.0
2023-11-23 17:55:20,230 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.52 vs. limit=15.0
2023-11-23 17:55:24,133 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11300, loss[loss=0.06324, simple_loss=0.08292, pruned_loss=0.01362, audio_tagging_loss=0.008156, over 16128.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09191, pruned_loss=0.01371, audio_tagging_loss=0.009281, over 3036210.02 frames. ], batch size: 62, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:55:36,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-23 17:55:39,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2480166.6666666665, ans=0.0
2023-11-23 17:56:05,833 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372050
2023-11-23 17:56:21,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2480366.6666666665, ans=0.0
2023-11-23 17:56:22,877 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.91 vs. limit=10.0
2023-11-23 17:56:27,521 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11350, loss[loss=0.07025, simple_loss=0.09335, pruned_loss=0.01554, audio_tagging_loss=0.008038, over 16054.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09198, pruned_loss=0.01369, audio_tagging_loss=0.009173, over 3042365.91 frames. ], batch size: 62, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:56:28,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2480433.3333333335, ans=0.0
2023-11-23 17:56:39,398 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.419e+01 8.973e+01 9.627e+01 1.152e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 17:56:52,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2480566.6666666665, ans=0.0
2023-11-23 17:56:56,816 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.01 vs. limit=22.5
2023-11-23 17:57:08,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372100
2023-11-23 17:57:28,755 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11400, loss[loss=0.1033, simple_loss=0.1456, pruned_loss=0.02364, audio_tagging_loss=0.006874, over 15951.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09262, pruned_loss=0.01386, audio_tagging_loss=0.009061, over 3045194.63 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:57:53,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2480900.0, ans=0.125
2023-11-23 17:58:03,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2480900.0, ans=0.125
2023-11-23 17:58:07,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2480966.6666666665, ans=0.0
2023-11-23 17:58:10,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372150
2023-11-23 17:58:31,095 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11450, loss[loss=0.07185, simple_loss=0.1001, pruned_loss=0.01144, audio_tagging_loss=0.01034, over 16349.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09217, pruned_loss=0.0137, audio_tagging_loss=0.009063, over 3049477.53 frames. ], batch size: 59, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:58:37,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2481100.0, ans=0.0
2023-11-23 17:58:43,912 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.130e+01 8.677e+01 9.441e+01 1.245e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 17:58:49,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2481166.6666666665, ans=0.125
2023-11-23 17:59:03,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2481233.3333333335, ans=0.125
2023-11-23 17:59:04,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2481233.3333333335, ans=0.0
2023-11-23 17:59:12,555 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372200
2023-11-23 17:59:17,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.76 vs. limit=15.0
2023-11-23 17:59:17,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2481300.0, ans=0.125
2023-11-23 17:59:20,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2481366.6666666665, ans=0.0
2023-11-23 17:59:34,005 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11500, loss[loss=0.06937, simple_loss=0.09128, pruned_loss=0.01278, audio_tagging_loss=0.01095, over 15022.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09211, pruned_loss=0.01373, audio_tagging_loss=0.009007, over 3048537.38 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:59:37,844 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.40 vs. limit=8.0
2023-11-23 17:59:56,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2481566.6666666665, ans=0.125
2023-11-23 18:00:06,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2481566.6666666665, ans=0.0
2023-11-23 18:00:09,521 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.08 vs. limit=15.0
2023-11-23 18:00:14,402 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372250
2023-11-23 18:00:35,653 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11550, loss[loss=0.05464, simple_loss=0.06992, pruned_loss=0.01001, audio_tagging_loss=0.009674, over 15830.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09312, pruned_loss=0.01375, audio_tagging_loss=0.008934, over 3052932.31 frames. ], batch size: 60, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 18:00:48,167 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.328e+01 8.420e+01 9.006e+01 9.672e+01 1.345e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 18:01:10,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2481900.0, ans=0.0
2023-11-23 18:01:11,900 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.78 vs. limit=15.0
2023-11-23 18:01:13,601 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:01:17,176 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372300
2023-11-23 18:01:34,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2482033.3333333335, ans=0.05
2023-11-23 18:01:36,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2482100.0, ans=0.125
2023-11-23 18:01:37,298 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11600, loss[loss=0.07014, simple_loss=0.09754, pruned_loss=0.01393, audio_tagging_loss=0.007444, over 15091.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09257, pruned_loss=0.01371, audio_tagging_loss=0.008975, over 3058572.58 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:01:37,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2482100.0, ans=0.2
2023-11-23 18:01:40,243 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.18 vs. limit=22.5
2023-11-23 18:01:43,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2482100.0, ans=0.125
2023-11-23 18:02:07,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=2482233.3333333335, ans=0.02
2023-11-23 18:02:10,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2482233.3333333335, ans=0.2
2023-11-23 18:02:17,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372350
2023-11-23 18:02:22,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2482300.0, ans=0.125
2023-11-23 18:02:22,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2482300.0, ans=0.2
2023-11-23 18:02:38,547 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11650, loss[loss=0.09591, simple_loss=0.1214, pruned_loss=0.02669, audio_tagging_loss=0.008499, over 16141.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09283, pruned_loss=0.01388, audio_tagging_loss=0.008954, over 3048732.39 frames. ], batch size: 60, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:02:50,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.302e+01 8.966e+01 9.575e+01 1.248e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 18:02:56,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2482500.0, ans=0.2
2023-11-23 18:03:01,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2482566.6666666665, ans=0.125
2023-11-23 18:03:08,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2482566.6666666665, ans=0.125
2023-11-23 18:03:10,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2482566.6666666665, ans=0.05
2023-11-23 18:03:10,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2482566.6666666665, ans=0.125
2023-11-23 18:03:15,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2482633.3333333335, ans=0.125
2023-11-23 18:03:18,828 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372400
2023-11-23 18:03:36,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2482700.0, ans=0.125
2023-11-23 18:03:40,966 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11700, loss[loss=0.04992, simple_loss=0.06419, pruned_loss=0.007962, audio_tagging_loss=0.009856, over 15032.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09139, pruned_loss=0.01354, audio_tagging_loss=0.009035, over 3042644.12 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:03:47,414 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.01 vs. limit=15.0
2023-11-23 18:04:22,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372450
2023-11-23 18:04:28,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2482966.6666666665, ans=0.0
2023-11-23 18:04:31,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2483033.3333333335, ans=0.2
2023-11-23 18:04:42,735 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11750, loss[loss=0.05958, simple_loss=0.07458, pruned_loss=0.01202, audio_tagging_loss=0.01027, over 13847.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09209, pruned_loss=0.01377, audio_tagging_loss=0.009128, over 3047720.64 frames. ], batch size: 55, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:04:54,927 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.352e+01 9.061e+01 9.652e+01 1.174e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 18:05:05,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2483166.6666666665, ans=0.125
2023-11-23 18:05:23,275 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372500
2023-11-23 18:05:27,856 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.78 vs. limit=15.0
2023-11-23 18:05:44,254 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11800, loss[loss=0.08315, simple_loss=0.118, pruned_loss=0.01494, audio_tagging_loss=0.009206, over 16641.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09201, pruned_loss=0.01395, audio_tagging_loss=0.009159, over 3041283.41 frames. ], batch size: 59, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:06:01,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2483500.0, ans=0.0
2023-11-23 18:06:15,215 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.52 vs. limit=15.0
2023-11-23 18:06:22,718 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.75 vs. limit=15.0
2023-11-23 18:06:25,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372550
2023-11-23 18:06:46,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2483766.6666666665, ans=0.125
2023-11-23 18:06:46,834 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11850, loss[loss=0.06836, simple_loss=0.09397, pruned_loss=0.0129, audio_tagging_loss=0.008478, over 15746.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09237, pruned_loss=0.014, audio_tagging_loss=0.009259, over 3038463.91 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:06:56,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2483766.6666666665, ans=0.0
2023-11-23 18:06:59,891 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.390e+01 9.057e+01 9.896e+01 1.267e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 18:07:04,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2483833.3333333335, ans=0.125
2023-11-23 18:07:06,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2483833.3333333335, ans=0.07
2023-11-23 18:07:18,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2483900.0, ans=0.1
2023-11-23 18:07:28,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372600
2023-11-23 18:07:43,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.41 vs. limit=15.0
2023-11-23 18:07:44,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2484033.3333333335, ans=0.2
2023-11-23 18:07:49,512 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11900, loss[loss=0.07723, simple_loss=0.11, pruned_loss=0.01413, audio_tagging_loss=0.008109, over 15283.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09218, pruned_loss=0.0139, audio_tagging_loss=0.009339, over 3037626.24 frames. ], batch size: 53, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:07:56,270 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.61 vs. limit=15.0
2023-11-23 18:07:56,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2484100.0, ans=0.1
2023-11-23 18:08:01,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2484166.6666666665, ans=0.125
2023-11-23 18:08:30,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372650
2023-11-23 18:08:30,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2484300.0, ans=0.125
2023-11-23 18:08:51,639 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 11950, loss[loss=0.06691, simple_loss=0.08618, pruned_loss=0.0138, audio_tagging_loss=0.01003, over 15034.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09191, pruned_loss=0.01388, audio_tagging_loss=0.009421, over 3037018.92 frames. ], batch size: 58, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:08:54,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2484433.3333333335, ans=0.0
2023-11-23 18:09:04,021 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.358e+01 9.149e+01 9.907e+01 1.513e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 18:09:12,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2484500.0, ans=0.1
2023-11-23 18:09:15,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.41 vs. limit=15.0
2023-11-23 18:09:22,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2484566.6666666665, ans=0.125
2023-11-23 18:09:22,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2484566.6666666665, ans=0.1
2023-11-23 18:09:24,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2484566.6666666665, ans=0.05
2023-11-23 18:09:27,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2484633.3333333335, ans=0.125
2023-11-23 18:09:31,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372700
2023-11-23 18:09:44,476 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.13 vs. limit=15.0
2023-11-23 18:09:50,724 INFO [train_asr.py:1221] (2/4) Epoch 31, batch 12000, loss[loss=0.06421, simple_loss=0.07437, pruned_loss=0.01337, audio_tagging_loss=0.01365, over 15214.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09157, pruned_loss=0.01373, audio_tagging_loss=0.009583, over 3035872.62 frames. ], batch size: 59, lr: 2.17e-03, grad_scale: 32.0
2023-11-23 18:09:50,725 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 18:10:15,636 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([6.4523, 6.1569, 6.4160, 5.9623], device='cuda:2')
2023-11-23 18:10:31,555 INFO [train_asr.py:1253] (2/4) Epoch 31, validation: loss=0.05782, simple_loss=0.05111, pruned_loss=0.00519, audio_tagging_loss=0.02708, over 4681554.00 frames. 
2023-11-23 18:10:31,556 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 18:10:31,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2484766.6666666665, ans=0.125
2023-11-23 18:10:36,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2484766.6666666665, ans=0.125
2023-11-23 18:10:36,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2484766.6666666665, ans=0.125
2023-11-23 18:10:40,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2484766.6666666665, ans=0.125
2023-11-23 18:10:48,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2484833.3333333335, ans=0.1
2023-11-23 18:11:32,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2484926.6666666665, ans=0.1
2023-11-23 18:11:33,926 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 0, loss[loss=0.09288, simple_loss=0.1116, pruned_loss=0.01903, audio_tagging_loss=0.01804, over 15325.00 frames. ], tot_loss[loss=0.09288, simple_loss=0.1116, pruned_loss=0.01903, audio_tagging_loss=0.01804, over 15325.00 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:11:33,927 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 18:11:59,746 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.2732, 2.9279, 3.1613, 2.9586, 3.7076, 3.7620, 3.2675, 3.1410],
       device='cuda:2')
2023-11-23 18:12:09,590 INFO [train_asr.py:1253] (2/4) Epoch 32, validation: loss=0.05806, simple_loss=0.0511, pruned_loss=0.005184, audio_tagging_loss=0.02732, over 4681554.00 frames. 
2023-11-23 18:12:09,591 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 18:12:20,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372750
2023-11-23 18:12:25,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2484993.3333333335, ans=0.04949747468305833
2023-11-23 18:12:35,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2485060.0, ans=0.0
2023-11-23 18:12:53,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2485126.6666666665, ans=0.1
2023-11-23 18:12:54,221 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.631e+01 9.416e+01 1.070e+02 1.612e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-23 18:13:00,239 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.20 vs. limit=12.0
2023-11-23 18:13:08,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2485193.3333333335, ans=0.1
2023-11-23 18:13:11,372 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 50, loss[loss=0.07222, simple_loss=0.09547, pruned_loss=0.008983, audio_tagging_loss=0.0155, over 14794.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09228, pruned_loss=0.01412, audio_tagging_loss=0.01712, over 690590.62 frames. ], batch size: 55, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:13:15,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2485260.0, ans=0.2
2023-11-23 18:13:22,507 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372800
2023-11-23 18:13:30,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=2485326.6666666665, ans=0.2
2023-11-23 18:13:37,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2485393.3333333335, ans=0.0
2023-11-23 18:14:11,961 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.37 vs. limit=15.0
2023-11-23 18:14:13,859 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 100, loss[loss=0.07173, simple_loss=0.0889, pruned_loss=0.01352, audio_tagging_loss=0.01376, over 14426.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.0919, pruned_loss=0.01372, audio_tagging_loss=0.01641, over 1206534.80 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:14:25,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372850
2023-11-23 18:14:27,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2485660.0, ans=0.125
2023-11-23 18:14:28,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2485660.0, ans=0.125
2023-11-23 18:14:59,365 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.698e+01 8.892e+01 9.428e+01 1.026e+02 1.429e+02, threshold=1.886e+02, percent-clipped=0.0
2023-11-23 18:15:00,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2485793.3333333335, ans=0.0
2023-11-23 18:15:01,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2485793.3333333335, ans=0.0
2023-11-23 18:15:17,195 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 150, loss[loss=0.05554, simple_loss=0.05909, pruned_loss=0.01102, audio_tagging_loss=0.01498, over 15076.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09046, pruned_loss=0.01324, audio_tagging_loss=0.01492, over 1617284.99 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:15:22,690 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:15:28,445 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372900
2023-11-23 18:15:37,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2485993.3333333335, ans=0.04949747468305833
2023-11-23 18:15:42,660 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.05 vs. limit=15.0
2023-11-23 18:15:56,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2486126.6666666665, ans=0.125
2023-11-23 18:15:59,655 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.50 vs. limit=12.0
2023-11-23 18:16:06,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2486193.3333333335, ans=0.125
2023-11-23 18:16:18,931 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 200, loss[loss=0.05761, simple_loss=0.07386, pruned_loss=0.009858, audio_tagging_loss=0.01082, over 14892.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09103, pruned_loss=0.01337, audio_tagging_loss=0.01319, over 1937741.48 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:16:24,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2486260.0, ans=0.1
2023-11-23 18:16:26,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2486260.0, ans=0.07
2023-11-23 18:16:30,260 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 372950
2023-11-23 18:16:48,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2486393.3333333335, ans=0.0
2023-11-23 18:16:49,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2486393.3333333335, ans=0.05
2023-11-23 18:16:51,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.14 vs. limit=22.5
2023-11-23 18:17:04,694 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.397e+01 8.534e+01 9.090e+01 9.831e+01 1.209e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 18:17:07,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2486526.6666666665, ans=0.125
2023-11-23 18:17:20,650 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 250, loss[loss=0.05864, simple_loss=0.08077, pruned_loss=0.009344, audio_tagging_loss=0.008906, over 14870.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09081, pruned_loss=0.01324, audio_tagging_loss=0.01196, over 2183434.44 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:17:31,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373000
2023-11-23 18:17:38,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2486660.0, ans=0.2
2023-11-23 18:18:12,814 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:18:16,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2486860.0, ans=0.125
2023-11-23 18:18:22,653 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 300, loss[loss=0.07638, simple_loss=0.1006, pruned_loss=0.0175, audio_tagging_loss=0.008558, over 14249.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09112, pruned_loss=0.01358, audio_tagging_loss=0.01105, over 2375789.39 frames. ], batch size: 55, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:18:34,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373050
2023-11-23 18:18:52,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2487060.0, ans=0.1
2023-11-23 18:19:03,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2487126.6666666665, ans=0.125
2023-11-23 18:19:08,696 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.767e+01 9.315e+01 9.987e+01 1.181e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-23 18:19:24,681 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 350, loss[loss=0.08438, simple_loss=0.1156, pruned_loss=0.01741, audio_tagging_loss=0.009181, over 15471.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.0916, pruned_loss=0.01381, audio_tagging_loss=0.01055, over 2520419.26 frames. ], batch size: 55, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:19:36,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373100
2023-11-23 18:19:41,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2487326.6666666665, ans=0.125
2023-11-23 18:20:10,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2487460.0, ans=0.0
2023-11-23 18:20:20,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.77 vs. limit=10.0
2023-11-23 18:20:21,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2487526.6666666665, ans=0.125
2023-11-23 18:20:26,827 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 400, loss[loss=0.05783, simple_loss=0.07351, pruned_loss=0.01083, audio_tagging_loss=0.01024, over 16327.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09168, pruned_loss=0.01376, audio_tagging_loss=0.01019, over 2637608.76 frames. ], batch size: 62, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:20:32,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.20 vs. limit=15.0
2023-11-23 18:20:37,959 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373150
2023-11-23 18:20:50,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2487726.6666666665, ans=0.09899494936611666
2023-11-23 18:20:57,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2487726.6666666665, ans=0.125
2023-11-23 18:21:00,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2487726.6666666665, ans=0.2
2023-11-23 18:21:12,520 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.510e+01 9.091e+01 9.716e+01 1.253e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 18:21:19,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2487860.0, ans=0.1
2023-11-23 18:21:23,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2487860.0, ans=0.0
2023-11-23 18:21:28,544 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 450, loss[loss=0.05048, simple_loss=0.06202, pruned_loss=0.007853, audio_tagging_loss=0.01162, over 14566.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09112, pruned_loss=0.01362, audio_tagging_loss=0.009981, over 2723994.91 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:21:33,998 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.82 vs. limit=6.0
2023-11-23 18:21:39,991 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373200
2023-11-23 18:22:00,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2488060.0, ans=0.0
2023-11-23 18:22:15,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2488126.6666666665, ans=0.1
2023-11-23 18:22:15,767 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.97 vs. limit=15.0
2023-11-23 18:22:16,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2488126.6666666665, ans=0.1
2023-11-23 18:22:26,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2488193.3333333335, ans=0.2
2023-11-23 18:22:31,179 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 500, loss[loss=0.08335, simple_loss=0.1178, pruned_loss=0.01593, audio_tagging_loss=0.008505, over 16052.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09159, pruned_loss=0.01372, audio_tagging_loss=0.009709, over 2790979.20 frames. ], batch size: 59, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:22:42,950 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373250
2023-11-23 18:22:47,081 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0
2023-11-23 18:23:00,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2488393.3333333335, ans=0.125
2023-11-23 18:23:17,404 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.516e+01 9.137e+01 9.960e+01 1.270e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 18:23:31,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.65 vs. limit=22.5
2023-11-23 18:23:34,140 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 550, loss[loss=0.09464, simple_loss=0.1313, pruned_loss=0.02201, audio_tagging_loss=0.006973, over 14330.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09314, pruned_loss=0.01395, audio_tagging_loss=0.009519, over 2846768.73 frames. ], batch size: 55, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:23:44,836 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373300
2023-11-23 18:24:04,018 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:24:12,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2488793.3333333335, ans=0.125
2023-11-23 18:24:14,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2488793.3333333335, ans=0.125
2023-11-23 18:24:17,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2488793.3333333335, ans=0.125
2023-11-23 18:24:25,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2488860.0, ans=0.125
2023-11-23 18:24:36,007 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 600, loss[loss=0.05077, simple_loss=0.06425, pruned_loss=0.008807, audio_tagging_loss=0.009834, over 14931.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09233, pruned_loss=0.01376, audio_tagging_loss=0.009323, over 2894313.47 frames. ], batch size: 59, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:24:47,246 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373350
2023-11-23 18:24:48,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.46 vs. limit=15.0
2023-11-23 18:25:01,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2489060.0, ans=0.125
2023-11-23 18:25:23,228 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.713e+01 9.332e+01 1.014e+02 1.298e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-23 18:25:38,128 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 650, loss[loss=0.07584, simple_loss=0.1065, pruned_loss=0.01498, audio_tagging_loss=0.0076, over 15774.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09157, pruned_loss=0.01363, audio_tagging_loss=0.009382, over 2929856.64 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:25:49,544 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373400
2023-11-23 18:25:51,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2489326.6666666665, ans=0.0
2023-11-23 18:26:03,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2489393.3333333335, ans=0.0
2023-11-23 18:26:07,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2489393.3333333335, ans=0.1
2023-11-23 18:26:36,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2489526.6666666665, ans=0.125
2023-11-23 18:26:41,065 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 700, loss[loss=0.06076, simple_loss=0.08263, pruned_loss=0.01129, audio_tagging_loss=0.008153, over 16060.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.0914, pruned_loss=0.01365, audio_tagging_loss=0.00928, over 2956480.74 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:26:41,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2489593.3333333335, ans=0.1
2023-11-23 18:26:51,886 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373450
2023-11-23 18:26:53,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2489660.0, ans=0.125
2023-11-23 18:27:02,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2489660.0, ans=0.125
2023-11-23 18:27:03,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.89 vs. limit=15.0
2023-11-23 18:27:09,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2489726.6666666665, ans=0.1
2023-11-23 18:27:16,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2489793.3333333335, ans=0.0
2023-11-23 18:27:18,158 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.02 vs. limit=10.0
2023-11-23 18:27:22,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2489793.3333333335, ans=0.125
2023-11-23 18:27:27,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.06 vs. limit=10.0
2023-11-23 18:27:29,304 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.089e+01 8.956e+01 9.594e+01 1.307e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 18:27:42,825 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 750, loss[loss=0.07597, simple_loss=0.1101, pruned_loss=0.01447, audio_tagging_loss=0.006437, over 17062.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09171, pruned_loss=0.01356, audio_tagging_loss=0.009206, over 2981625.59 frames. ], batch size: 62, lr: 2.14e-03, grad_scale: 8.0
2023-11-23 18:27:53,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373500
2023-11-23 18:28:14,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.78 vs. limit=22.5
2023-11-23 18:28:16,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.05 vs. limit=22.5
2023-11-23 18:28:17,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2490060.0, ans=0.1
2023-11-23 18:28:33,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2490193.3333333335, ans=10.0
2023-11-23 18:28:44,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2490260.0, ans=0.1
2023-11-23 18:28:45,048 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 800, loss[loss=0.06678, simple_loss=0.08117, pruned_loss=0.01307, audio_tagging_loss=0.01312, over 16301.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09107, pruned_loss=0.01364, audio_tagging_loss=0.009278, over 3002421.93 frames. ], batch size: 63, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:28:46,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2490260.0, ans=0.125
2023-11-23 18:28:55,836 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373550
2023-11-23 18:29:07,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2490326.6666666665, ans=0.125
2023-11-23 18:29:11,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2490393.3333333335, ans=0.0
2023-11-23 18:29:16,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2490393.3333333335, ans=0.1
2023-11-23 18:29:17,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2490393.3333333335, ans=0.125
2023-11-23 18:29:33,492 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.422e+01 8.967e+01 9.780e+01 1.208e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 18:29:39,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2490526.6666666665, ans=0.0
2023-11-23 18:29:45,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2490593.3333333335, ans=0.0
2023-11-23 18:29:46,995 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 850, loss[loss=0.07981, simple_loss=0.1157, pruned_loss=0.01569, audio_tagging_loss=0.006274, over 15001.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.0922, pruned_loss=0.01384, audio_tagging_loss=0.009321, over 3011232.74 frames. ], batch size: 53, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:29:50,014 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.24 vs. limit=22.5
2023-11-23 18:29:58,474 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373600
2023-11-23 18:29:58,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2490660.0, ans=0.0
2023-11-23 18:30:02,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2490660.0, ans=0.125
2023-11-23 18:30:12,917 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.27 vs. limit=15.0
2023-11-23 18:30:27,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2490793.3333333335, ans=0.0
2023-11-23 18:30:38,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2490860.0, ans=0.125
2023-11-23 18:30:49,685 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 900, loss[loss=0.06346, simple_loss=0.08335, pruned_loss=0.01199, audio_tagging_loss=0.009799, over 14964.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09233, pruned_loss=0.01391, audio_tagging_loss=0.0093, over 3018382.51 frames. ], batch size: 56, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:31:01,127 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373650
2023-11-23 18:31:11,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2490993.3333333335, ans=0.1
2023-11-23 18:31:35,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2491126.6666666665, ans=0.125
2023-11-23 18:31:35,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2491126.6666666665, ans=0.5
2023-11-23 18:31:37,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2491126.6666666665, ans=0.125
2023-11-23 18:31:38,018 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.169e+01 8.291e+01 8.989e+01 9.605e+01 1.191e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 18:31:40,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2491193.3333333335, ans=0.0
2023-11-23 18:31:45,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.52 vs. limit=15.0
2023-11-23 18:31:45,431 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.35 vs. limit=15.0
2023-11-23 18:31:51,782 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 950, loss[loss=0.07944, simple_loss=0.1105, pruned_loss=0.01567, audio_tagging_loss=0.00853, over 15526.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09281, pruned_loss=0.01387, audio_tagging_loss=0.009207, over 3029544.88 frames. ], batch size: 55, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:31:51,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2491260.0, ans=0.0
2023-11-23 18:32:01,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2491260.0, ans=0.125
2023-11-23 18:32:03,022 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373700
2023-11-23 18:32:05,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2491326.6666666665, ans=0.1
2023-11-23 18:32:20,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2491393.3333333335, ans=0.0
2023-11-23 18:32:44,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2491526.6666666665, ans=0.1
2023-11-23 18:32:48,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2491526.6666666665, ans=0.09899494936611666
2023-11-23 18:32:53,756 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1000, loss[loss=0.07904, simple_loss=0.1014, pruned_loss=0.0191, audio_tagging_loss=0.009264, over 14334.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09139, pruned_loss=0.01353, audio_tagging_loss=0.009215, over 3031720.82 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:32:59,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2491593.3333333335, ans=0.2
2023-11-23 18:33:00,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2491593.3333333335, ans=0.125
2023-11-23 18:33:00,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2491593.3333333335, ans=0.1
2023-11-23 18:33:05,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373750
2023-11-23 18:33:06,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2491660.0, ans=0.0
2023-11-23 18:33:09,680 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:33:20,535 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:33:21,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2491726.6666666665, ans=0.1
2023-11-23 18:33:27,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.55 vs. limit=22.5
2023-11-23 18:33:32,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2491793.3333333335, ans=0.1
2023-11-23 18:33:42,320 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.226e+01 8.316e+01 8.954e+01 9.512e+01 1.152e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 18:33:42,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2491860.0, ans=0.1
2023-11-23 18:33:49,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2491860.0, ans=0.125
2023-11-23 18:33:55,919 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1050, loss[loss=0.06913, simple_loss=0.086, pruned_loss=0.01532, audio_tagging_loss=0.01082, over 17144.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09154, pruned_loss=0.01375, audio_tagging_loss=0.009088, over 3035048.72 frames. ], batch size: 65, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:34:06,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2491926.6666666665, ans=0.125
2023-11-23 18:34:07,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373800
2023-11-23 18:34:07,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2491993.3333333335, ans=0.0
2023-11-23 18:34:35,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2492126.6666666665, ans=0.0
2023-11-23 18:34:38,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.15 vs. limit=15.0
2023-11-23 18:34:59,571 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1100, loss[loss=0.06063, simple_loss=0.07791, pruned_loss=0.01116, audio_tagging_loss=0.01052, over 13955.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09207, pruned_loss=0.01384, audio_tagging_loss=0.008938, over 3033420.61 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:35:02,525 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:35:04,287 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.66 vs. limit=15.0
2023-11-23 18:35:10,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373850
2023-11-23 18:35:12,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2492326.6666666665, ans=0.0
2023-11-23 18:35:29,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2492393.3333333335, ans=0.125
2023-11-23 18:35:40,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2492460.0, ans=0.0
2023-11-23 18:35:48,096 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.110e+01 8.335e+01 9.013e+01 9.723e+01 1.742e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 18:35:48,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2492526.6666666665, ans=0.0
2023-11-23 18:35:57,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2492526.6666666665, ans=0.125
2023-11-23 18:35:57,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2492526.6666666665, ans=0.125
2023-11-23 18:36:02,046 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1150, loss[loss=0.06839, simple_loss=0.08912, pruned_loss=0.01503, audio_tagging_loss=0.008795, over 14400.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09179, pruned_loss=0.01369, audio_tagging_loss=0.008937, over 3036236.20 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:36:13,575 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373900
2023-11-23 18:36:33,292 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:36:51,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2492860.0, ans=0.0
2023-11-23 18:36:54,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2492860.0, ans=0.0
2023-11-23 18:36:56,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2492860.0, ans=0.125
2023-11-23 18:37:04,181 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1200, loss[loss=0.06789, simple_loss=0.08791, pruned_loss=0.0132, audio_tagging_loss=0.01074, over 16814.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09163, pruned_loss=0.01363, audio_tagging_loss=0.008941, over 3035346.40 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:37:16,253 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 373950
2023-11-23 18:37:52,690 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.465e+01 8.983e+01 9.576e+01 1.363e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 18:37:53,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2493193.3333333335, ans=0.1
2023-11-23 18:38:06,966 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1250, loss[loss=0.07478, simple_loss=0.09289, pruned_loss=0.01798, audio_tagging_loss=0.01035, over 16117.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09076, pruned_loss=0.01339, audio_tagging_loss=0.009018, over 3043539.69 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:38:15,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.15 vs. limit=15.0
2023-11-23 18:38:18,456 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374000
2023-11-23 18:38:21,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2493326.6666666665, ans=0.125
2023-11-23 18:38:28,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2493326.6666666665, ans=0.125
2023-11-23 18:38:51,226 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.52 vs. limit=15.0
2023-11-23 18:39:06,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.69 vs. limit=6.0
2023-11-23 18:39:08,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2493593.3333333335, ans=0.125
2023-11-23 18:39:09,326 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1300, loss[loss=0.06125, simple_loss=0.08111, pruned_loss=0.01149, audio_tagging_loss=0.009197, over 15045.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.0908, pruned_loss=0.0135, audio_tagging_loss=0.009, over 3042416.07 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:39:15,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2493593.3333333335, ans=0.125
2023-11-23 18:39:20,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374050
2023-11-23 18:39:23,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2493660.0, ans=0.125
2023-11-23 18:39:26,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2493660.0, ans=0.1
2023-11-23 18:39:39,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2493726.6666666665, ans=0.125
2023-11-23 18:39:59,214 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.307e+01 8.471e+01 8.997e+01 9.654e+01 1.238e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 18:40:11,606 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1350, loss[loss=0.06975, simple_loss=0.08571, pruned_loss=0.01639, audio_tagging_loss=0.0105, over 15901.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09071, pruned_loss=0.01341, audio_tagging_loss=0.008992, over 3044193.41 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:40:14,889 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.37 vs. limit=5.0
2023-11-23 18:40:22,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374100
2023-11-23 18:40:24,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2493993.3333333335, ans=0.125
2023-11-23 18:40:30,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2493993.3333333335, ans=0.0
2023-11-23 18:40:34,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2493993.3333333335, ans=0.125
2023-11-23 18:40:37,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2494060.0, ans=0.0
2023-11-23 18:40:46,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2494060.0, ans=0.0
2023-11-23 18:40:47,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2494126.6666666665, ans=0.0
2023-11-23 18:40:54,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2494126.6666666665, ans=10.0
2023-11-23 18:40:55,029 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:40:58,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2494126.6666666665, ans=0.5
2023-11-23 18:41:04,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0
2023-11-23 18:41:11,753 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.15 vs. limit=15.0
2023-11-23 18:41:13,471 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1400, loss[loss=0.06138, simple_loss=0.07882, pruned_loss=0.01356, audio_tagging_loss=0.008406, over 15078.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09105, pruned_loss=0.01338, audio_tagging_loss=0.008947, over 3047844.98 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:41:13,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2494260.0, ans=0.1
2023-11-23 18:41:20,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2494260.0, ans=0.1
2023-11-23 18:41:25,512 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374150
2023-11-23 18:41:26,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.34 vs. limit=12.0
2023-11-23 18:41:26,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2494326.6666666665, ans=0.125
2023-11-23 18:41:28,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2494326.6666666665, ans=0.125
2023-11-23 18:41:37,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2494393.3333333335, ans=15.0
2023-11-23 18:41:41,704 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.11 vs. limit=12.0
2023-11-23 18:41:41,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.21 vs. limit=15.0
2023-11-23 18:41:49,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2494460.0, ans=0.125
2023-11-23 18:42:03,222 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.059e+01 8.437e+01 8.832e+01 9.783e+01 1.644e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 18:42:15,818 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1450, loss[loss=0.06953, simple_loss=0.09983, pruned_loss=0.01227, audio_tagging_loss=0.007344, over 15217.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09072, pruned_loss=0.01347, audio_tagging_loss=0.009047, over 3048498.04 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:42:19,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2494593.3333333335, ans=0.1
2023-11-23 18:42:27,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374200
2023-11-23 18:42:36,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2494660.0, ans=0.125
2023-11-23 18:42:37,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2494660.0, ans=0.0
2023-11-23 18:42:57,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2494793.3333333335, ans=0.125
2023-11-23 18:42:58,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2494793.3333333335, ans=0.125
2023-11-23 18:43:01,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.43 vs. limit=12.0
2023-11-23 18:43:11,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2494860.0, ans=0.125
2023-11-23 18:43:12,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2494860.0, ans=0.05
2023-11-23 18:43:12,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2494860.0, ans=0.125
2023-11-23 18:43:18,566 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1500, loss[loss=0.08242, simple_loss=0.11, pruned_loss=0.017, audio_tagging_loss=0.01041, over 15916.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09036, pruned_loss=0.01351, audio_tagging_loss=0.009111, over 3043117.76 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:43:20,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2494926.6666666665, ans=0.125
2023-11-23 18:43:22,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2494926.6666666665, ans=0.0
2023-11-23 18:43:29,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374250
2023-11-23 18:43:32,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2494993.3333333335, ans=0.125
2023-11-23 18:43:32,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2494993.3333333335, ans=0.125
2023-11-23 18:43:35,361 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.12 vs. limit=15.0
2023-11-23 18:43:53,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2495060.0, ans=0.125
2023-11-23 18:44:03,910 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.96 vs. limit=10.0
2023-11-23 18:44:07,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2495193.3333333335, ans=0.125
2023-11-23 18:44:08,637 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.534e+01 8.386e+01 9.077e+01 9.732e+01 1.149e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 18:44:21,222 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1550, loss[loss=0.06413, simple_loss=0.07335, pruned_loss=0.01624, audio_tagging_loss=0.01122, over 14964.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09097, pruned_loss=0.01362, audio_tagging_loss=0.009262, over 3051736.65 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:44:32,807 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374300
2023-11-23 18:44:53,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2495393.3333333335, ans=0.0
2023-11-23 18:45:03,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2495460.0, ans=0.95
2023-11-23 18:45:08,476 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.44 vs. limit=22.5
2023-11-23 18:45:13,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2495526.6666666665, ans=0.125
2023-11-23 18:45:24,448 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1600, loss[loss=0.06656, simple_loss=0.09814, pruned_loss=0.01154, audio_tagging_loss=0.00596, over 15365.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.0913, pruned_loss=0.0137, audio_tagging_loss=0.009309, over 3052615.46 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:45:25,964 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:45:27,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2495593.3333333335, ans=0.2
2023-11-23 18:45:35,057 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374350
2023-11-23 18:45:40,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2495660.0, ans=0.125
2023-11-23 18:45:44,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2495660.0, ans=0.125
2023-11-23 18:45:51,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2495726.6666666665, ans=0.07
2023-11-23 18:46:14,056 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.412e+01 9.043e+01 9.774e+01 1.173e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 18:46:24,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-23 18:46:25,940 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1650, loss[loss=0.07357, simple_loss=0.09284, pruned_loss=0.0161, audio_tagging_loss=0.01105, over 15352.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09094, pruned_loss=0.01356, audio_tagging_loss=0.009424, over 3054172.07 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:46:37,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374400
2023-11-23 18:46:59,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2496060.0, ans=0.0
2023-11-23 18:47:13,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2496126.6666666665, ans=0.07
2023-11-23 18:47:26,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2496193.3333333335, ans=0.07
2023-11-23 18:47:28,988 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1700, loss[loss=0.08228, simple_loss=0.1102, pruned_loss=0.01778, audio_tagging_loss=0.009374, over 16638.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09076, pruned_loss=0.01358, audio_tagging_loss=0.009475, over 3054313.48 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:47:39,711 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374450
2023-11-23 18:47:41,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2496326.6666666665, ans=0.0
2023-11-23 18:48:04,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2496393.3333333335, ans=0.1
2023-11-23 18:48:10,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2496460.0, ans=0.125
2023-11-23 18:48:18,899 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.196e+01 8.417e+01 9.193e+01 9.971e+01 1.264e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 18:48:22,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2496526.6666666665, ans=0.2
2023-11-23 18:48:22,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2496526.6666666665, ans=0.125
2023-11-23 18:48:29,516 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.85 vs. limit=22.5
2023-11-23 18:48:31,080 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1750, loss[loss=0.07189, simple_loss=0.08972, pruned_loss=0.01924, audio_tagging_loss=0.007789, over 16585.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09128, pruned_loss=0.01376, audio_tagging_loss=0.009293, over 3048050.74 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:48:41,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2496593.3333333335, ans=0.0
2023-11-23 18:48:42,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374500
2023-11-23 18:48:50,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2496660.0, ans=0.0
2023-11-23 18:48:54,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2496660.0, ans=10.0
2023-11-23 18:48:55,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2496726.6666666665, ans=0.1
2023-11-23 18:48:56,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2496726.6666666665, ans=0.125
2023-11-23 18:49:27,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2496860.0, ans=0.1
2023-11-23 18:49:33,572 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1800, loss[loss=0.07903, simple_loss=0.109, pruned_loss=0.01549, audio_tagging_loss=0.009041, over 15489.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09119, pruned_loss=0.01368, audio_tagging_loss=0.00919, over 3048294.56 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:49:41,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2496926.6666666665, ans=0.125
2023-11-23 18:49:45,326 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374550
2023-11-23 18:50:12,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2497126.6666666665, ans=0.125
2023-11-23 18:50:23,586 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.228e+01 8.763e+01 9.519e+01 1.190e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-23 18:50:35,990 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1850, loss[loss=0.06271, simple_loss=0.08769, pruned_loss=0.01332, audio_tagging_loss=0.005545, over 15830.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09085, pruned_loss=0.01367, audio_tagging_loss=0.009116, over 3051847.55 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:50:47,259 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374600
2023-11-23 18:50:52,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2497326.6666666665, ans=0.0
2023-11-23 18:50:56,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2497326.6666666665, ans=0.125
2023-11-23 18:51:38,714 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1900, loss[loss=0.07405, simple_loss=0.1008, pruned_loss=0.01364, audio_tagging_loss=0.009988, over 14106.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09112, pruned_loss=0.01366, audio_tagging_loss=0.009099, over 3053194.25 frames. ], batch size: 52, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:51:44,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2497593.3333333335, ans=0.1
2023-11-23 18:51:50,216 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374650
2023-11-23 18:52:13,655 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.81 vs. limit=15.0
2023-11-23 18:52:18,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.67 vs. limit=15.0
2023-11-23 18:52:29,659 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.323e+01 8.306e+01 8.793e+01 9.456e+01 1.356e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 18:52:31,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2497860.0, ans=0.2
2023-11-23 18:52:41,803 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 1950, loss[loss=0.06062, simple_loss=0.0775, pruned_loss=0.01108, audio_tagging_loss=0.0108, over 16354.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09074, pruned_loss=0.01357, audio_tagging_loss=0.009057, over 3053697.31 frames. ], batch size: 65, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:52:51,131 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.09 vs. limit=6.0
2023-11-23 18:52:53,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374700
2023-11-23 18:53:10,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2498060.0, ans=0.125
2023-11-23 18:53:20,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2498126.6666666665, ans=0.2
2023-11-23 18:53:43,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2498193.3333333335, ans=0.125
2023-11-23 18:53:43,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2498193.3333333335, ans=0.125
2023-11-23 18:53:45,442 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2000, loss[loss=0.08744, simple_loss=0.1151, pruned_loss=0.02223, audio_tagging_loss=0.00768, over 15906.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09008, pruned_loss=0.01347, audio_tagging_loss=0.009126, over 3047900.46 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:53:46,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2498260.0, ans=0.125
2023-11-23 18:53:49,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2498260.0, ans=0.125
2023-11-23 18:53:56,649 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374750
2023-11-23 18:54:08,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2498326.6666666665, ans=0.0
2023-11-23 18:54:15,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2498393.3333333335, ans=0.2
2023-11-23 18:54:25,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2498460.0, ans=0.125
2023-11-23 18:54:35,240 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.533e+01 8.441e+01 9.066e+01 9.690e+01 1.160e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 18:54:35,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2498526.6666666665, ans=0.2
2023-11-23 18:54:48,069 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2050, loss[loss=0.09941, simple_loss=0.129, pruned_loss=0.02621, audio_tagging_loss=0.008699, over 15322.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09199, pruned_loss=0.01381, audio_tagging_loss=0.009105, over 3054161.89 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:54:58,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2498593.3333333335, ans=0.125
2023-11-23 18:54:59,451 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374800
2023-11-23 18:55:02,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2498660.0, ans=0.125
2023-11-23 18:55:28,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2498793.3333333335, ans=0.5
2023-11-23 18:55:50,647 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2100, loss[loss=0.06317, simple_loss=0.09173, pruned_loss=0.0102, audio_tagging_loss=0.007099, over 16515.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09187, pruned_loss=0.01384, audio_tagging_loss=0.009031, over 3043487.29 frames. ], batch size: 63, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:55:54,904 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0
2023-11-23 18:55:55,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2498926.6666666665, ans=0.0
2023-11-23 18:56:01,931 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374850
2023-11-23 18:56:32,267 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.74 vs. limit=15.0
2023-11-23 18:56:39,980 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.878e+01 8.622e+01 9.048e+01 9.766e+01 1.225e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 18:56:45,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2499193.3333333335, ans=0.2
2023-11-23 18:56:52,563 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2150, loss[loss=0.06676, simple_loss=0.08932, pruned_loss=0.01223, audio_tagging_loss=0.009861, over 15582.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09114, pruned_loss=0.01384, audio_tagging_loss=0.009075, over 3040151.35 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:57:03,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2499260.0, ans=0.125
2023-11-23 18:57:04,354 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374900
2023-11-23 18:57:19,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.22 vs. limit=15.0
2023-11-23 18:57:25,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2499393.3333333335, ans=0.0
2023-11-23 18:57:29,788 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:57:48,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.01 vs. limit=15.0
2023-11-23 18:57:50,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2499526.6666666665, ans=0.125
2023-11-23 18:57:54,949 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2200, loss[loss=0.08628, simple_loss=0.1097, pruned_loss=0.0241, audio_tagging_loss=0.007334, over 14774.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09109, pruned_loss=0.01371, audio_tagging_loss=0.009139, over 3040651.07 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:58:05,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.24 vs. limit=15.0
2023-11-23 18:58:06,031 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 374950
2023-11-23 18:58:09,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2499660.0, ans=0.125
2023-11-23 18:58:18,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2499726.6666666665, ans=0.125
2023-11-23 18:58:19,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2499726.6666666665, ans=0.1
2023-11-23 18:58:28,661 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.53 vs. limit=15.0
2023-11-23 18:58:35,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.63 vs. limit=15.0
2023-11-23 18:58:42,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2499793.3333333335, ans=0.5
2023-11-23 18:58:44,342 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.234e+01 8.395e+01 9.074e+01 9.645e+01 1.441e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 18:58:54,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2499860.0, ans=0.125
2023-11-23 18:58:56,707 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2250, loss[loss=0.0779, simple_loss=0.1134, pruned_loss=0.01415, audio_tagging_loss=0.007044, over 16091.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09197, pruned_loss=0.01378, audio_tagging_loss=0.009129, over 3040697.88 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:59:01,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2499926.6666666665, ans=0.125
2023-11-23 18:59:06,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2499926.6666666665, ans=0.0
2023-11-23 18:59:06,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2499926.6666666665, ans=0.0
2023-11-23 18:59:06,856 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.95 vs. limit=15.0
2023-11-23 18:59:07,934 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375000
2023-11-23 18:59:29,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2500060.0, ans=0.125
2023-11-23 18:59:42,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2500126.6666666665, ans=0.1
2023-11-23 18:59:43,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.03 vs. limit=15.0
2023-11-23 18:59:46,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2500193.3333333335, ans=0.05
2023-11-23 18:59:49,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.72 vs. limit=15.0
2023-11-23 18:59:54,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2500193.3333333335, ans=0.0
2023-11-23 18:59:58,412 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2300, loss[loss=0.05954, simple_loss=0.07602, pruned_loss=0.01173, audio_tagging_loss=0.009806, over 14808.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09203, pruned_loss=0.01392, audio_tagging_loss=0.009135, over 3040246.66 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:00:06,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2500260.0, ans=0.05
2023-11-23 19:00:10,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375050
2023-11-23 19:00:27,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2500393.3333333335, ans=0.1
2023-11-23 19:00:28,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2500393.3333333335, ans=0.125
2023-11-23 19:00:33,805 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.77 vs. limit=15.0
2023-11-23 19:00:45,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2500460.0, ans=0.1
2023-11-23 19:00:47,716 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.606e+01 9.166e+01 9.861e+01 1.207e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 19:00:50,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2500526.6666666665, ans=0.1
2023-11-23 19:00:52,502 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:00:56,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2500526.6666666665, ans=0.1
2023-11-23 19:01:00,769 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2350, loss[loss=0.05859, simple_loss=0.07288, pruned_loss=0.01151, audio_tagging_loss=0.01064, over 14642.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09206, pruned_loss=0.01391, audio_tagging_loss=0.009187, over 3039112.86 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:01:12,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375100
2023-11-23 19:01:12,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2500660.0, ans=0.125
2023-11-23 19:01:14,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2500660.0, ans=0.125
2023-11-23 19:01:19,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2500660.0, ans=0.125
2023-11-23 19:01:23,213 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.54 vs. limit=15.0
2023-11-23 19:01:25,704 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.49 vs. limit=15.0
2023-11-23 19:01:46,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2500793.3333333335, ans=0.1
2023-11-23 19:01:55,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2500860.0, ans=0.125
2023-11-23 19:02:01,586 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.26 vs. limit=15.0
2023-11-23 19:02:03,150 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2400, loss[loss=0.06481, simple_loss=0.09563, pruned_loss=0.007969, audio_tagging_loss=0.00903, over 14300.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09212, pruned_loss=0.01371, audio_tagging_loss=0.009208, over 3047482.53 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:02:13,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375150
2023-11-23 19:02:42,039 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:02:54,390 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.466e+01 8.908e+01 9.614e+01 2.076e+02, threshold=1.782e+02, percent-clipped=1.0
2023-11-23 19:02:57,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2501193.3333333335, ans=0.0
2023-11-23 19:03:01,738 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:03:05,680 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2450, loss[loss=0.04634, simple_loss=0.05304, pruned_loss=0.007331, audio_tagging_loss=0.01248, over 14490.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09214, pruned_loss=0.01361, audio_tagging_loss=0.00923, over 3044068.53 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:03:11,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2501260.0, ans=0.0
2023-11-23 19:03:16,922 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375200
2023-11-23 19:03:20,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2501326.6666666665, ans=0.125
2023-11-23 19:03:58,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2501526.6666666665, ans=0.125
2023-11-23 19:04:00,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2501526.6666666665, ans=0.125
2023-11-23 19:04:00,535 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.21 vs. limit=15.0
2023-11-23 19:04:07,615 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2500, loss[loss=0.07699, simple_loss=0.1077, pruned_loss=0.01618, audio_tagging_loss=0.00695, over 14508.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09231, pruned_loss=0.01382, audio_tagging_loss=0.00928, over 3052036.57 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:04:16,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2501593.3333333335, ans=0.0
2023-11-23 19:04:19,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375250
2023-11-23 19:04:41,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.42 vs. limit=15.0
2023-11-23 19:04:45,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2501793.3333333335, ans=0.0
2023-11-23 19:04:59,240 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.56 vs. limit=22.5
2023-11-23 19:05:01,027 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.221e+01 8.437e+01 9.142e+01 9.814e+01 1.303e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 19:05:10,380 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2550, loss[loss=0.07876, simple_loss=0.1118, pruned_loss=0.01487, audio_tagging_loss=0.007998, over 14464.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09206, pruned_loss=0.0137, audio_tagging_loss=0.009223, over 3050171.10 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:05:18,325 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.62 vs. limit=22.5
2023-11-23 19:05:21,361 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375300
2023-11-23 19:05:43,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2502060.0, ans=0.1
2023-11-23 19:05:48,288 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.07 vs. limit=22.5
2023-11-23 19:06:02,254 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.76 vs. limit=15.0
2023-11-23 19:06:10,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.71 vs. limit=10.0
2023-11-23 19:06:12,337 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2600, loss[loss=0.0651, simple_loss=0.09141, pruned_loss=0.01055, audio_tagging_loss=0.008838, over 13819.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09229, pruned_loss=0.01376, audio_tagging_loss=0.009123, over 3040617.61 frames. ], batch size: 52, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:06:16,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2502260.0, ans=0.07
2023-11-23 19:06:23,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375350
2023-11-23 19:06:38,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2502393.3333333335, ans=0.0
2023-11-23 19:07:06,774 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.389e+01 8.376e+01 8.898e+01 9.967e+01 2.098e+02, threshold=1.780e+02, percent-clipped=2.0
2023-11-23 19:07:15,704 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2650, loss[loss=0.08224, simple_loss=0.1177, pruned_loss=0.01556, audio_tagging_loss=0.007806, over 15743.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09305, pruned_loss=0.0139, audio_tagging_loss=0.008942, over 3044024.39 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:07:15,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2502593.3333333335, ans=0.0
2023-11-23 19:07:26,395 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375400
2023-11-23 19:07:38,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.73 vs. limit=22.5
2023-11-23 19:07:39,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.04 vs. limit=12.0
2023-11-23 19:07:43,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2502726.6666666665, ans=0.5
2023-11-23 19:07:47,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2502726.6666666665, ans=0.125
2023-11-23 19:07:49,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2502726.6666666665, ans=0.125
2023-11-23 19:08:03,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2502793.3333333335, ans=0.125
2023-11-23 19:08:17,551 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2700, loss[loss=0.07504, simple_loss=0.1043, pruned_loss=0.01408, audio_tagging_loss=0.008801, over 15302.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09189, pruned_loss=0.01368, audio_tagging_loss=0.008959, over 3051335.99 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:08:27,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2502926.6666666665, ans=0.125
2023-11-23 19:08:28,730 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375450
2023-11-23 19:08:31,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2502993.3333333335, ans=0.125
2023-11-23 19:08:55,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2503126.6666666665, ans=0.125
2023-11-23 19:09:07,892 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:09:10,589 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.343e+01 9.045e+01 9.938e+01 1.315e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 19:09:12,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2503193.3333333335, ans=0.125
2023-11-23 19:09:12,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.48 vs. limit=15.0
2023-11-23 19:09:18,822 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2750, loss[loss=0.06742, simple_loss=0.0864, pruned_loss=0.01488, audio_tagging_loss=0.00934, over 14707.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09117, pruned_loss=0.0136, audio_tagging_loss=0.009048, over 3044218.85 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:09:30,059 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375500
2023-11-23 19:10:11,500 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:10:17,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2503526.6666666665, ans=0.125
2023-11-23 19:10:20,906 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2800, loss[loss=0.08257, simple_loss=0.1172, pruned_loss=0.01776, audio_tagging_loss=0.006227, over 15500.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.0912, pruned_loss=0.01353, audio_tagging_loss=0.009015, over 3046449.93 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:10:24,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2503593.3333333335, ans=0.125
2023-11-23 19:10:32,238 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375550
2023-11-23 19:10:32,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2503660.0, ans=0.0
2023-11-23 19:11:06,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2503793.3333333335, ans=0.0
2023-11-23 19:11:14,365 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.252e+01 8.866e+01 9.555e+01 1.297e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 19:11:16,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2503860.0, ans=0.2
2023-11-23 19:11:20,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2503860.0, ans=0.125
2023-11-23 19:11:22,795 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2850, loss[loss=0.06529, simple_loss=0.08435, pruned_loss=0.01261, audio_tagging_loss=0.01051, over 15149.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09125, pruned_loss=0.01354, audio_tagging_loss=0.009053, over 3046706.20 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:11:26,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2503926.6666666665, ans=0.125
2023-11-23 19:11:26,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2503926.6666666665, ans=0.125
2023-11-23 19:11:28,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2503926.6666666665, ans=0.125
2023-11-23 19:11:30,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2503926.6666666665, ans=0.125
2023-11-23 19:11:34,014 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375600
2023-11-23 19:11:38,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2503993.3333333335, ans=0.125
2023-11-23 19:11:45,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2503993.3333333335, ans=0.1
2023-11-23 19:11:58,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.06 vs. limit=6.0
2023-11-23 19:12:26,389 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2900, loss[loss=0.06477, simple_loss=0.08562, pruned_loss=0.01228, audio_tagging_loss=0.009678, over 15127.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.0908, pruned_loss=0.01333, audio_tagging_loss=0.009112, over 3048510.47 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:12:34,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2504260.0, ans=0.125
2023-11-23 19:12:35,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.36 vs. limit=15.0
2023-11-23 19:12:37,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375650
2023-11-23 19:13:06,153 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:13:08,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2504460.0, ans=0.04949747468305833
2023-11-23 19:13:12,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2504460.0, ans=0.1
2023-11-23 19:13:20,154 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.540e+01 9.133e+01 9.789e+01 1.546e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 19:13:22,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2504526.6666666665, ans=0.1
2023-11-23 19:13:28,624 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 2950, loss[loss=0.0774, simple_loss=0.0979, pruned_loss=0.0177, audio_tagging_loss=0.01075, over 15134.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.0915, pruned_loss=0.01352, audio_tagging_loss=0.009126, over 3050657.07 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:13:34,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.46 vs. limit=15.0
2023-11-23 19:13:35,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2504593.3333333335, ans=0.0
2023-11-23 19:13:36,917 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=15.0
2023-11-23 19:13:40,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375700
2023-11-23 19:13:40,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2504660.0, ans=0.2
2023-11-23 19:13:42,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2504660.0, ans=0.0
2023-11-23 19:13:54,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2504726.6666666665, ans=0.125
2023-11-23 19:14:02,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2504726.6666666665, ans=0.0
2023-11-23 19:14:10,941 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.44 vs. limit=12.0
2023-11-23 19:14:31,246 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3000, loss[loss=0.06353, simple_loss=0.08387, pruned_loss=0.009808, audio_tagging_loss=0.01179, over 15433.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09145, pruned_loss=0.01359, audio_tagging_loss=0.009211, over 3054979.66 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:14:31,247 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 19:15:09,807 INFO [train_asr.py:1253] (2/4) Epoch 32, validation: loss=0.05818, simple_loss=0.05104, pruned_loss=0.005158, audio_tagging_loss=0.0275, over 4681554.00 frames. 
2023-11-23 19:15:09,808 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 19:15:10,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2504926.6666666665, ans=0.125
2023-11-23 19:15:11,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2504926.6666666665, ans=0.04949747468305833
2023-11-23 19:15:16,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.94 vs. limit=10.0
2023-11-23 19:15:21,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375750
2023-11-23 19:15:21,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2504993.3333333335, ans=0.0
2023-11-23 19:15:31,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2504993.3333333335, ans=0.125
2023-11-23 19:15:33,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2505060.0, ans=0.125
2023-11-23 19:16:03,446 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.656e+01 9.141e+01 9.887e+01 1.235e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 19:16:08,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2505193.3333333335, ans=0.04949747468305833
2023-11-23 19:16:11,766 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3050, loss[loss=0.07814, simple_loss=0.09797, pruned_loss=0.0157, audio_tagging_loss=0.01345, over 16068.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09209, pruned_loss=0.01371, audio_tagging_loss=0.009272, over 3055348.09 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:16:23,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375800
2023-11-23 19:16:36,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.59 vs. limit=15.0
2023-11-23 19:16:37,746 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.61 vs. limit=12.0
2023-11-23 19:16:38,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2505393.3333333335, ans=0.2
2023-11-23 19:16:48,173 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:16:52,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2505460.0, ans=0.1
2023-11-23 19:17:06,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2505526.6666666665, ans=0.0
2023-11-23 19:17:13,715 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3100, loss[loss=0.06746, simple_loss=0.09325, pruned_loss=0.01499, audio_tagging_loss=0.005844, over 14725.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09241, pruned_loss=0.01378, audio_tagging_loss=0.009254, over 3061648.87 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:17:20,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2505593.3333333335, ans=0.125
2023-11-23 19:17:25,014 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375850
2023-11-23 19:17:32,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2505660.0, ans=0.0
2023-11-23 19:18:07,373 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.481e+01 9.058e+01 9.503e+01 1.492e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 19:18:08,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2505860.0, ans=0.125
2023-11-23 19:18:15,794 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3150, loss[loss=0.06615, simple_loss=0.09317, pruned_loss=0.01142, audio_tagging_loss=0.008147, over 16169.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09322, pruned_loss=0.01392, audio_tagging_loss=0.009248, over 3057583.24 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:18:18,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2505926.6666666665, ans=0.125
2023-11-23 19:18:24,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2505926.6666666665, ans=0.125
2023-11-23 19:18:24,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2505926.6666666665, ans=0.125
2023-11-23 19:18:27,730 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375900
2023-11-23 19:18:32,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2505993.3333333335, ans=0.1
2023-11-23 19:18:38,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2505993.3333333335, ans=0.125
2023-11-23 19:19:14,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.25 vs. limit=10.0
2023-11-23 19:19:18,393 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3200, loss[loss=0.07049, simple_loss=0.1075, pruned_loss=0.009555, audio_tagging_loss=0.007175, over 15323.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09323, pruned_loss=0.01391, audio_tagging_loss=0.009177, over 3047325.44 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:19:29,686 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 375950
2023-11-23 19:19:29,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2506326.6666666665, ans=0.0
2023-11-23 19:19:34,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2506326.6666666665, ans=0.0
2023-11-23 19:19:42,493 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.31 vs. limit=15.0
2023-11-23 19:19:48,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2506393.3333333335, ans=0.1
2023-11-23 19:19:55,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2506460.0, ans=0.125
2023-11-23 19:20:08,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2506526.6666666665, ans=0.1
2023-11-23 19:20:13,191 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.152e+01 8.777e+01 9.495e+01 2.540e+02, threshold=1.755e+02, percent-clipped=1.0
2023-11-23 19:20:14,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2506526.6666666665, ans=0.125
2023-11-23 19:20:15,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2506526.6666666665, ans=0.0
2023-11-23 19:20:20,385 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3250, loss[loss=0.05938, simple_loss=0.07719, pruned_loss=0.01192, audio_tagging_loss=0.00886, over 15087.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09332, pruned_loss=0.01401, audio_tagging_loss=0.009179, over 3051643.85 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:20:31,862 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376000
2023-11-23 19:20:51,651 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.46 vs. limit=15.0
2023-11-23 19:21:16,373 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2506860.0, ans=0.125
2023-11-23 19:21:21,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2506860.0, ans=0.125
2023-11-23 19:21:26,144 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3300, loss[loss=0.06205, simple_loss=0.08184, pruned_loss=0.01201, audio_tagging_loss=0.009122, over 15547.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09316, pruned_loss=0.01393, audio_tagging_loss=0.009313, over 3049455.52 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:21:37,342 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376050
2023-11-23 19:21:42,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0
2023-11-23 19:21:49,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2506993.3333333335, ans=0.125
2023-11-23 19:22:20,310 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.632e+01 8.685e+01 9.326e+01 1.032e+02 1.222e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-23 19:22:28,032 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3350, loss[loss=0.0775, simple_loss=0.1049, pruned_loss=0.01717, audio_tagging_loss=0.007859, over 15564.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09266, pruned_loss=0.01385, audio_tagging_loss=0.009294, over 3053775.24 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:22:39,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376100
2023-11-23 19:22:40,539 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=16.15 vs. limit=15.0
2023-11-23 19:23:01,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2507393.3333333335, ans=0.125
2023-11-23 19:23:08,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2507460.0, ans=0.125
2023-11-23 19:23:15,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2507460.0, ans=0.125
2023-11-23 19:23:20,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2507526.6666666665, ans=0.2
2023-11-23 19:23:30,757 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3400, loss[loss=0.07714, simple_loss=0.1219, pruned_loss=0.01141, audio_tagging_loss=0.004782, over 15759.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09301, pruned_loss=0.01388, audio_tagging_loss=0.009176, over 3055279.07 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:23:41,854 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376150
2023-11-23 19:23:43,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2507660.0, ans=0.07
2023-11-23 19:24:01,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2507726.6666666665, ans=0.1
2023-11-23 19:24:01,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2507726.6666666665, ans=0.125
2023-11-23 19:24:03,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2507726.6666666665, ans=0.125
2023-11-23 19:24:18,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2507793.3333333335, ans=0.0
2023-11-23 19:24:19,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2507860.0, ans=0.95
2023-11-23 19:24:24,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.340e+01 8.680e+01 9.645e+01 1.165e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 19:24:27,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2507860.0, ans=0.0
2023-11-23 19:24:32,681 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3450, loss[loss=0.06021, simple_loss=0.08801, pruned_loss=0.00901, audio_tagging_loss=0.007201, over 13959.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09306, pruned_loss=0.01402, audio_tagging_loss=0.009101, over 3052593.27 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:24:34,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2507926.6666666665, ans=0.2
2023-11-23 19:24:40,313 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.07 vs. limit=6.0
2023-11-23 19:24:43,393 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376200
2023-11-23 19:24:51,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2507993.3333333335, ans=0.125
2023-11-23 19:24:55,018 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:24:56,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2508060.0, ans=0.1
2023-11-23 19:25:00,889 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:25:01,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2508060.0, ans=0.0
2023-11-23 19:25:30,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2508193.3333333335, ans=0.07
2023-11-23 19:25:33,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.13 vs. limit=22.5
2023-11-23 19:25:33,324 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.81 vs. limit=22.5
2023-11-23 19:25:35,027 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3500, loss[loss=0.07505, simple_loss=0.1067, pruned_loss=0.01309, audio_tagging_loss=0.008605, over 16604.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09265, pruned_loss=0.01383, audio_tagging_loss=0.009037, over 3053095.13 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:25:35,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2508260.0, ans=0.0
2023-11-23 19:25:43,390 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.79 vs. limit=8.0
2023-11-23 19:25:46,590 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376250
2023-11-23 19:26:06,162 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.07 vs. limit=22.5
2023-11-23 19:26:06,585 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:26:29,755 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.248e+01 9.005e+01 9.777e+01 1.238e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 19:26:38,181 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3550, loss[loss=0.05945, simple_loss=0.08067, pruned_loss=0.01167, audio_tagging_loss=0.007443, over 15176.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09244, pruned_loss=0.01376, audio_tagging_loss=0.009031, over 3048911.20 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:26:43,400 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:26:49,831 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376300
2023-11-23 19:27:04,819 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.69 vs. limit=22.5
2023-11-23 19:27:19,967 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:27:21,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.89 vs. limit=15.0
2023-11-23 19:27:29,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2508860.0, ans=0.125
2023-11-23 19:27:31,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.20 vs. limit=15.0
2023-11-23 19:27:33,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2508860.0, ans=0.2
2023-11-23 19:27:40,971 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3600, loss[loss=0.04441, simple_loss=0.04442, pruned_loss=0.008343, audio_tagging_loss=0.01386, over 16503.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09185, pruned_loss=0.01376, audio_tagging_loss=0.009037, over 3053925.51 frames. ], batch size: 64, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:27:45,390 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.18 vs. limit=22.5
2023-11-23 19:27:51,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376350
2023-11-23 19:28:35,122 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.359e+01 9.090e+01 9.890e+01 1.396e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 19:28:39,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2509193.3333333335, ans=0.1
2023-11-23 19:28:42,897 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3650, loss[loss=0.06647, simple_loss=0.09334, pruned_loss=0.01344, audio_tagging_loss=0.006359, over 15813.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09289, pruned_loss=0.01403, audio_tagging_loss=0.008991, over 3053242.48 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:28:54,287 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376400
2023-11-23 19:28:55,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2509326.6666666665, ans=0.0
2023-11-23 19:29:07,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2509393.3333333335, ans=0.125
2023-11-23 19:29:08,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.16 vs. limit=22.5
2023-11-23 19:29:35,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2509526.6666666665, ans=0.125
2023-11-23 19:29:44,982 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3700, loss[loss=0.0634, simple_loss=0.07335, pruned_loss=0.01767, audio_tagging_loss=0.009052, over 15431.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09255, pruned_loss=0.01394, audio_tagging_loss=0.009018, over 3050700.68 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:29:56,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376450
2023-11-23 19:29:58,296 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.57 vs. limit=15.0
2023-11-23 19:30:05,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff3.min_abs, batch_count=2509660.0, ans=0.2
2023-11-23 19:30:06,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2509660.0, ans=0.2
2023-11-23 19:30:12,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2509726.6666666665, ans=0.125
2023-11-23 19:30:15,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2509726.6666666665, ans=0.1
2023-11-23 19:30:16,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2509726.6666666665, ans=0.125
2023-11-23 19:30:18,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2509726.6666666665, ans=0.1
2023-11-23 19:30:22,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2509793.3333333335, ans=0.125
2023-11-23 19:30:36,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2509860.0, ans=0.0
2023-11-23 19:30:36,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2509860.0, ans=10.0
2023-11-23 19:30:36,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2509860.0, ans=0.125
2023-11-23 19:30:40,600 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.564e+01 8.395e+01 9.094e+01 9.680e+01 1.215e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-23 19:30:44,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2509860.0, ans=0.1
2023-11-23 19:30:48,255 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3750, loss[loss=0.07629, simple_loss=0.1084, pruned_loss=0.01325, audio_tagging_loss=0.008851, over 16989.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09336, pruned_loss=0.01411, audio_tagging_loss=0.009, over 3056183.74 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:30:59,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376500
2023-11-23 19:31:08,855 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.27 vs. limit=15.0
2023-11-23 19:31:15,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2510060.0, ans=0.125
2023-11-23 19:31:25,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2510126.6666666665, ans=0.125
2023-11-23 19:31:30,744 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:31:34,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2510126.6666666665, ans=0.125
2023-11-23 19:31:41,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2510193.3333333335, ans=0.0
2023-11-23 19:31:43,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2510193.3333333335, ans=0.0
2023-11-23 19:31:44,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2510193.3333333335, ans=0.1
2023-11-23 19:31:50,358 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3800, loss[loss=0.07551, simple_loss=0.09848, pruned_loss=0.01853, audio_tagging_loss=0.007738, over 14795.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09198, pruned_loss=0.0138, audio_tagging_loss=0.009068, over 3053270.50 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:32:01,463 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376550
2023-11-23 19:32:11,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2510326.6666666665, ans=0.125
2023-11-23 19:32:15,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2510393.3333333335, ans=0.125
2023-11-23 19:32:20,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2510393.3333333335, ans=0.125
2023-11-23 19:32:31,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2510460.0, ans=0.125
2023-11-23 19:32:45,289 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.658e+01 9.226e+01 1.003e+02 1.868e+02, threshold=1.845e+02, percent-clipped=1.0
2023-11-23 19:32:52,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2510593.3333333335, ans=0.125
2023-11-23 19:32:52,902 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3850, loss[loss=0.06745, simple_loss=0.0808, pruned_loss=0.01631, audio_tagging_loss=0.01073, over 14514.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09206, pruned_loss=0.01384, audio_tagging_loss=0.009194, over 3049628.49 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:32:54,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2510593.3333333335, ans=0.125
2023-11-23 19:32:59,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.72 vs. limit=22.5
2023-11-23 19:33:03,583 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376600
2023-11-23 19:33:04,107 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.52 vs. limit=15.0
2023-11-23 19:33:09,846 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:33:16,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2510726.6666666665, ans=0.1
2023-11-23 19:33:30,783 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:33:36,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=2510793.3333333335, ans=0.02
2023-11-23 19:33:54,690 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3900, loss[loss=0.0406, simple_loss=0.04983, pruned_loss=0.004496, audio_tagging_loss=0.01119, over 15525.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09212, pruned_loss=0.0138, audio_tagging_loss=0.009135, over 3044571.06 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:34:05,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376650
2023-11-23 19:34:18,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2511060.0, ans=0.125
2023-11-23 19:34:42,528 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=12.0
2023-11-23 19:34:49,478 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.099e+01 8.305e+01 8.896e+01 9.680e+01 1.276e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 19:34:56,882 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 3950, loss[loss=0.06226, simple_loss=0.07472, pruned_loss=0.01311, audio_tagging_loss=0.01179, over 14255.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09238, pruned_loss=0.01382, audio_tagging_loss=0.009272, over 3043779.33 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:35:08,289 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376700
2023-11-23 19:35:16,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2511326.6666666665, ans=0.0
2023-11-23 19:35:29,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2511393.3333333335, ans=0.0
2023-11-23 19:35:47,270 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.73 vs. limit=15.0
2023-11-23 19:35:59,031 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4000, loss[loss=0.06856, simple_loss=0.09081, pruned_loss=0.01255, audio_tagging_loss=0.0106, over 14865.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09213, pruned_loss=0.01371, audio_tagging_loss=0.009383, over 3048726.65 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:36:10,326 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376750
2023-11-23 19:36:19,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.17 vs. limit=15.0
2023-11-23 19:36:20,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2511660.0, ans=0.125
2023-11-23 19:36:41,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2511793.3333333335, ans=0.0
2023-11-23 19:36:53,799 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.509e+01 8.978e+01 9.607e+01 1.233e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 19:37:00,847 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4050, loss[loss=0.05571, simple_loss=0.07475, pruned_loss=0.00917, audio_tagging_loss=0.009171, over 13851.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09245, pruned_loss=0.01379, audio_tagging_loss=0.009361, over 3046256.73 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:37:03,260 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:37:07,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2511926.6666666665, ans=0.125
2023-11-23 19:37:12,145 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.37 vs. limit=15.0
2023-11-23 19:37:12,605 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376800
2023-11-23 19:37:12,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2511993.3333333335, ans=0.125
2023-11-23 19:37:29,471 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.49 vs. limit=6.0
2023-11-23 19:37:41,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2512126.6666666665, ans=0.1
2023-11-23 19:37:43,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2512126.6666666665, ans=0.0
2023-11-23 19:38:03,250 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4100, loss[loss=0.0595, simple_loss=0.07853, pruned_loss=0.01124, audio_tagging_loss=0.008985, over 15562.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09237, pruned_loss=0.0137, audio_tagging_loss=0.00937, over 3044966.52 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:38:14,706 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376850
2023-11-23 19:38:26,997 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:38:32,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2512393.3333333335, ans=0.1
2023-11-23 19:38:40,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2512460.0, ans=0.1
2023-11-23 19:38:42,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2512460.0, ans=0.1
2023-11-23 19:38:59,476 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.223e+01 8.590e+01 9.258e+01 1.002e+02 1.554e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-23 19:39:03,461 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:39:05,552 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4150, loss[loss=0.05261, simple_loss=0.06627, pruned_loss=0.009077, audio_tagging_loss=0.0104, over 15451.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09214, pruned_loss=0.01363, audio_tagging_loss=0.009217, over 3043954.20 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:39:16,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376900
2023-11-23 19:39:41,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2512793.3333333335, ans=0.125
2023-11-23 19:39:49,377 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:40:07,868 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4200, loss[loss=0.07413, simple_loss=0.09488, pruned_loss=0.01564, audio_tagging_loss=0.01105, over 15721.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09101, pruned_loss=0.01342, audio_tagging_loss=0.009167, over 3047738.47 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:40:10,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2512926.6666666665, ans=0.1
2023-11-23 19:40:19,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 376950
2023-11-23 19:40:19,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2512993.3333333335, ans=0.05
2023-11-23 19:40:28,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2512993.3333333335, ans=0.125
2023-11-23 19:41:01,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2513193.3333333335, ans=0.125
2023-11-23 19:41:02,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.21 vs. limit=10.0
2023-11-23 19:41:04,177 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.429e+01 8.916e+01 9.542e+01 1.145e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 19:41:10,204 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4250, loss[loss=0.0613, simple_loss=0.08452, pruned_loss=0.0112, audio_tagging_loss=0.007839, over 14818.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09172, pruned_loss=0.01348, audio_tagging_loss=0.009058, over 3047039.84 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:41:22,238 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377000
2023-11-23 19:41:25,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2513326.6666666665, ans=0.2
2023-11-23 19:41:27,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2513326.6666666665, ans=0.125
2023-11-23 19:41:45,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2513393.3333333335, ans=0.125
2023-11-23 19:41:56,164 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.71 vs. limit=15.0
2023-11-23 19:42:09,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2513526.6666666665, ans=0.125
2023-11-23 19:42:13,218 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4300, loss[loss=0.06892, simple_loss=0.09438, pruned_loss=0.014, audio_tagging_loss=0.007732, over 16552.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09133, pruned_loss=0.0134, audio_tagging_loss=0.009052, over 3051234.35 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:42:24,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377050
2023-11-23 19:42:31,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2513660.0, ans=0.125
2023-11-23 19:42:43,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2513726.6666666665, ans=0.125
2023-11-23 19:42:51,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2513793.3333333335, ans=0.125
2023-11-23 19:42:57,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2513793.3333333335, ans=0.1
2023-11-23 19:43:09,396 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.731e+01 8.254e+01 8.867e+01 9.585e+01 1.161e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 19:43:15,449 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4350, loss[loss=0.0474, simple_loss=0.05448, pruned_loss=0.008514, audio_tagging_loss=0.01165, over 14630.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09194, pruned_loss=0.01354, audio_tagging_loss=0.008966, over 3039734.08 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:43:26,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377100
2023-11-23 19:43:29,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2513993.3333333335, ans=0.125
2023-11-23 19:43:29,563 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.43 vs. limit=15.0
2023-11-23 19:43:41,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2514060.0, ans=0.1
2023-11-23 19:43:46,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2514060.0, ans=0.0
2023-11-23 19:43:51,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2514126.6666666665, ans=0.025
2023-11-23 19:43:51,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2514126.6666666665, ans=0.2
2023-11-23 19:43:56,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2514126.6666666665, ans=0.125
2023-11-23 19:43:58,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2514126.6666666665, ans=0.125
2023-11-23 19:44:00,574 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.39 vs. limit=15.0
2023-11-23 19:44:08,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2514193.3333333335, ans=0.125
2023-11-23 19:44:11,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2514193.3333333335, ans=0.125
2023-11-23 19:44:11,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2514193.3333333335, ans=0.125
2023-11-23 19:44:16,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2514260.0, ans=0.125
2023-11-23 19:44:17,152 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4400, loss[loss=0.08542, simple_loss=0.1153, pruned_loss=0.01863, audio_tagging_loss=0.009145, over 15563.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09178, pruned_loss=0.0137, audio_tagging_loss=0.008906, over 3035114.27 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:44:17,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff3.min_abs, batch_count=2514260.0, ans=0.2
2023-11-23 19:44:18,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2514260.0, ans=0.125
2023-11-23 19:44:28,422 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377150
2023-11-23 19:44:31,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2514326.6666666665, ans=0.125
2023-11-23 19:44:33,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2514326.6666666665, ans=0.025
2023-11-23 19:44:35,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.04 vs. limit=22.5
2023-11-23 19:44:36,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.38 vs. limit=5.0
2023-11-23 19:44:37,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2514326.6666666665, ans=0.125
2023-11-23 19:44:56,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2514460.0, ans=0.1
2023-11-23 19:45:03,123 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.63 vs. limit=8.0
2023-11-23 19:45:13,013 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.470e+01 9.052e+01 9.834e+01 1.276e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 19:45:20,344 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4450, loss[loss=0.05962, simple_loss=0.07581, pruned_loss=0.01066, audio_tagging_loss=0.01105, over 15444.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.0921, pruned_loss=0.01385, audio_tagging_loss=0.008899, over 3044407.91 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:45:31,741 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377200
2023-11-23 19:45:58,476 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.48 vs. limit=15.0
2023-11-23 19:46:14,969 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0
2023-11-23 19:46:15,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2514860.0, ans=0.0
2023-11-23 19:46:20,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2514860.0, ans=0.125
2023-11-23 19:46:23,442 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4500, loss[loss=0.05605, simple_loss=0.08401, pruned_loss=0.007101, audio_tagging_loss=0.006944, over 15317.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09181, pruned_loss=0.01371, audio_tagging_loss=0.008895, over 3050015.73 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:46:34,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377250
2023-11-23 19:46:35,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.28 vs. limit=22.5
2023-11-23 19:46:35,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2514993.3333333335, ans=0.0
2023-11-23 19:47:04,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2515126.6666666665, ans=0.125
2023-11-23 19:47:07,439 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2515126.6666666665, ans=0.0
2023-11-23 19:47:13,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2515193.3333333335, ans=0.125
2023-11-23 19:47:18,871 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.414e+01 9.190e+01 9.821e+01 1.226e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-23 19:47:25,488 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4550, loss[loss=0.07522, simple_loss=0.09624, pruned_loss=0.01902, audio_tagging_loss=0.008078, over 16243.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09191, pruned_loss=0.01363, audio_tagging_loss=0.008909, over 3045766.38 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:47:26,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2515260.0, ans=0.125
2023-11-23 19:47:35,586 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.14 vs. limit=22.5
2023-11-23 19:47:36,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377300
2023-11-23 19:47:41,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2515326.6666666665, ans=0.125
2023-11-23 19:47:43,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2515326.6666666665, ans=0.125
2023-11-23 19:47:44,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2515326.6666666665, ans=0.2
2023-11-23 19:47:45,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2515326.6666666665, ans=0.1
2023-11-23 19:48:03,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.65 vs. limit=10.0
2023-11-23 19:48:11,889 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:48:15,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2515526.6666666665, ans=0.2
2023-11-23 19:48:25,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2515526.6666666665, ans=0.125
2023-11-23 19:48:25,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2515526.6666666665, ans=0.125
2023-11-23 19:48:28,008 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4600, loss[loss=0.06072, simple_loss=0.08312, pruned_loss=0.01057, audio_tagging_loss=0.008589, over 14915.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09226, pruned_loss=0.0138, audio_tagging_loss=0.008935, over 3046452.76 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:48:30,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2515593.3333333335, ans=0.0
2023-11-23 19:48:31,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2515593.3333333335, ans=0.025
2023-11-23 19:48:33,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2515593.3333333335, ans=0.125
2023-11-23 19:48:39,058 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377350
2023-11-23 19:48:42,471 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.37 vs. limit=22.5
2023-11-23 19:48:53,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.51 vs. limit=5.0
2023-11-23 19:49:05,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2515793.3333333335, ans=0.1
2023-11-23 19:49:07,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2515793.3333333335, ans=0.0
2023-11-23 19:49:11,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2515793.3333333335, ans=0.125
2023-11-23 19:49:15,691 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=15.0
2023-11-23 19:49:23,552 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.494e+01 9.080e+01 9.829e+01 1.199e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 19:49:30,634 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4650, loss[loss=0.06093, simple_loss=0.08486, pruned_loss=0.009374, audio_tagging_loss=0.009126, over 15324.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09241, pruned_loss=0.01376, audio_tagging_loss=0.009028, over 3044381.19 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:49:41,359 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377400
2023-11-23 19:49:45,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2515993.3333333335, ans=0.2
2023-11-23 19:50:06,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2516126.6666666665, ans=0.0
2023-11-23 19:50:25,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2516193.3333333335, ans=0.125
2023-11-23 19:50:33,072 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4700, loss[loss=0.04389, simple_loss=0.04859, pruned_loss=0.007447, audio_tagging_loss=0.01215, over 14294.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09185, pruned_loss=0.01372, audio_tagging_loss=0.009194, over 3048873.78 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:50:43,757 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377450
2023-11-23 19:51:14,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2516460.0, ans=0.125
2023-11-23 19:51:20,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2516460.0, ans=0.1
2023-11-23 19:51:28,164 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.521e+01 9.088e+01 9.578e+01 1.245e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 19:51:34,063 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4750, loss[loss=0.05953, simple_loss=0.07563, pruned_loss=0.01032, audio_tagging_loss=0.0114, over 16605.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09239, pruned_loss=0.01367, audio_tagging_loss=0.009173, over 3049764.20 frames. ], batch size: 69, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:51:42,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2516593.3333333335, ans=0.125
2023-11-23 19:51:45,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377500
2023-11-23 19:51:49,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2516660.0, ans=0.0
2023-11-23 19:52:04,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2516726.6666666665, ans=0.2
2023-11-23 19:52:15,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2516793.3333333335, ans=0.125
2023-11-23 19:52:20,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2516793.3333333335, ans=0.2
2023-11-23 19:52:27,397 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.86 vs. limit=15.0
2023-11-23 19:52:29,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2516860.0, ans=0.0
2023-11-23 19:52:36,087 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4800, loss[loss=0.06227, simple_loss=0.08024, pruned_loss=0.0119, audio_tagging_loss=0.01025, over 16283.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09039, pruned_loss=0.01339, audio_tagging_loss=0.009368, over 3049466.34 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:52:47,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377550
2023-11-23 19:52:56,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2516993.3333333335, ans=0.1
2023-11-23 19:53:23,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2517126.6666666665, ans=0.2
2023-11-23 19:53:32,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2517193.3333333335, ans=0.0
2023-11-23 19:53:33,629 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.424e+01 8.895e+01 9.657e+01 1.306e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 19:53:38,424 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4850, loss[loss=0.06569, simple_loss=0.08612, pruned_loss=0.0128, audio_tagging_loss=0.009825, over 16656.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.0908, pruned_loss=0.01334, audio_tagging_loss=0.009387, over 3054097.59 frames. ], batch size: 64, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:53:41,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2517260.0, ans=0.0
2023-11-23 19:53:41,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2517260.0, ans=0.125
2023-11-23 19:53:47,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2517260.0, ans=0.0
2023-11-23 19:53:49,678 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377600
2023-11-23 19:54:29,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2517526.6666666665, ans=0.1
2023-11-23 19:54:29,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.69 vs. limit=22.5
2023-11-23 19:54:40,504 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4900, loss[loss=0.05953, simple_loss=0.07729, pruned_loss=0.01239, audio_tagging_loss=0.008497, over 15414.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09014, pruned_loss=0.01331, audio_tagging_loss=0.009386, over 3038679.01 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:54:50,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2517593.3333333335, ans=0.125
2023-11-23 19:54:51,775 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377650
2023-11-23 19:55:02,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2517660.0, ans=0.0
2023-11-23 19:55:09,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2517726.6666666665, ans=0.0
2023-11-23 19:55:11,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2517726.6666666665, ans=0.07
2023-11-23 19:55:26,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2517793.3333333335, ans=0.2
2023-11-23 19:55:38,611 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.557e+01 8.293e+01 8.753e+01 9.560e+01 1.963e+02, threshold=1.751e+02, percent-clipped=1.0
2023-11-23 19:55:43,393 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 4950, loss[loss=0.06106, simple_loss=0.08125, pruned_loss=0.0123, audio_tagging_loss=0.008131, over 15436.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.0905, pruned_loss=0.01342, audio_tagging_loss=0.00922, over 3038965.05 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:55:52,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2517926.6666666665, ans=0.125
2023-11-23 19:55:54,834 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377700
2023-11-23 19:55:55,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2517993.3333333335, ans=0.125
2023-11-23 19:56:00,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2517993.3333333335, ans=0.0
2023-11-23 19:56:45,963 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5000, loss[loss=0.05594, simple_loss=0.07488, pruned_loss=0.009859, audio_tagging_loss=0.008644, over 14294.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09095, pruned_loss=0.01353, audio_tagging_loss=0.008987, over 3037028.62 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:56:54,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2518260.0, ans=0.125
2023-11-23 19:56:54,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2518260.0, ans=0.2
2023-11-23 19:56:58,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377750
2023-11-23 19:57:14,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2518393.3333333335, ans=0.2
2023-11-23 19:57:21,921 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.69 vs. limit=22.5
2023-11-23 19:57:25,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2518460.0, ans=0.125
2023-11-23 19:57:27,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2518460.0, ans=0.125
2023-11-23 19:57:29,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2518460.0, ans=0.1
2023-11-23 19:57:30,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2518460.0, ans=0.1
2023-11-23 19:57:35,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2518526.6666666665, ans=0.0
2023-11-23 19:57:43,974 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.445e+01 8.514e+01 9.045e+01 9.813e+01 1.165e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 19:57:48,791 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5050, loss[loss=0.06357, simple_loss=0.0869, pruned_loss=0.01332, audio_tagging_loss=0.006808, over 16775.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09131, pruned_loss=0.01369, audio_tagging_loss=0.008899, over 3040275.41 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:57:50,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2518593.3333333335, ans=0.125
2023-11-23 19:57:51,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.77 vs. limit=10.0
2023-11-23 19:58:00,106 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377800
2023-11-23 19:58:08,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2518660.0, ans=0.025
2023-11-23 19:58:28,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2518793.3333333335, ans=0.2
2023-11-23 19:58:35,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2518793.3333333335, ans=0.125
2023-11-23 19:58:50,378 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5100, loss[loss=0.07, simple_loss=0.105, pruned_loss=0.01213, audio_tagging_loss=0.005367, over 16462.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09122, pruned_loss=0.01363, audio_tagging_loss=0.008931, over 3040963.37 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:59:01,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377850
2023-11-23 19:59:06,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2518993.3333333335, ans=0.2
2023-11-23 19:59:21,355 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.28 vs. limit=22.5
2023-11-23 19:59:22,054 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:59:31,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2519126.6666666665, ans=0.1
2023-11-23 19:59:47,572 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.550e+01 9.235e+01 1.014e+02 1.258e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-23 19:59:52,255 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5150, loss[loss=0.06399, simple_loss=0.08739, pruned_loss=0.01157, audio_tagging_loss=0.008728, over 15391.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09061, pruned_loss=0.01338, audio_tagging_loss=0.009007, over 3046109.99 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:00:03,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377900
2023-11-23 20:00:05,711 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.87 vs. limit=15.0
2023-11-23 20:00:09,921 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:00:17,134 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.70 vs. limit=10.0
2023-11-23 20:00:21,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2519393.3333333335, ans=0.0
2023-11-23 20:00:26,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2519393.3333333335, ans=0.125
2023-11-23 20:00:41,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.19 vs. limit=15.0
2023-11-23 20:00:54,657 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5200, loss[loss=0.0752, simple_loss=0.1113, pruned_loss=0.01252, audio_tagging_loss=0.007014, over 15671.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09098, pruned_loss=0.01336, audio_tagging_loss=0.008982, over 3042509.31 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:00:55,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.68 vs. limit=15.0
2023-11-23 20:01:05,938 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 377950
2023-11-23 20:01:07,841 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.33 vs. limit=22.5
2023-11-23 20:01:12,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2519660.0, ans=0.0
2023-11-23 20:01:16,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2519660.0, ans=0.0
2023-11-23 20:01:21,376 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.12 vs. limit=22.5
2023-11-23 20:01:39,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2519793.3333333335, ans=0.1
2023-11-23 20:01:44,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2519860.0, ans=0.125
2023-11-23 20:01:51,892 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.115e+01 8.438e+01 9.195e+01 9.999e+01 1.195e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 20:01:53,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2519860.0, ans=0.125
2023-11-23 20:01:53,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2519860.0, ans=0.125
2023-11-23 20:01:56,761 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5250, loss[loss=0.09208, simple_loss=0.1285, pruned_loss=0.02235, audio_tagging_loss=0.005501, over 16321.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09176, pruned_loss=0.01347, audio_tagging_loss=0.008905, over 3045301.24 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:02:01,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2519926.6666666665, ans=0.0
2023-11-23 20:02:07,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378000
2023-11-23 20:02:08,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2519993.3333333335, ans=0.125
2023-11-23 20:02:13,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2519993.3333333335, ans=0.0
2023-11-23 20:02:15,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2519993.3333333335, ans=0.125
2023-11-23 20:02:32,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2520060.0, ans=0.0
2023-11-23 20:02:49,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2520193.3333333335, ans=0.2
2023-11-23 20:02:59,105 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5300, loss[loss=0.06938, simple_loss=0.09723, pruned_loss=0.01585, audio_tagging_loss=0.004913, over 15306.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09212, pruned_loss=0.01374, audio_tagging_loss=0.008849, over 3039386.86 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:03:10,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378050
2023-11-23 20:03:27,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2520393.3333333335, ans=0.0
2023-11-23 20:03:40,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2520460.0, ans=0.0
2023-11-23 20:03:48,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2520526.6666666665, ans=0.0
2023-11-23 20:03:56,740 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.515e+01 8.458e+01 9.027e+01 9.985e+01 1.662e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 20:04:01,554 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5350, loss[loss=0.05516, simple_loss=0.06988, pruned_loss=0.01022, audio_tagging_loss=0.01, over 14721.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09247, pruned_loss=0.01379, audio_tagging_loss=0.008922, over 3045154.24 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:04:13,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378100
2023-11-23 20:04:48,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2520793.3333333335, ans=15.0
2023-11-23 20:04:49,913 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.53 vs. limit=15.0
2023-11-23 20:04:59,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2520860.0, ans=0.125
2023-11-23 20:05:04,832 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5400, loss[loss=0.05347, simple_loss=0.06732, pruned_loss=0.00963, audio_tagging_loss=0.01018, over 14590.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09206, pruned_loss=0.01367, audio_tagging_loss=0.008941, over 3045596.52 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:05:06,612 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.83 vs. limit=10.0
2023-11-23 20:05:08,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2520926.6666666665, ans=0.125
2023-11-23 20:05:15,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378150
2023-11-23 20:05:33,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2521060.0, ans=0.125
2023-11-23 20:05:57,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2521193.3333333335, ans=0.0
2023-11-23 20:06:01,386 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.400e+01 9.032e+01 9.850e+01 1.216e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 20:06:03,409 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.13 vs. limit=15.0
2023-11-23 20:06:06,763 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5450, loss[loss=0.08298, simple_loss=0.1145, pruned_loss=0.01771, audio_tagging_loss=0.008006, over 14458.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09269, pruned_loss=0.01379, audio_tagging_loss=0.008972, over 3043863.16 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:06:17,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378200
2023-11-23 20:07:00,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2521526.6666666665, ans=0.125
2023-11-23 20:07:09,657 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5500, loss[loss=0.1108, simple_loss=0.1562, pruned_loss=0.02586, audio_tagging_loss=0.006801, over 14890.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09377, pruned_loss=0.01406, audio_tagging_loss=0.008989, over 3044866.67 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:07:20,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378250
2023-11-23 20:07:22,270 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.70 vs. limit=10.0
2023-11-23 20:07:26,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=15.0
2023-11-23 20:07:31,876 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.66 vs. limit=22.5
2023-11-23 20:07:33,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2521726.6666666665, ans=0.0
2023-11-23 20:07:38,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2521726.6666666665, ans=0.125
2023-11-23 20:07:38,754 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.79 vs. limit=10.0
2023-11-23 20:07:52,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2521793.3333333335, ans=0.0
2023-11-23 20:08:06,545 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.320e+01 8.896e+01 9.871e+01 1.353e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 20:08:12,058 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5550, loss[loss=0.05833, simple_loss=0.08152, pruned_loss=0.009205, audio_tagging_loss=0.008368, over 14787.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09358, pruned_loss=0.014, audio_tagging_loss=0.009105, over 3039160.37 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:08:23,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378300
2023-11-23 20:08:34,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2521993.3333333335, ans=0.125
2023-11-23 20:08:38,434 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.81 vs. limit=15.0
2023-11-23 20:08:40,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2522060.0, ans=0.035
2023-11-23 20:08:42,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2522060.0, ans=0.125
2023-11-23 20:09:12,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=2522193.3333333335, ans=15.0
2023-11-23 20:09:13,829 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5600, loss[loss=0.0734, simple_loss=0.09434, pruned_loss=0.01641, audio_tagging_loss=0.009828, over 14657.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.0939, pruned_loss=0.01404, audio_tagging_loss=0.009184, over 3045615.12 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:09:25,210 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378350
2023-11-23 20:09:34,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2522326.6666666665, ans=0.125
2023-11-23 20:09:34,432 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.85 vs. limit=15.0
2023-11-23 20:09:36,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2522326.6666666665, ans=0.125
2023-11-23 20:09:46,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0
2023-11-23 20:09:57,306 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:10:11,708 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.330e+01 9.147e+01 9.848e+01 1.519e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 20:10:15,351 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5650, loss[loss=0.07822, simple_loss=0.1073, pruned_loss=0.01705, audio_tagging_loss=0.007531, over 14686.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.0935, pruned_loss=0.01392, audio_tagging_loss=0.009229, over 3045638.70 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:10:18,549 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:10:19,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2522593.3333333335, ans=0.04949747468305833
2023-11-23 20:10:23,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2522593.3333333335, ans=0.1
2023-11-23 20:10:26,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378400
2023-11-23 20:10:58,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.96 vs. limit=15.0
2023-11-23 20:11:11,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2522860.0, ans=0.125
2023-11-23 20:11:17,840 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5700, loss[loss=0.06325, simple_loss=0.08547, pruned_loss=0.01109, audio_tagging_loss=0.009423, over 15803.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09308, pruned_loss=0.01374, audio_tagging_loss=0.009295, over 3045453.18 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:11:18,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2522926.6666666665, ans=0.125
2023-11-23 20:11:28,333 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378450
2023-11-23 20:11:35,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2522993.3333333335, ans=0.1
2023-11-23 20:12:06,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2523193.3333333335, ans=0.125
2023-11-23 20:12:12,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2523193.3333333335, ans=0.1
2023-11-23 20:12:14,661 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.181e+01 8.343e+01 8.908e+01 9.732e+01 1.345e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 20:12:18,283 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5750, loss[loss=0.07569, simple_loss=0.1002, pruned_loss=0.01851, audio_tagging_loss=0.007066, over 15990.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09282, pruned_loss=0.01387, audio_tagging_loss=0.009121, over 3053905.24 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:12:18,986 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.55 vs. limit=12.0
2023-11-23 20:12:29,534 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378500
2023-11-23 20:12:49,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.60 vs. limit=12.0
2023-11-23 20:12:56,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2523460.0, ans=0.125
2023-11-23 20:13:08,570 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.39 vs. limit=15.0
2023-11-23 20:13:09,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2523526.6666666665, ans=0.125
2023-11-23 20:13:18,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.08 vs. limit=15.0
2023-11-23 20:13:20,191 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5800, loss[loss=0.05042, simple_loss=0.06784, pruned_loss=0.008706, audio_tagging_loss=0.007793, over 16171.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09202, pruned_loss=0.01383, audio_tagging_loss=0.00907, over 3051150.18 frames. ], batch size: 64, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:13:20,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2523593.3333333335, ans=0.5
2023-11-23 20:13:31,201 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378550
2023-11-23 20:13:31,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2523660.0, ans=15.0
2023-11-23 20:13:37,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2523660.0, ans=0.0
2023-11-23 20:13:44,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2523726.6666666665, ans=0.125
2023-11-23 20:13:48,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.12 vs. limit=22.5
2023-11-23 20:14:03,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2523793.3333333335, ans=0.125
2023-11-23 20:14:13,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2523860.0, ans=0.125
2023-11-23 20:14:19,415 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.431e+01 8.343e+01 9.004e+01 9.818e+01 1.259e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 20:14:21,783 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5850, loss[loss=0.04262, simple_loss=0.05463, pruned_loss=0.006094, audio_tagging_loss=0.009208, over 15621.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09156, pruned_loss=0.01374, audio_tagging_loss=0.008978, over 3050857.43 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 8.0
2023-11-23 20:14:33,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378600
2023-11-23 20:14:44,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2523993.3333333335, ans=0.2
2023-11-23 20:14:45,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2524060.0, ans=0.1
2023-11-23 20:15:16,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2524193.3333333335, ans=0.0
2023-11-23 20:15:22,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2524193.3333333335, ans=0.125
2023-11-23 20:15:24,460 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5900, loss[loss=0.09417, simple_loss=0.1339, pruned_loss=0.01897, audio_tagging_loss=0.008234, over 15307.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09221, pruned_loss=0.01379, audio_tagging_loss=0.008882, over 3052237.12 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 8.0
2023-11-23 20:15:31,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2524260.0, ans=0.125
2023-11-23 20:15:35,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378650
2023-11-23 20:16:02,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2524460.0, ans=0.1
2023-11-23 20:16:04,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2524460.0, ans=0.125
2023-11-23 20:16:23,875 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.483e+01 8.986e+01 9.661e+01 2.610e+02, threshold=1.797e+02, percent-clipped=1.0
2023-11-23 20:16:26,889 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 5950, loss[loss=0.07349, simple_loss=0.1045, pruned_loss=0.01315, audio_tagging_loss=0.00807, over 15679.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.0924, pruned_loss=0.0137, audio_tagging_loss=0.008882, over 3053967.99 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 8.0
2023-11-23 20:16:30,018 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.68 vs. limit=12.0
2023-11-23 20:16:33,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2524593.3333333335, ans=0.0
2023-11-23 20:16:38,034 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378700
2023-11-23 20:16:40,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2524660.0, ans=0.125
2023-11-23 20:16:44,375 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:16:59,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.70 vs. limit=15.0
2023-11-23 20:17:15,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2524860.0, ans=0.125
2023-11-23 20:17:24,338 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0
2023-11-23 20:17:28,284 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6000, loss[loss=0.06718, simple_loss=0.08829, pruned_loss=0.01467, audio_tagging_loss=0.008356, over 14337.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09181, pruned_loss=0.01358, audio_tagging_loss=0.008916, over 3054669.76 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:17:28,285 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 20:18:07,451 INFO [train_asr.py:1253] (2/4) Epoch 32, validation: loss=0.05807, simple_loss=0.05104, pruned_loss=0.005144, audio_tagging_loss=0.02741, over 4681554.00 frames. 
2023-11-23 20:18:07,452 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 20:18:18,760 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378750
2023-11-23 20:18:20,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2524993.3333333335, ans=0.125
2023-11-23 20:18:23,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2524993.3333333335, ans=0.0
2023-11-23 20:18:52,182 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:18:57,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=2525193.3333333335, ans=22.5
2023-11-23 20:19:06,948 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.325e+01 9.027e+01 9.614e+01 1.325e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 20:19:10,002 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6050, loss[loss=0.0728, simple_loss=0.09558, pruned_loss=0.01457, audio_tagging_loss=0.01044, over 16293.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09226, pruned_loss=0.01366, audio_tagging_loss=0.00882, over 3055951.42 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:19:13,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2525260.0, ans=22.5
2023-11-23 20:19:19,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2525260.0, ans=15.0
2023-11-23 20:19:21,274 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378800
2023-11-23 20:19:21,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2525326.6666666665, ans=0.1
2023-11-23 20:19:37,016 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.54 vs. limit=22.5
2023-11-23 20:19:39,105 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:20:06,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2525526.6666666665, ans=0.125
2023-11-23 20:20:12,596 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6100, loss[loss=0.07045, simple_loss=0.1034, pruned_loss=0.01316, audio_tagging_loss=0.005584, over 15898.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09172, pruned_loss=0.01365, audio_tagging_loss=0.008858, over 3052942.76 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:20:18,751 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.87 vs. limit=6.0
2023-11-23 20:20:24,139 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378850
2023-11-23 20:20:29,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2525660.0, ans=0.125
2023-11-23 20:20:37,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.69 vs. limit=22.5
2023-11-23 20:20:39,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2525726.6666666665, ans=0.125
2023-11-23 20:20:43,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2525726.6666666665, ans=0.125
2023-11-23 20:21:12,673 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.468e+01 9.222e+01 1.005e+02 1.157e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-23 20:21:15,189 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6150, loss[loss=0.08218, simple_loss=0.1114, pruned_loss=0.01883, audio_tagging_loss=0.007673, over 16037.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09221, pruned_loss=0.0137, audio_tagging_loss=0.008919, over 3056806.13 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:21:26,615 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378900
2023-11-23 20:21:44,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.85 vs. limit=15.0
2023-11-23 20:21:54,349 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.05 vs. limit=15.0
2023-11-23 20:22:04,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2526193.3333333335, ans=0.125
2023-11-23 20:22:14,919 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:22:17,127 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6200, loss[loss=0.08342, simple_loss=0.0995, pruned_loss=0.02352, audio_tagging_loss=0.01015, over 14634.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09207, pruned_loss=0.0137, audio_tagging_loss=0.009008, over 3052738.21 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:22:28,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2526260.0, ans=0.125
2023-11-23 20:22:29,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 378950
2023-11-23 20:22:30,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2526326.6666666665, ans=0.09899494936611666
2023-11-23 20:23:02,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2526460.0, ans=0.1
2023-11-23 20:23:02,974 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.97 vs. limit=15.0
2023-11-23 20:23:17,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.457e+01 9.040e+01 9.939e+01 1.355e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-23 20:23:20,355 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6250, loss[loss=0.09828, simple_loss=0.1348, pruned_loss=0.02506, audio_tagging_loss=0.005837, over 15537.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09213, pruned_loss=0.01355, audio_tagging_loss=0.009172, over 3046210.68 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:23:31,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379000
2023-11-23 20:24:22,789 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6300, loss[loss=0.08569, simple_loss=0.1117, pruned_loss=0.02095, audio_tagging_loss=0.008906, over 15290.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09169, pruned_loss=0.01351, audio_tagging_loss=0.009214, over 3041440.12 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:24:31,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.97 vs. limit=10.0
2023-11-23 20:24:34,251 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379050
2023-11-23 20:24:43,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2526993.3333333335, ans=0.2
2023-11-23 20:24:45,230 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.53 vs. limit=15.0
2023-11-23 20:24:52,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2527060.0, ans=0.5
2023-11-23 20:25:04,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2527126.6666666665, ans=0.125
2023-11-23 20:25:08,037 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.52 vs. limit=22.5
2023-11-23 20:25:13,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2527193.3333333335, ans=0.09899494936611666
2023-11-23 20:25:20,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2527193.3333333335, ans=0.2
2023-11-23 20:25:21,985 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.520e+01 9.067e+01 9.897e+01 1.384e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 20:25:24,463 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6350, loss[loss=0.07004, simple_loss=0.09217, pruned_loss=0.01367, audio_tagging_loss=0.01029, over 15745.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09117, pruned_loss=0.01345, audio_tagging_loss=0.009286, over 3042607.02 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:25:24,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2527260.0, ans=0.1
2023-11-23 20:25:35,671 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379100
2023-11-23 20:25:43,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2527326.6666666665, ans=0.1
2023-11-23 20:25:58,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2527393.3333333335, ans=0.2
2023-11-23 20:25:58,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0
2023-11-23 20:26:07,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2527460.0, ans=0.04949747468305833
2023-11-23 20:26:27,042 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6400, loss[loss=0.07284, simple_loss=0.09964, pruned_loss=0.01607, audio_tagging_loss=0.006948, over 15786.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.0913, pruned_loss=0.01351, audio_tagging_loss=0.00936, over 3038202.34 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:26:28,984 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=13.03 vs. limit=15.0
2023-11-23 20:26:29,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2527593.3333333335, ans=0.04949747468305833
2023-11-23 20:26:33,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2527593.3333333335, ans=0.0
2023-11-23 20:26:38,941 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379150
2023-11-23 20:27:02,875 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.22 vs. limit=15.0
2023-11-23 20:27:12,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2527793.3333333335, ans=0.1
2023-11-23 20:27:27,506 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.271e+01 8.849e+01 9.635e+01 1.697e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 20:27:29,884 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6450, loss[loss=0.06666, simple_loss=0.08479, pruned_loss=0.01272, audio_tagging_loss=0.01155, over 15446.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09053, pruned_loss=0.01329, audio_tagging_loss=0.009416, over 3033006.52 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:27:37,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2527926.6666666665, ans=0.0
2023-11-23 20:27:41,490 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379200
2023-11-23 20:27:59,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2528060.0, ans=0.1
2023-11-23 20:28:12,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2528126.6666666665, ans=0.2
2023-11-23 20:28:17,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2528126.6666666665, ans=0.125
2023-11-23 20:28:24,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2528193.3333333335, ans=0.0
2023-11-23 20:28:32,330 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6500, loss[loss=0.07, simple_loss=0.08657, pruned_loss=0.01756, audio_tagging_loss=0.009155, over 15319.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09107, pruned_loss=0.01337, audio_tagging_loss=0.009444, over 3043327.29 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:28:43,254 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379250
2023-11-23 20:28:45,797 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.35 vs. limit=15.0
2023-11-23 20:28:47,615 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:28:47,945 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.96 vs. limit=15.0
2023-11-23 20:28:54,202 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.19 vs. limit=15.0
2023-11-23 20:29:02,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2528393.3333333335, ans=0.0
2023-11-23 20:29:32,127 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.606e+01 9.202e+01 9.720e+01 1.328e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 20:29:35,066 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6550, loss[loss=0.07632, simple_loss=0.108, pruned_loss=0.01627, audio_tagging_loss=0.006067, over 14808.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09139, pruned_loss=0.01348, audio_tagging_loss=0.009334, over 3047399.31 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:29:45,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2528593.3333333335, ans=0.125
2023-11-23 20:29:46,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379300
2023-11-23 20:30:26,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2528860.0, ans=0.125
2023-11-23 20:30:37,592 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6600, loss[loss=0.06862, simple_loss=0.0922, pruned_loss=0.01489, audio_tagging_loss=0.007629, over 14622.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09248, pruned_loss=0.01364, audio_tagging_loss=0.009205, over 3046148.22 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:30:48,444 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379350
2023-11-23 20:30:53,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2528993.3333333335, ans=0.125
2023-11-23 20:31:22,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2529126.6666666665, ans=0.125
2023-11-23 20:31:28,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.66 vs. limit=15.0
2023-11-23 20:31:30,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2529193.3333333335, ans=0.125
2023-11-23 20:31:35,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.71 vs. limit=10.0
2023-11-23 20:31:37,894 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.263e+01 8.994e+01 9.633e+01 1.189e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 20:31:40,322 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6650, loss[loss=0.0691, simple_loss=0.08867, pruned_loss=0.01455, audio_tagging_loss=0.01021, over 14803.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09142, pruned_loss=0.01351, audio_tagging_loss=0.009138, over 3048042.29 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:31:51,010 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379400
2023-11-23 20:32:37,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2529526.6666666665, ans=0.0
2023-11-23 20:32:42,227 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6700, loss[loss=0.06883, simple_loss=0.09075, pruned_loss=0.01242, audio_tagging_loss=0.01103, over 14549.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09224, pruned_loss=0.01356, audio_tagging_loss=0.009153, over 3044436.83 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:32:53,742 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379450
2023-11-23 20:33:17,129 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:33:24,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2529793.3333333335, ans=0.125
2023-11-23 20:33:39,419 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.27 vs. limit=12.0
2023-11-23 20:33:42,133 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.739e+01 8.296e+01 8.825e+01 9.532e+01 1.402e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 20:33:45,226 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6750, loss[loss=0.06074, simple_loss=0.07588, pruned_loss=0.01205, audio_tagging_loss=0.01075, over 15700.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09185, pruned_loss=0.01354, audio_tagging_loss=0.009077, over 3044666.32 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:33:50,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2529926.6666666665, ans=0.0
2023-11-23 20:33:53,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2529926.6666666665, ans=0.1
2023-11-23 20:33:56,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379500
2023-11-23 20:34:19,881 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.49 vs. limit=8.0
2023-11-23 20:34:31,934 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:34:47,420 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6800, loss[loss=0.06491, simple_loss=0.09286, pruned_loss=0.01164, audio_tagging_loss=0.006847, over 15105.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09276, pruned_loss=0.01366, audio_tagging_loss=0.008951, over 3049639.51 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:34:58,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379550
2023-11-23 20:35:04,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2530326.6666666665, ans=0.125
2023-11-23 20:35:05,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2530326.6666666665, ans=0.0
2023-11-23 20:35:07,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2530326.6666666665, ans=0.1
2023-11-23 20:35:09,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.03 vs. limit=10.0
2023-11-23 20:35:24,082 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-23 20:35:36,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=15.0
2023-11-23 20:35:40,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2530526.6666666665, ans=0.125
2023-11-23 20:35:45,192 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0
2023-11-23 20:35:47,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2530526.6666666665, ans=0.125
2023-11-23 20:35:48,136 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.273e+01 8.968e+01 9.529e+01 1.211e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 20:35:49,317 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6850, loss[loss=0.06618, simple_loss=0.07797, pruned_loss=0.01681, audio_tagging_loss=0.01038, over 15508.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09212, pruned_loss=0.01369, audio_tagging_loss=0.008964, over 3046686.05 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:36:00,773 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379600
2023-11-23 20:36:04,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2530660.0, ans=0.125
2023-11-23 20:36:51,603 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6900, loss[loss=0.06712, simple_loss=0.08879, pruned_loss=0.01307, audio_tagging_loss=0.009648, over 15112.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.0917, pruned_loss=0.0136, audio_tagging_loss=0.009009, over 3056826.85 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:36:53,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2530926.6666666665, ans=0.125
2023-11-23 20:37:02,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379650
2023-11-23 20:37:06,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2530993.3333333335, ans=0.125
2023-11-23 20:37:11,833 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.71 vs. limit=12.0
2023-11-23 20:37:30,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2531126.6666666665, ans=0.0
2023-11-23 20:37:38,831 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:37:46,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2531193.3333333335, ans=0.0
2023-11-23 20:37:52,176 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.634e+01 8.329e+01 8.996e+01 9.675e+01 1.152e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 20:37:52,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2531260.0, ans=0.125
2023-11-23 20:37:53,382 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 6950, loss[loss=0.05895, simple_loss=0.07304, pruned_loss=0.01128, audio_tagging_loss=0.01115, over 15731.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09117, pruned_loss=0.01331, audio_tagging_loss=0.009049, over 3051793.79 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:37:54,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2531260.0, ans=0.125
2023-11-23 20:38:01,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2531260.0, ans=0.125
2023-11-23 20:38:03,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2531260.0, ans=0.125
2023-11-23 20:38:04,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379700
2023-11-23 20:38:07,664 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.76 vs. limit=12.0
2023-11-23 20:38:18,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2531393.3333333335, ans=0.1
2023-11-23 20:38:27,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2531393.3333333335, ans=0.05
2023-11-23 20:38:28,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2531393.3333333335, ans=0.0
2023-11-23 20:38:34,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2531460.0, ans=10.0
2023-11-23 20:38:48,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2531526.6666666665, ans=0.0
2023-11-23 20:38:55,292 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7000, loss[loss=0.06147, simple_loss=0.08244, pruned_loss=0.009924, audio_tagging_loss=0.01033, over 16815.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09109, pruned_loss=0.01328, audio_tagging_loss=0.009037, over 3054861.97 frames. ], batch size: 65, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:39:06,347 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379750
2023-11-23 20:39:08,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2531660.0, ans=0.125
2023-11-23 20:39:15,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2531660.0, ans=0.125
2023-11-23 20:39:16,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2531660.0, ans=0.1
2023-11-23 20:39:24,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2531726.6666666665, ans=0.1
2023-11-23 20:39:26,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.99 vs. limit=15.0
2023-11-23 20:39:35,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2531793.3333333335, ans=0.0
2023-11-23 20:39:35,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2531793.3333333335, ans=0.2
2023-11-23 20:39:48,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.80 vs. limit=10.0
2023-11-23 20:39:55,965 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.253e+01 8.867e+01 9.647e+01 1.261e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 20:39:57,192 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7050, loss[loss=0.0792, simple_loss=0.1087, pruned_loss=0.01866, audio_tagging_loss=0.006185, over 15195.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09065, pruned_loss=0.01327, audio_tagging_loss=0.009213, over 3055089.93 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:40:08,503 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379800
2023-11-23 20:40:13,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2531993.3333333335, ans=0.0
2023-11-23 20:40:17,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2531993.3333333335, ans=0.1
2023-11-23 20:40:40,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2532126.6666666665, ans=0.125
2023-11-23 20:40:43,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2532126.6666666665, ans=0.125
2023-11-23 20:40:50,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2532193.3333333335, ans=0.1
2023-11-23 20:40:54,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2532193.3333333335, ans=0.0
2023-11-23 20:40:59,237 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7100, loss[loss=0.08602, simple_loss=0.1189, pruned_loss=0.01848, audio_tagging_loss=0.008071, over 15668.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09131, pruned_loss=0.01335, audio_tagging_loss=0.009267, over 3062599.12 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:41:00,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2532260.0, ans=0.2
2023-11-23 20:41:11,189 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379850
2023-11-23 20:41:38,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2532460.0, ans=0.2
2023-11-23 20:41:41,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.39 vs. limit=12.0
2023-11-23 20:42:01,915 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.887e+01 8.490e+01 9.157e+01 9.882e+01 1.439e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 20:42:01,962 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7150, loss[loss=0.05971, simple_loss=0.07536, pruned_loss=0.01071, audio_tagging_loss=0.01132, over 14754.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09225, pruned_loss=0.01357, audio_tagging_loss=0.009251, over 3065306.17 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:42:13,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379900
2023-11-23 20:42:14,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2532660.0, ans=0.2
2023-11-23 20:42:38,022 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:43:04,351 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7200, loss[loss=0.07114, simple_loss=0.09802, pruned_loss=0.01533, audio_tagging_loss=0.006799, over 14877.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09193, pruned_loss=0.01353, audio_tagging_loss=0.009296, over 3059175.03 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:43:12,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2532926.6666666665, ans=0.125
2023-11-23 20:43:15,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 379950
2023-11-23 20:43:32,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2533060.0, ans=0.025
2023-11-23 20:43:58,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.98 vs. limit=15.0
2023-11-23 20:44:00,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2533193.3333333335, ans=0.125
2023-11-23 20:44:05,867 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7250, loss[loss=0.07437, simple_loss=0.1071, pruned_loss=0.01306, audio_tagging_loss=0.007746, over 14472.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09275, pruned_loss=0.01368, audio_tagging_loss=0.009327, over 3060618.65 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:44:06,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.235e+01 8.202e+01 8.739e+01 9.383e+01 1.704e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 20:44:12,040 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:44:17,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380000
2023-11-23 20:44:48,212 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.54 vs. limit=12.0
2023-11-23 20:44:52,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2533460.0, ans=0.1
2023-11-23 20:44:57,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.49 vs. limit=15.0
2023-11-23 20:45:10,595 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7300, loss[loss=0.07392, simple_loss=0.1073, pruned_loss=0.01387, audio_tagging_loss=0.006405, over 15510.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09244, pruned_loss=0.01367, audio_tagging_loss=0.009202, over 3051047.97 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:45:12,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2533593.3333333335, ans=0.125
2023-11-23 20:45:21,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2533593.3333333335, ans=0.0
2023-11-23 20:45:22,447 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380050
2023-11-23 20:45:36,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2533726.6666666665, ans=0.125
2023-11-23 20:45:48,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2533793.3333333335, ans=0.2
2023-11-23 20:45:54,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2533793.3333333335, ans=0.1
2023-11-23 20:46:05,138 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:46:09,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2533860.0, ans=0.125
2023-11-23 20:46:14,071 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7350, loss[loss=0.08775, simple_loss=0.1285, pruned_loss=0.01586, audio_tagging_loss=0.007623, over 15179.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09411, pruned_loss=0.01402, audio_tagging_loss=0.008915, over 3054086.60 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:46:15,161 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.465e+01 9.022e+01 1.001e+02 1.675e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 20:46:25,452 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380100
2023-11-23 20:46:25,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2533993.3333333335, ans=0.0
2023-11-23 20:46:26,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2533993.3333333335, ans=0.125
2023-11-23 20:46:29,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2533993.3333333335, ans=0.0
2023-11-23 20:46:42,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2534060.0, ans=0.125
2023-11-23 20:46:43,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2534060.0, ans=0.125
2023-11-23 20:46:50,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2534126.6666666665, ans=0.125
2023-11-23 20:47:06,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2534193.3333333335, ans=0.2
2023-11-23 20:47:06,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2534193.3333333335, ans=0.125
2023-11-23 20:47:16,159 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7400, loss[loss=0.04598, simple_loss=0.05262, pruned_loss=0.007944, audio_tagging_loss=0.01173, over 14575.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09293, pruned_loss=0.01381, audio_tagging_loss=0.00888, over 3056052.08 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:47:27,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380150
2023-11-23 20:47:36,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2534326.6666666665, ans=0.1
2023-11-23 20:48:01,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2534460.0, ans=0.0
2023-11-23 20:48:18,445 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7450, loss[loss=0.06089, simple_loss=0.07333, pruned_loss=0.01394, audio_tagging_loss=0.01028, over 14439.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09224, pruned_loss=0.01378, audio_tagging_loss=0.008866, over 3051517.28 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:48:19,600 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.435e+01 9.167e+01 9.796e+01 1.283e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 20:48:29,288 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380200
2023-11-23 20:49:07,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2534860.0, ans=0.0
2023-11-23 20:49:14,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2534860.0, ans=0.0
2023-11-23 20:49:20,999 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7500, loss[loss=0.07521, simple_loss=0.1051, pruned_loss=0.01603, audio_tagging_loss=0.006636, over 14450.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09231, pruned_loss=0.0138, audio_tagging_loss=0.008845, over 3043297.61 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:49:23,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2534926.6666666665, ans=0.125
2023-11-23 20:49:28,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2534926.6666666665, ans=0.125
2023-11-23 20:49:30,461 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.48 vs. limit=15.0
2023-11-23 20:49:32,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380250
2023-11-23 20:50:11,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2535193.3333333335, ans=0.1
2023-11-23 20:50:13,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2535193.3333333335, ans=0.2
2023-11-23 20:50:15,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2535193.3333333335, ans=0.125
2023-11-23 20:50:20,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2535193.3333333335, ans=0.1
2023-11-23 20:50:22,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2535260.0, ans=0.125
2023-11-23 20:50:23,263 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7550, loss[loss=0.07082, simple_loss=0.09017, pruned_loss=0.01804, audio_tagging_loss=0.007698, over 15727.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09178, pruned_loss=0.01376, audio_tagging_loss=0.00885, over 3039350.78 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:50:24,461 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.259e+01 8.879e+01 9.874e+01 1.226e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 20:50:27,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2535260.0, ans=0.125
2023-11-23 20:50:34,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380300
2023-11-23 20:50:39,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2535326.6666666665, ans=0.125
2023-11-23 20:51:13,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2535526.6666666665, ans=0.125
2023-11-23 20:51:25,743 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7600, loss[loss=0.06142, simple_loss=0.08123, pruned_loss=0.01325, audio_tagging_loss=0.00756, over 16535.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09154, pruned_loss=0.01373, audio_tagging_loss=0.008853, over 3040782.50 frames. ], batch size: 63, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:51:32,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2535593.3333333335, ans=0.125
2023-11-23 20:51:36,506 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380350
2023-11-23 20:51:39,012 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:51:50,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2535726.6666666665, ans=0.2
2023-11-23 20:52:08,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2535793.3333333335, ans=0.125
2023-11-23 20:52:27,484 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7650, loss[loss=0.05631, simple_loss=0.06949, pruned_loss=0.01203, audio_tagging_loss=0.009538, over 14853.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09143, pruned_loss=0.01362, audio_tagging_loss=0.008843, over 3041774.56 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:52:29,183 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.957e+01 8.327e+01 8.912e+01 9.610e+01 1.311e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 20:52:39,390 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380400
2023-11-23 20:52:45,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2535993.3333333335, ans=0.125
2023-11-23 20:53:04,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2536126.6666666665, ans=0.125
2023-11-23 20:53:23,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.62 vs. limit=15.0
2023-11-23 20:53:31,357 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7700, loss[loss=0.06609, simple_loss=0.08186, pruned_loss=0.01537, audio_tagging_loss=0.009786, over 15284.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09173, pruned_loss=0.01376, audio_tagging_loss=0.008891, over 3029316.07 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:53:35,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.70 vs. limit=15.0
2023-11-23 20:53:37,989 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.93 vs. limit=15.0
2023-11-23 20:53:42,045 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380450
2023-11-23 20:54:10,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2536460.0, ans=0.07
2023-11-23 20:54:22,878 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.23 vs. limit=22.5
2023-11-23 20:54:24,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2536526.6666666665, ans=0.125
2023-11-23 20:54:28,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2536526.6666666665, ans=0.125
2023-11-23 20:54:32,978 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7750, loss[loss=0.06019, simple_loss=0.08009, pruned_loss=0.01039, audio_tagging_loss=0.009757, over 15259.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09269, pruned_loss=0.0139, audio_tagging_loss=0.008839, over 3032525.19 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:54:33,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2536593.3333333335, ans=0.0
2023-11-23 20:54:34,648 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.446e+01 9.126e+01 9.765e+01 1.172e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 20:54:44,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380500
2023-11-23 20:55:09,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0
2023-11-23 20:55:18,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2536793.3333333335, ans=0.1
2023-11-23 20:55:34,686 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7800, loss[loss=0.06638, simple_loss=0.08232, pruned_loss=0.01591, audio_tagging_loss=0.009312, over 14230.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09152, pruned_loss=0.01359, audio_tagging_loss=0.008808, over 3040847.21 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:55:46,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380550
2023-11-23 20:55:52,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2536993.3333333335, ans=0.125
2023-11-23 20:55:52,504 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.21 vs. limit=10.0
2023-11-23 20:56:04,194 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.58 vs. limit=15.0
2023-11-23 20:56:05,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2537060.0, ans=0.125
2023-11-23 20:56:18,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2537126.6666666665, ans=0.1
2023-11-23 20:56:21,046 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:56:37,562 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7850, loss[loss=0.05988, simple_loss=0.07644, pruned_loss=0.01259, audio_tagging_loss=0.009073, over 14467.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09144, pruned_loss=0.01351, audio_tagging_loss=0.008953, over 3043120.80 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:56:38,727 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.376e+01 9.070e+01 9.715e+01 1.480e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-23 20:56:40,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2537260.0, ans=0.125
2023-11-23 20:56:40,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2537260.0, ans=0.05
2023-11-23 20:56:49,060 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380600
2023-11-23 20:56:50,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2537326.6666666665, ans=0.125
2023-11-23 20:56:51,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2537326.6666666665, ans=0.125
2023-11-23 20:57:06,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2537393.3333333335, ans=0.07
2023-11-23 20:57:20,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2537460.0, ans=0.0
2023-11-23 20:57:24,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2537460.0, ans=0.035
2023-11-23 20:57:28,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2537526.6666666665, ans=0.0
2023-11-23 20:57:40,337 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7900, loss[loss=0.07244, simple_loss=0.0975, pruned_loss=0.01452, audio_tagging_loss=0.009174, over 15158.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09108, pruned_loss=0.01358, audio_tagging_loss=0.00902, over 3040361.94 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:57:41,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.68 vs. limit=15.0
2023-11-23 20:57:51,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380650
2023-11-23 20:58:24,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2537793.3333333335, ans=0.125
2023-11-23 20:58:42,645 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 7950, loss[loss=0.07076, simple_loss=0.09814, pruned_loss=0.01542, audio_tagging_loss=0.006262, over 15114.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09022, pruned_loss=0.01326, audio_tagging_loss=0.009227, over 3045552.67 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:58:43,739 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.414e+01 9.153e+01 9.684e+01 1.303e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 20:58:48,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2537926.6666666665, ans=0.0
2023-11-23 20:58:53,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2537926.6666666665, ans=0.0
2023-11-23 20:58:54,040 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380700
2023-11-23 20:58:57,539 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:59:30,144 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:59:44,786 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8000, loss[loss=0.08456, simple_loss=0.1092, pruned_loss=0.02005, audio_tagging_loss=0.009913, over 15860.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.0898, pruned_loss=0.01325, audio_tagging_loss=0.009314, over 3050080.00 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:59:56,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380750
2023-11-23 21:00:24,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2538460.0, ans=0.0
2023-11-23 21:00:47,832 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8050, loss[loss=0.05884, simple_loss=0.07503, pruned_loss=0.01041, audio_tagging_loss=0.01091, over 14200.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.08938, pruned_loss=0.01312, audio_tagging_loss=0.009269, over 3043363.31 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 21:00:48,940 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.054e+01 8.478e+01 9.043e+01 9.667e+01 1.192e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 21:00:59,101 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380800
2023-11-23 21:01:15,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2538726.6666666665, ans=0.1
2023-11-23 21:01:18,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2538726.6666666665, ans=0.1
2023-11-23 21:01:50,367 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8100, loss[loss=0.06981, simple_loss=0.09458, pruned_loss=0.01349, audio_tagging_loss=0.009025, over 15207.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.08961, pruned_loss=0.01329, audio_tagging_loss=0.009236, over 3038968.89 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:01:53,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2538926.6666666665, ans=0.125
2023-11-23 21:02:01,486 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380850
2023-11-23 21:02:05,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2538993.3333333335, ans=0.125
2023-11-23 21:02:35,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2539126.6666666665, ans=0.1
2023-11-23 21:02:42,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2539193.3333333335, ans=0.1
2023-11-23 21:02:50,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2539193.3333333335, ans=0.125
2023-11-23 21:02:52,170 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8150, loss[loss=0.07178, simple_loss=0.1029, pruned_loss=0.01233, audio_tagging_loss=0.008005, over 16364.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08904, pruned_loss=0.01319, audio_tagging_loss=0.009193, over 3044327.00 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:02:54,490 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.341e+01 9.005e+01 9.405e+01 1.221e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 21:03:03,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380900
2023-11-23 21:03:06,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2539326.6666666665, ans=0.0
2023-11-23 21:03:33,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2539460.0, ans=0.0
2023-11-23 21:03:35,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2539460.0, ans=0.09899494936611666
2023-11-23 21:03:39,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2539460.0, ans=0.125
2023-11-23 21:03:45,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2539526.6666666665, ans=0.0
2023-11-23 21:03:54,164 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8200, loss[loss=0.07495, simple_loss=0.1053, pruned_loss=0.01247, audio_tagging_loss=0.009802, over 14679.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.08952, pruned_loss=0.01325, audio_tagging_loss=0.009043, over 3039902.88 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:03:55,303 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:04:05,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 380950
2023-11-23 21:04:09,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2539660.0, ans=0.125
2023-11-23 21:04:21,592 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.18 vs. limit=22.5
2023-11-23 21:04:35,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2539793.3333333335, ans=0.07
2023-11-23 21:04:42,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2539793.3333333335, ans=0.125
2023-11-23 21:04:56,983 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8250, loss[loss=0.07616, simple_loss=0.1017, pruned_loss=0.01517, audio_tagging_loss=0.01012, over 14750.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.0899, pruned_loss=0.01318, audio_tagging_loss=0.00896, over 3048819.93 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:04:59,274 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.237e+01 8.988e+01 9.644e+01 1.224e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 21:04:59,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2539926.6666666665, ans=0.0
2023-11-23 21:05:08,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381000
2023-11-23 21:05:16,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2539993.3333333335, ans=0.125
2023-11-23 21:05:32,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.32 vs. limit=22.5
2023-11-23 21:05:49,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2540193.3333333335, ans=0.125
2023-11-23 21:05:59,292 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8300, loss[loss=0.05205, simple_loss=0.07203, pruned_loss=0.01011, audio_tagging_loss=0.005929, over 15855.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09071, pruned_loss=0.01338, audio_tagging_loss=0.009021, over 3055077.07 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:06:01,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2540260.0, ans=0.125
2023-11-23 21:06:02,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2540260.0, ans=0.125
2023-11-23 21:06:10,713 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381050
2023-11-23 21:06:12,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2540326.6666666665, ans=0.125
2023-11-23 21:06:18,451 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.43 vs. limit=12.0
2023-11-23 21:06:38,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2540460.0, ans=0.09899494936611666
2023-11-23 21:06:49,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2540526.6666666665, ans=0.0
2023-11-23 21:06:52,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2540526.6666666665, ans=0.125
2023-11-23 21:06:54,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2540526.6666666665, ans=0.1
2023-11-23 21:07:01,057 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8350, loss[loss=0.07673, simple_loss=0.1017, pruned_loss=0.01593, audio_tagging_loss=0.009966, over 15553.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.0913, pruned_loss=0.01347, audio_tagging_loss=0.008942, over 3046016.33 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:07:02,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2540593.3333333335, ans=0.0
2023-11-23 21:07:03,382 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.724e+01 8.451e+01 9.185e+01 9.824e+01 1.570e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 21:07:08,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2540593.3333333335, ans=0.0
2023-11-23 21:07:09,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.30 vs. limit=15.0
2023-11-23 21:07:11,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381100
2023-11-23 21:07:32,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2540726.6666666665, ans=0.1
2023-11-23 21:07:50,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2540860.0, ans=0.95
2023-11-23 21:07:52,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2540860.0, ans=0.125
2023-11-23 21:08:00,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2540860.0, ans=0.125
2023-11-23 21:08:01,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2540926.6666666665, ans=0.125
2023-11-23 21:08:02,555 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8400, loss[loss=0.06344, simple_loss=0.07989, pruned_loss=0.01161, audio_tagging_loss=0.01189, over 14463.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09063, pruned_loss=0.01327, audio_tagging_loss=0.008961, over 3040506.66 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:08:02,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2540926.6666666665, ans=0.125
2023-11-23 21:08:04,591 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.99 vs. limit=15.0
2023-11-23 21:08:13,815 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381150
2023-11-23 21:08:20,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2540993.3333333335, ans=0.125
2023-11-23 21:08:37,861 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.19 vs. limit=12.0
2023-11-23 21:08:44,972 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.96 vs. limit=15.0
2023-11-23 21:08:48,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2541126.6666666665, ans=0.0
2023-11-23 21:09:04,789 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8450, loss[loss=0.08245, simple_loss=0.1134, pruned_loss=0.0173, audio_tagging_loss=0.008466, over 16153.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09069, pruned_loss=0.01323, audio_tagging_loss=0.008931, over 3043679.52 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:09:07,961 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.24 vs. limit=22.5
2023-11-23 21:09:08,216 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.427e+01 8.936e+01 9.652e+01 1.220e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 21:09:09,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2541260.0, ans=0.0
2023-11-23 21:09:15,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381200
2023-11-23 21:09:17,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2541326.6666666665, ans=0.5
2023-11-23 21:09:20,498 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.72 vs. limit=15.0
2023-11-23 21:09:25,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2541326.6666666665, ans=0.1
2023-11-23 21:09:51,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2541460.0, ans=0.2
2023-11-23 21:10:06,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2541593.3333333335, ans=0.125
2023-11-23 21:10:07,171 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8500, loss[loss=0.07115, simple_loss=0.09698, pruned_loss=0.0127, audio_tagging_loss=0.009955, over 15350.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09137, pruned_loss=0.01339, audio_tagging_loss=0.008987, over 3042238.66 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:10:12,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2541593.3333333335, ans=0.0
2023-11-23 21:10:18,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381250
2023-11-23 21:10:30,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2541726.6666666665, ans=0.125
2023-11-23 21:10:33,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2541726.6666666665, ans=0.2
2023-11-23 21:10:35,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2541726.6666666665, ans=0.125
2023-11-23 21:11:09,327 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8550, loss[loss=0.06178, simple_loss=0.08467, pruned_loss=0.01115, audio_tagging_loss=0.008302, over 15031.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09078, pruned_loss=0.01319, audio_tagging_loss=0.009085, over 3042459.24 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:11:12,873 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.400e+01 9.293e+01 9.776e+01 1.237e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-23 21:11:20,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381300
2023-11-23 21:11:20,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2541993.3333333335, ans=0.125
2023-11-23 21:11:25,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2541993.3333333335, ans=0.09899494936611666
2023-11-23 21:11:46,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2542126.6666666665, ans=0.0
2023-11-23 21:12:07,956 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.27 vs. limit=10.0
2023-11-23 21:12:11,446 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8600, loss[loss=0.08287, simple_loss=0.1114, pruned_loss=0.01966, audio_tagging_loss=0.007483, over 15311.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09171, pruned_loss=0.01346, audio_tagging_loss=0.009042, over 3048495.61 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:12:16,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2542260.0, ans=10.0
2023-11-23 21:12:18,571 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.30 vs. limit=10.0
2023-11-23 21:12:22,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381350
2023-11-23 21:12:27,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.72 vs. limit=10.0
2023-11-23 21:12:59,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2542526.6666666665, ans=0.125
2023-11-23 21:13:13,092 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8650, loss[loss=0.07335, simple_loss=0.09836, pruned_loss=0.01754, audio_tagging_loss=0.006631, over 14906.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09176, pruned_loss=0.01355, audio_tagging_loss=0.009032, over 3047008.10 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:13:16,581 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.050e+01 8.560e+01 9.209e+01 9.798e+01 1.197e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-23 21:13:23,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381400
2023-11-23 21:13:30,360 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-23 21:14:02,950 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.39 vs. limit=12.0
2023-11-23 21:14:15,426 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8700, loss[loss=0.057, simple_loss=0.0716, pruned_loss=0.01054, audio_tagging_loss=0.01066, over 16007.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09197, pruned_loss=0.01361, audio_tagging_loss=0.009143, over 3047507.72 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:14:26,566 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381450
2023-11-23 21:14:28,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2542993.3333333335, ans=6.0
2023-11-23 21:14:34,654 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.14 vs. limit=10.0
2023-11-23 21:14:38,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2542993.3333333335, ans=0.0
2023-11-23 21:15:17,578 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8750, loss[loss=0.07494, simple_loss=0.09988, pruned_loss=0.01371, audio_tagging_loss=0.01129, over 15347.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09181, pruned_loss=0.01357, audio_tagging_loss=0.009259, over 3046616.70 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:15:21,702 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.453e+01 8.955e+01 9.752e+01 1.523e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 21:15:29,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381500
2023-11-23 21:15:38,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2543326.6666666665, ans=0.0
2023-11-23 21:15:44,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2543393.3333333335, ans=0.125
2023-11-23 21:16:11,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2543526.6666666665, ans=0.5
2023-11-23 21:16:20,094 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8800, loss[loss=0.08511, simple_loss=0.1177, pruned_loss=0.01679, audio_tagging_loss=0.009492, over 15045.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09281, pruned_loss=0.01381, audio_tagging_loss=0.009309, over 3046471.14 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:16:31,272 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381550
2023-11-23 21:16:35,191 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.79 vs. limit=15.0
2023-11-23 21:17:08,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.35 vs. limit=22.5
2023-11-23 21:17:12,164 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.46 vs. limit=15.0
2023-11-23 21:17:22,298 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8850, loss[loss=0.07211, simple_loss=0.09797, pruned_loss=0.01417, audio_tagging_loss=0.008957, over 14805.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09268, pruned_loss=0.0138, audio_tagging_loss=0.009267, over 3046322.32 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:17:25,952 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.726e+01 8.519e+01 9.142e+01 9.754e+01 1.117e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 21:17:29,372 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:17:33,913 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381600
2023-11-23 21:17:33,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2543993.3333333335, ans=0.1
2023-11-23 21:17:34,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2543993.3333333335, ans=0.125
2023-11-23 21:17:35,039 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:17:41,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2543993.3333333335, ans=0.05
2023-11-23 21:17:56,640 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.59 vs. limit=22.5
2023-11-23 21:17:56,644 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.46 vs. limit=15.0
2023-11-23 21:17:59,127 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:17:59,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2544060.0, ans=0.2
2023-11-23 21:18:25,677 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8900, loss[loss=0.06571, simple_loss=0.08329, pruned_loss=0.01109, audio_tagging_loss=0.01297, over 13937.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09311, pruned_loss=0.01387, audio_tagging_loss=0.009207, over 3047663.14 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:18:34,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2544260.0, ans=0.1
2023-11-23 21:18:36,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381650
2023-11-23 21:18:36,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2544326.6666666665, ans=0.125
2023-11-23 21:18:40,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2544326.6666666665, ans=0.1
2023-11-23 21:18:55,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2544393.3333333335, ans=0.0
2023-11-23 21:19:06,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2544460.0, ans=0.125
2023-11-23 21:19:08,010 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:19:17,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2544526.6666666665, ans=0.0
2023-11-23 21:19:21,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0
2023-11-23 21:19:22,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.29 vs. limit=15.0
2023-11-23 21:19:24,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2544526.6666666665, ans=0.0
2023-11-23 21:19:27,560 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 8950, loss[loss=0.0611, simple_loss=0.0761, pruned_loss=0.01483, audio_tagging_loss=0.008219, over 15763.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09427, pruned_loss=0.01392, audio_tagging_loss=0.008973, over 3057727.30 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:19:32,777 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.301e+01 8.310e+01 8.895e+01 9.557e+01 1.131e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 21:19:37,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2544593.3333333335, ans=0.125
2023-11-23 21:19:39,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381700
2023-11-23 21:20:20,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2544860.0, ans=0.125
2023-11-23 21:20:30,423 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9000, loss[loss=0.06368, simple_loss=0.08764, pruned_loss=0.01162, audio_tagging_loss=0.008235, over 16536.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09297, pruned_loss=0.01376, audio_tagging_loss=0.008942, over 3057196.65 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:20:30,424 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 21:21:10,445 INFO [train_asr.py:1253] (2/4) Epoch 32, validation: loss=0.05909, simple_loss=0.05099, pruned_loss=0.005136, audio_tagging_loss=0.02845, over 4681554.00 frames. 
2023-11-23 21:21:10,446 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 21:21:18,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2544926.6666666665, ans=0.07
2023-11-23 21:21:21,698 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381750
2023-11-23 21:21:23,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2544993.3333333335, ans=0.2
2023-11-23 21:21:32,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2544993.3333333335, ans=0.125
2023-11-23 21:21:44,908 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.80 vs. limit=6.0
2023-11-23 21:22:05,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2545193.3333333335, ans=0.0
2023-11-23 21:22:12,943 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9050, loss[loss=0.07681, simple_loss=0.1131, pruned_loss=0.01278, audio_tagging_loss=0.007472, over 14577.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09209, pruned_loss=0.01351, audio_tagging_loss=0.008966, over 3059726.73 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:22:17,518 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.661e+01 9.152e+01 9.951e+01 1.252e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 21:22:20,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2545260.0, ans=0.1
2023-11-23 21:22:24,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381800
2023-11-23 21:23:02,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2545526.6666666665, ans=0.1
2023-11-23 21:23:15,520 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9100, loss[loss=0.09246, simple_loss=0.1133, pruned_loss=0.02544, audio_tagging_loss=0.01038, over 14610.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09147, pruned_loss=0.01347, audio_tagging_loss=0.008954, over 3059703.43 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:23:15,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2545593.3333333335, ans=0.125
2023-11-23 21:23:26,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381850
2023-11-23 21:23:30,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2545660.0, ans=0.125
2023-11-23 21:23:33,279 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.67 vs. limit=10.0
2023-11-23 21:23:37,846 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.35 vs. limit=8.0
2023-11-23 21:23:39,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2545726.6666666665, ans=0.125
2023-11-23 21:23:51,363 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0
2023-11-23 21:23:57,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2545793.3333333335, ans=15.0
2023-11-23 21:24:11,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2545860.0, ans=0.0
2023-11-23 21:24:17,815 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9150, loss[loss=0.05048, simple_loss=0.06228, pruned_loss=0.008064, audio_tagging_loss=0.01128, over 15629.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09293, pruned_loss=0.01359, audio_tagging_loss=0.008831, over 3066384.33 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:24:23,234 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.696e+01 8.321e+01 8.921e+01 9.557e+01 1.166e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 21:24:26,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.81 vs. limit=15.0
2023-11-23 21:24:29,243 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381900
2023-11-23 21:24:30,941 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.47 vs. limit=12.0
2023-11-23 21:24:32,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2545993.3333333335, ans=0.1
2023-11-23 21:24:50,146 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.67 vs. limit=10.0
2023-11-23 21:24:50,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2546060.0, ans=0.125
2023-11-23 21:25:02,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2546126.6666666665, ans=0.0
2023-11-23 21:25:10,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2546193.3333333335, ans=0.2
2023-11-23 21:25:20,075 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9200, loss[loss=0.06476, simple_loss=0.09764, pruned_loss=0.01036, audio_tagging_loss=0.005574, over 14399.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09173, pruned_loss=0.01329, audio_tagging_loss=0.008794, over 3053434.98 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:25:31,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 381950
2023-11-23 21:25:39,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2546326.6666666665, ans=0.0
2023-11-23 21:25:50,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2546393.3333333335, ans=0.0
2023-11-23 21:25:54,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2546393.3333333335, ans=0.1
2023-11-23 21:26:07,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2546460.0, ans=0.125
2023-11-23 21:26:11,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2546526.6666666665, ans=0.125
2023-11-23 21:26:11,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2546526.6666666665, ans=0.125
2023-11-23 21:26:22,418 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9250, loss[loss=0.07778, simple_loss=0.1041, pruned_loss=0.01674, audio_tagging_loss=0.009002, over 14435.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09157, pruned_loss=0.01333, audio_tagging_loss=0.008849, over 3053012.66 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:26:24,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2546593.3333333335, ans=0.125
2023-11-23 21:26:28,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2546593.3333333335, ans=0.125
2023-11-23 21:26:28,944 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.321e+01 8.948e+01 9.831e+01 1.146e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 21:26:33,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382000
2023-11-23 21:27:14,200 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.67 vs. limit=10.0
2023-11-23 21:27:25,739 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9300, loss[loss=0.08292, simple_loss=0.108, pruned_loss=0.01805, audio_tagging_loss=0.01089, over 15331.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09149, pruned_loss=0.01326, audio_tagging_loss=0.008875, over 3048404.81 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:27:36,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382050
2023-11-23 21:27:58,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2547060.0, ans=0.1
2023-11-23 21:28:27,241 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9350, loss[loss=0.07448, simple_loss=0.1037, pruned_loss=0.01322, audio_tagging_loss=0.0094, over 15258.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09152, pruned_loss=0.01337, audio_tagging_loss=0.008921, over 3049146.96 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:28:33,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.368e+01 8.391e+01 8.927e+01 9.865e+01 1.174e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 21:28:38,357 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382100
2023-11-23 21:28:45,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2547326.6666666665, ans=0.1
2023-11-23 21:28:59,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2547393.3333333335, ans=0.125
2023-11-23 21:29:04,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2547460.0, ans=0.0
2023-11-23 21:29:29,308 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9400, loss[loss=0.07082, simple_loss=0.08991, pruned_loss=0.01255, audio_tagging_loss=0.01331, over 15241.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09175, pruned_loss=0.01364, audio_tagging_loss=0.009125, over 3051730.50 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:29:40,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382150
2023-11-23 21:29:47,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2547660.0, ans=0.0
2023-11-23 21:30:10,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2547793.3333333335, ans=0.0
2023-11-23 21:30:29,120 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:30:31,447 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9450, loss[loss=0.05258, simple_loss=0.0682, pruned_loss=0.01062, audio_tagging_loss=0.007864, over 13919.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09151, pruned_loss=0.01356, audio_tagging_loss=0.009179, over 3053325.43 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:30:37,932 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.414e+01 9.183e+01 9.854e+01 1.204e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 21:30:43,342 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382200
2023-11-23 21:31:34,405 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9500, loss[loss=0.04131, simple_loss=0.05452, pruned_loss=0.003854, audio_tagging_loss=0.0102, over 15064.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09105, pruned_loss=0.01358, audio_tagging_loss=0.009217, over 3048063.19 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:31:45,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382250
2023-11-23 21:31:45,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2548326.6666666665, ans=0.1
2023-11-23 21:31:54,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2548326.6666666665, ans=0.125
2023-11-23 21:31:54,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.48 vs. limit=15.0
2023-11-23 21:32:17,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.05 vs. limit=15.0
2023-11-23 21:32:20,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2548460.0, ans=0.2
2023-11-23 21:32:21,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2548460.0, ans=0.0
2023-11-23 21:32:27,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2548526.6666666665, ans=0.0
2023-11-23 21:32:36,311 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9550, loss[loss=0.05378, simple_loss=0.07177, pruned_loss=0.009036, audio_tagging_loss=0.008857, over 15219.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09202, pruned_loss=0.01368, audio_tagging_loss=0.009202, over 3048402.88 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:32:42,273 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.598e+01 9.160e+01 9.932e+01 1.326e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-23 21:32:47,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382300
2023-11-23 21:32:53,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2548660.0, ans=0.0
2023-11-23 21:33:00,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2548726.6666666665, ans=0.0
2023-11-23 21:33:12,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2548793.3333333335, ans=0.125
2023-11-23 21:33:16,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2548793.3333333335, ans=0.0
2023-11-23 21:33:21,917 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.94 vs. limit=15.0
2023-11-23 21:33:22,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2548793.3333333335, ans=0.125
2023-11-23 21:33:22,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2548793.3333333335, ans=0.1
2023-11-23 21:33:23,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2548793.3333333335, ans=0.0
2023-11-23 21:33:32,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2548860.0, ans=0.125
2023-11-23 21:33:33,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2548860.0, ans=0.125
2023-11-23 21:33:37,961 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9600, loss[loss=0.09602, simple_loss=0.1345, pruned_loss=0.0205, audio_tagging_loss=0.00825, over 15981.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09261, pruned_loss=0.01372, audio_tagging_loss=0.00922, over 3049821.84 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:33:46,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2548926.6666666665, ans=0.0
2023-11-23 21:33:50,047 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382350
2023-11-23 21:34:10,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2549060.0, ans=0.125
2023-11-23 21:34:38,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2549193.3333333335, ans=0.125
2023-11-23 21:34:41,430 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9650, loss[loss=0.04595, simple_loss=0.05872, pruned_loss=0.00913, audio_tagging_loss=0.007454, over 15208.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09252, pruned_loss=0.01355, audio_tagging_loss=0.009023, over 3051987.03 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:34:47,294 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.285e+01 8.434e+01 9.003e+01 9.684e+01 1.226e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 21:34:52,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382400
2023-11-23 21:34:54,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2549326.6666666665, ans=0.125
2023-11-23 21:35:18,162 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.35 vs. limit=15.0
2023-11-23 21:35:27,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.77 vs. limit=22.5
2023-11-23 21:35:44,307 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9700, loss[loss=0.05755, simple_loss=0.07868, pruned_loss=0.01016, audio_tagging_loss=0.008055, over 13968.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09316, pruned_loss=0.01365, audio_tagging_loss=0.008898, over 3045578.02 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:35:55,005 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382450
2023-11-23 21:36:13,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.61 vs. limit=15.0
2023-11-23 21:36:27,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2549793.3333333335, ans=0.125
2023-11-23 21:36:29,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2549793.3333333335, ans=0.0
2023-11-23 21:36:45,375 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9750, loss[loss=0.07676, simple_loss=0.1109, pruned_loss=0.01551, audio_tagging_loss=0.005789, over 16195.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09328, pruned_loss=0.0137, audio_tagging_loss=0.008865, over 3047766.67 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:36:51,912 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.362e+01 8.986e+01 9.723e+01 1.186e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 21:36:56,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382500
2023-11-23 21:36:59,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2549993.3333333335, ans=0.125
2023-11-23 21:37:06,894 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:37:20,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2550060.0, ans=0.125
2023-11-23 21:37:47,723 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9800, loss[loss=0.07108, simple_loss=0.08907, pruned_loss=0.01739, audio_tagging_loss=0.009156, over 15765.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09355, pruned_loss=0.01377, audio_tagging_loss=0.008848, over 3044792.18 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:37:50,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2550260.0, ans=0.5
2023-11-23 21:37:59,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382550
2023-11-23 21:38:28,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2550460.0, ans=0.125
2023-11-23 21:38:36,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2550526.6666666665, ans=0.125
2023-11-23 21:38:37,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2550526.6666666665, ans=0.125
2023-11-23 21:38:42,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2550526.6666666665, ans=0.0
2023-11-23 21:38:43,004 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:38:50,708 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9850, loss[loss=0.06814, simple_loss=0.08677, pruned_loss=0.01333, audio_tagging_loss=0.01143, over 14672.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09354, pruned_loss=0.01394, audio_tagging_loss=0.00884, over 3045637.33 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:38:52,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2550593.3333333335, ans=0.125
2023-11-23 21:38:56,524 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.500e+01 9.067e+01 9.984e+01 1.563e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 21:38:58,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2550593.3333333335, ans=0.025
2023-11-23 21:39:01,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382600
2023-11-23 21:39:01,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2550660.0, ans=0.0
2023-11-23 21:39:07,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2550660.0, ans=0.125
2023-11-23 21:39:15,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2550726.6666666665, ans=0.125
2023-11-23 21:39:52,830 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9900, loss[loss=0.07023, simple_loss=0.09955, pruned_loss=0.01419, audio_tagging_loss=0.006267, over 14914.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09307, pruned_loss=0.01394, audio_tagging_loss=0.008831, over 3040432.21 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:40:04,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382650
2023-11-23 21:40:09,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2550993.3333333335, ans=0.125
2023-11-23 21:40:41,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2551126.6666666665, ans=0.5
2023-11-23 21:40:49,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2551193.3333333335, ans=0.1
2023-11-23 21:40:55,636 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 9950, loss[loss=0.05544, simple_loss=0.07525, pruned_loss=0.007873, audio_tagging_loss=0.009944, over 16294.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09337, pruned_loss=0.01387, audio_tagging_loss=0.00882, over 3040921.91 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:41:03,344 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.223e+01 9.080e+01 9.967e+01 1.331e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 21:41:07,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382700
2023-11-23 21:41:12,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2551326.6666666665, ans=0.04949747468305833
2023-11-23 21:41:19,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2551326.6666666665, ans=0.0
2023-11-23 21:41:39,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2551460.0, ans=0.0
2023-11-23 21:41:43,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2551460.0, ans=0.0
2023-11-23 21:41:48,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2551526.6666666665, ans=0.1
2023-11-23 21:41:59,256 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10000, loss[loss=0.06703, simple_loss=0.0812, pruned_loss=0.01684, audio_tagging_loss=0.009585, over 14390.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09312, pruned_loss=0.0138, audio_tagging_loss=0.008856, over 3047839.00 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:42:03,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2551593.3333333335, ans=0.125
2023-11-23 21:42:09,921 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382750
2023-11-23 21:42:27,680 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2551726.6666666665, ans=0.125
2023-11-23 21:42:47,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.87 vs. limit=10.0
2023-11-23 21:42:54,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2551860.0, ans=0.125
2023-11-23 21:43:00,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2551926.6666666665, ans=0.125
2023-11-23 21:43:00,961 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10050, loss[loss=0.06836, simple_loss=0.08883, pruned_loss=0.01463, audio_tagging_loss=0.009316, over 15431.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09219, pruned_loss=0.01363, audio_tagging_loss=0.008798, over 3044903.07 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:43:08,055 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.368e+01 8.922e+01 9.527e+01 1.170e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 21:43:11,930 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382800
2023-11-23 21:43:16,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2551993.3333333335, ans=0.125
2023-11-23 21:43:35,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2552060.0, ans=0.07
2023-11-23 21:43:57,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2552193.3333333335, ans=0.125
2023-11-23 21:44:01,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2552193.3333333335, ans=0.1
2023-11-23 21:44:03,493 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10100, loss[loss=0.07543, simple_loss=0.1027, pruned_loss=0.01552, audio_tagging_loss=0.008551, over 14960.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09224, pruned_loss=0.0136, audio_tagging_loss=0.00878, over 3046919.68 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:44:06,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2552260.0, ans=0.09899494936611666
2023-11-23 21:44:14,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382850
2023-11-23 21:44:16,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2552326.6666666665, ans=0.125
2023-11-23 21:44:28,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2552393.3333333335, ans=0.0
2023-11-23 21:44:32,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2552393.3333333335, ans=0.1
2023-11-23 21:44:40,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2552460.0, ans=0.125
2023-11-23 21:44:45,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2552460.0, ans=0.125
2023-11-23 21:44:46,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2552460.0, ans=15.0
2023-11-23 21:44:51,055 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:44:53,134 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:45:06,040 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10150, loss[loss=0.06188, simple_loss=0.07019, pruned_loss=0.01465, audio_tagging_loss=0.01214, over 15405.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09165, pruned_loss=0.01347, audio_tagging_loss=0.008869, over 3047858.47 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:45:11,971 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.35 vs. limit=15.0
2023-11-23 21:45:14,975 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.658e+01 9.213e+01 1.003e+02 1.366e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-23 21:45:17,496 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382900
2023-11-23 21:45:33,724 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=15.0
2023-11-23 21:45:34,772 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:45:35,473 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.64 vs. limit=15.0
2023-11-23 21:45:51,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2552793.3333333335, ans=0.125
2023-11-23 21:46:04,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2552860.0, ans=0.0
2023-11-23 21:46:06,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2552860.0, ans=0.0
2023-11-23 21:46:08,452 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10200, loss[loss=0.04658, simple_loss=0.05747, pruned_loss=0.007655, audio_tagging_loss=0.01019, over 15548.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09159, pruned_loss=0.01338, audio_tagging_loss=0.009038, over 3052947.49 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:46:11,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2552926.6666666665, ans=0.2
2023-11-23 21:46:19,150 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 382950
2023-11-23 21:46:30,762 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:46:34,697 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:46:50,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2553126.6666666665, ans=0.125
2023-11-23 21:47:10,200 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10250, loss[loss=0.08394, simple_loss=0.1184, pruned_loss=0.01704, audio_tagging_loss=0.007716, over 15542.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09181, pruned_loss=0.01352, audio_tagging_loss=0.009062, over 3053026.78 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:47:15,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2553260.0, ans=0.125
2023-11-23 21:47:18,953 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.706e+01 9.282e+01 1.010e+02 1.139e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-23 21:47:21,385 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383000
2023-11-23 21:47:31,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.33 vs. limit=22.5
2023-11-23 21:47:49,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2553460.0, ans=0.2
2023-11-23 21:47:50,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2553460.0, ans=0.125
2023-11-23 21:47:57,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2553460.0, ans=0.5
2023-11-23 21:48:07,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2553526.6666666665, ans=0.1
2023-11-23 21:48:12,186 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10300, loss[loss=0.06512, simple_loss=0.08269, pruned_loss=0.01369, audio_tagging_loss=0.01009, over 15293.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09207, pruned_loss=0.0136, audio_tagging_loss=0.009124, over 3051698.41 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:48:23,503 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383050
2023-11-23 21:48:28,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2553660.0, ans=0.0
2023-11-23 21:48:36,220 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.77 vs. limit=15.0
2023-11-23 21:48:48,719 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:48:53,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.66 vs. limit=10.0
2023-11-23 21:49:12,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2553860.0, ans=0.0
2023-11-23 21:49:14,278 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10350, loss[loss=0.05987, simple_loss=0.08557, pruned_loss=0.0096, audio_tagging_loss=0.00748, over 16295.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09221, pruned_loss=0.01359, audio_tagging_loss=0.009254, over 3048153.30 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:49:20,411 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.94 vs. limit=15.0
2023-11-23 21:49:23,130 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.433e+01 8.904e+01 9.438e+01 1.477e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 21:49:25,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383100
2023-11-23 21:50:14,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2554193.3333333335, ans=0.0
2023-11-23 21:50:16,756 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10400, loss[loss=0.06091, simple_loss=0.08023, pruned_loss=0.0121, audio_tagging_loss=0.0087, over 14103.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09252, pruned_loss=0.01361, audio_tagging_loss=0.009286, over 3045744.36 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:50:21,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2554260.0, ans=0.07
2023-11-23 21:50:28,788 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383150
2023-11-23 21:50:28,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2554326.6666666665, ans=0.1
2023-11-23 21:50:44,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2554393.3333333335, ans=0.0
2023-11-23 21:50:48,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2554393.3333333335, ans=0.125
2023-11-23 21:50:50,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2554393.3333333335, ans=0.125
2023-11-23 21:51:19,715 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10450, loss[loss=0.06144, simple_loss=0.0855, pruned_loss=0.009769, audio_tagging_loss=0.008921, over 16238.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09366, pruned_loss=0.01386, audio_tagging_loss=0.00914, over 3043890.64 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:51:28,563 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.248e+01 8.875e+01 9.733e+01 1.236e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 21:51:30,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2554593.3333333335, ans=0.1
2023-11-23 21:51:31,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383200
2023-11-23 21:51:32,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2554660.0, ans=0.2
2023-11-23 21:51:35,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2554660.0, ans=0.0
2023-11-23 21:52:03,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2554793.3333333335, ans=10.0
2023-11-23 21:52:22,444 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10500, loss[loss=0.07882, simple_loss=0.1092, pruned_loss=0.01649, audio_tagging_loss=0.007733, over 15739.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09293, pruned_loss=0.01374, audio_tagging_loss=0.00907, over 3047105.74 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:52:32,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2554926.6666666665, ans=0.1
2023-11-23 21:52:33,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383250
2023-11-23 21:52:39,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-23 21:52:40,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2554993.3333333335, ans=0.07
2023-11-23 21:52:59,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2555126.6666666665, ans=0.1
2023-11-23 21:53:06,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2555126.6666666665, ans=0.07
2023-11-23 21:53:11,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2555193.3333333335, ans=0.125
2023-11-23 21:53:19,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2555193.3333333335, ans=0.125
2023-11-23 21:53:24,426 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10550, loss[loss=0.05285, simple_loss=0.0636, pruned_loss=0.0102, audio_tagging_loss=0.01086, over 15258.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09172, pruned_loss=0.01354, audio_tagging_loss=0.009037, over 3048171.10 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:53:27,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2555260.0, ans=0.125
2023-11-23 21:53:33,553 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:53:34,352 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.236e+01 8.886e+01 9.649e+01 1.811e+02, threshold=1.777e+02, percent-clipped=1.0
2023-11-23 21:53:36,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383300
2023-11-23 21:53:38,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2555326.6666666665, ans=0.125
2023-11-23 21:53:41,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.85 vs. limit=10.0
2023-11-23 21:53:43,860 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=15.0
2023-11-23 21:53:52,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2555393.3333333335, ans=0.05
2023-11-23 21:53:52,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.30 vs. limit=22.5
2023-11-23 21:54:02,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2555460.0, ans=0.125
2023-11-23 21:54:22,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0
2023-11-23 21:54:26,964 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10600, loss[loss=0.06495, simple_loss=0.09084, pruned_loss=0.01151, audio_tagging_loss=0.00802, over 15150.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09164, pruned_loss=0.0135, audio_tagging_loss=0.009059, over 3050283.54 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:54:38,493 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383350
2023-11-23 21:54:55,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2555726.6666666665, ans=0.125
2023-11-23 21:55:00,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2555726.6666666665, ans=0.0
2023-11-23 21:55:27,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2555860.0, ans=0.05
2023-11-23 21:55:29,651 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10650, loss[loss=0.06797, simple_loss=0.09114, pruned_loss=0.01585, audio_tagging_loss=0.006546, over 15023.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09126, pruned_loss=0.01338, audio_tagging_loss=0.008983, over 3044078.81 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:55:31,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2555926.6666666665, ans=0.1
2023-11-23 21:55:38,961 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.595e+01 9.053e+01 9.824e+01 1.350e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 21:55:40,260 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383400
2023-11-23 21:55:40,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2555993.3333333335, ans=0.2
2023-11-23 21:55:51,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2555993.3333333335, ans=0.125
2023-11-23 21:55:56,229 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.39 vs. limit=15.0
2023-11-23 21:56:17,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2556126.6666666665, ans=0.1
2023-11-23 21:56:26,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2556193.3333333335, ans=0.1
2023-11-23 21:56:31,443 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10700, loss[loss=0.08462, simple_loss=0.1236, pruned_loss=0.01525, audio_tagging_loss=0.007593, over 15133.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09267, pruned_loss=0.01347, audio_tagging_loss=0.008849, over 3048650.36 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:56:43,599 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383450
2023-11-23 21:56:54,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2556326.6666666665, ans=0.0
2023-11-23 21:57:09,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2556460.0, ans=0.125
2023-11-23 21:57:18,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2556460.0, ans=0.125
2023-11-23 21:57:22,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2556526.6666666665, ans=0.1
2023-11-23 21:57:29,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2556526.6666666665, ans=0.125
2023-11-23 21:57:33,460 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.63 vs. limit=22.5
2023-11-23 21:57:35,199 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10750, loss[loss=0.07691, simple_loss=0.103, pruned_loss=0.01599, audio_tagging_loss=0.00939, over 14604.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09341, pruned_loss=0.01376, audio_tagging_loss=0.008796, over 3046781.34 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:57:45,137 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.476e+01 9.127e+01 1.007e+02 1.402e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 21:57:46,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383500
2023-11-23 21:57:46,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2556660.0, ans=0.125
2023-11-23 21:58:01,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2556726.6666666665, ans=0.125
2023-11-23 21:58:08,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2556726.6666666665, ans=0.025
2023-11-23 21:58:08,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.59 vs. limit=15.0
2023-11-23 21:58:21,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2556793.3333333335, ans=0.125
2023-11-23 21:58:24,657 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.59 vs. limit=15.0
2023-11-23 21:58:27,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2556860.0, ans=0.04949747468305833
2023-11-23 21:58:36,764 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.32 vs. limit=12.0
2023-11-23 21:58:37,449 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10800, loss[loss=0.06206, simple_loss=0.07971, pruned_loss=0.01253, audio_tagging_loss=0.009677, over 14228.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09302, pruned_loss=0.01361, audio_tagging_loss=0.008759, over 3045164.99 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:58:41,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2556926.6666666665, ans=0.125
2023-11-23 21:58:48,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383550
2023-11-23 21:58:56,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2556993.3333333335, ans=0.07
2023-11-23 21:59:05,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2557060.0, ans=0.125
2023-11-23 21:59:08,194 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.04 vs. limit=15.0
2023-11-23 21:59:08,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2557060.0, ans=0.125
2023-11-23 21:59:08,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2557060.0, ans=0.2
2023-11-23 21:59:22,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2557126.6666666665, ans=0.2
2023-11-23 21:59:36,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2557193.3333333335, ans=0.125
2023-11-23 21:59:38,842 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10850, loss[loss=0.07233, simple_loss=0.1042, pruned_loss=0.01345, audio_tagging_loss=0.006761, over 15301.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.093, pruned_loss=0.01367, audio_tagging_loss=0.008753, over 3042735.52 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:59:49,891 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.543e+01 9.235e+01 1.001e+02 1.289e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-23 21:59:50,046 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383600
2023-11-23 21:59:58,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2557326.6666666665, ans=0.125
2023-11-23 22:00:36,668 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:00:41,466 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10900, loss[loss=0.06092, simple_loss=0.08625, pruned_loss=0.01003, audio_tagging_loss=0.007763, over 15299.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09232, pruned_loss=0.01364, audio_tagging_loss=0.008797, over 3045694.35 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:00:44,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2557593.3333333335, ans=0.125
2023-11-23 22:00:53,284 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383650
2023-11-23 22:01:17,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2557793.3333333335, ans=0.05
2023-11-23 22:01:17,966 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=15.0
2023-11-23 22:01:31,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2557860.0, ans=0.0
2023-11-23 22:01:40,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2557860.0, ans=0.1
2023-11-23 22:01:44,014 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 10950, loss[loss=0.05428, simple_loss=0.07441, pruned_loss=0.01023, audio_tagging_loss=0.006849, over 15506.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09217, pruned_loss=0.01357, audio_tagging_loss=0.00897, over 3053730.57 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:01:44,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2557926.6666666665, ans=0.04949747468305833
2023-11-23 22:01:54,642 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.274e+01 8.562e+01 9.166e+01 9.897e+01 1.361e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 22:01:54,797 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383700
2023-11-23 22:02:06,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2558060.0, ans=0.2
2023-11-23 22:02:17,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2558060.0, ans=0.0
2023-11-23 22:02:30,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2558126.6666666665, ans=0.125
2023-11-23 22:02:44,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2558260.0, ans=0.1
2023-11-23 22:02:45,431 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11000, loss[loss=0.1025, simple_loss=0.1358, pruned_loss=0.02554, audio_tagging_loss=0.00912, over 15962.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.0914, pruned_loss=0.01352, audio_tagging_loss=0.009138, over 3051325.01 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:02:51,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2558260.0, ans=0.125
2023-11-23 22:02:53,613 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:02:56,581 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383750
2023-11-23 22:03:14,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2558393.3333333335, ans=0.1
2023-11-23 22:03:27,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2558460.0, ans=0.0
2023-11-23 22:03:34,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2558526.6666666665, ans=0.0
2023-11-23 22:03:46,997 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11050, loss[loss=0.06362, simple_loss=0.08258, pruned_loss=0.0153, audio_tagging_loss=0.007027, over 17123.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09084, pruned_loss=0.01344, audio_tagging_loss=0.009211, over 3059207.08 frames. ], batch size: 66, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:03:58,697 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.213e+01 8.515e+01 9.099e+01 9.905e+01 1.187e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-23 22:03:58,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383800
2023-11-23 22:04:22,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2558726.6666666665, ans=0.1
2023-11-23 22:04:27,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2558793.3333333335, ans=0.0
2023-11-23 22:04:36,109 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:04:46,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2558860.0, ans=0.125
2023-11-23 22:04:50,541 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11100, loss[loss=0.06118, simple_loss=0.08463, pruned_loss=0.009152, audio_tagging_loss=0.00971, over 15754.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09055, pruned_loss=0.01334, audio_tagging_loss=0.009308, over 3057494.23 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:05:01,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383850
2023-11-23 22:05:04,962 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:05:41,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2559193.3333333335, ans=0.125
2023-11-23 22:05:46,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.46 vs. limit=22.5
2023-11-23 22:05:51,950 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11150, loss[loss=0.05457, simple_loss=0.07175, pruned_loss=0.009501, audio_tagging_loss=0.009197, over 16108.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09118, pruned_loss=0.01343, audio_tagging_loss=0.009313, over 3061681.99 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:06:02,623 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.354e+01 8.926e+01 9.504e+01 1.198e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 22:06:02,774 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383900
2023-11-23 22:06:18,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.83 vs. limit=6.0
2023-11-23 22:06:26,149 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:06:31,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2559460.0, ans=0.0
2023-11-23 22:06:42,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2559526.6666666665, ans=0.125
2023-11-23 22:06:53,538 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11200, loss[loss=0.06862, simple_loss=0.08817, pruned_loss=0.01196, audio_tagging_loss=0.01257, over 15738.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09201, pruned_loss=0.01353, audio_tagging_loss=0.009335, over 3057731.53 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:07:00,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2559593.3333333335, ans=0.0
2023-11-23 22:07:03,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2559593.3333333335, ans=0.0
2023-11-23 22:07:04,919 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 383950
2023-11-23 22:07:14,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2559660.0, ans=0.125
2023-11-23 22:07:19,738 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.02 vs. limit=15.0
2023-11-23 22:07:22,239 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.20 vs. limit=6.0
2023-11-23 22:07:25,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2559726.6666666665, ans=0.125
2023-11-23 22:07:28,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2559726.6666666665, ans=0.0
2023-11-23 22:07:33,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2559793.3333333335, ans=0.125
2023-11-23 22:07:37,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2559793.3333333335, ans=0.125
2023-11-23 22:07:48,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2559860.0, ans=0.125
2023-11-23 22:07:52,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2559860.0, ans=0.125
2023-11-23 22:07:55,768 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11250, loss[loss=0.04806, simple_loss=0.06796, pruned_loss=0.005953, audio_tagging_loss=0.008127, over 15623.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09076, pruned_loss=0.01333, audio_tagging_loss=0.009416, over 3048748.78 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:08:07,487 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.428e+01 9.033e+01 9.633e+01 1.168e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 22:08:07,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384000
2023-11-23 22:08:24,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2560060.0, ans=0.5
2023-11-23 22:08:27,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2560060.0, ans=0.1
2023-11-23 22:08:45,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2560126.6666666665, ans=15.0
2023-11-23 22:08:49,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn1.whiten.whitening_limit, batch_count=2560193.3333333335, ans=22.5
2023-11-23 22:09:00,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2560193.3333333335, ans=0.0
2023-11-23 22:09:02,410 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11300, loss[loss=0.07523, simple_loss=0.1023, pruned_loss=0.01702, audio_tagging_loss=0.007039, over 14573.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09084, pruned_loss=0.01343, audio_tagging_loss=0.009201, over 3044949.16 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:09:08,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2560260.0, ans=0.025
2023-11-23 22:09:13,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384050
2023-11-23 22:09:26,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2560393.3333333335, ans=0.05
2023-11-23 22:09:30,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.79 vs. limit=15.0
2023-11-23 22:09:33,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2560393.3333333335, ans=0.125
2023-11-23 22:09:50,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.63 vs. limit=15.0
2023-11-23 22:09:57,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.84 vs. limit=15.0
2023-11-23 22:09:58,378 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:10:04,300 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11350, loss[loss=0.08031, simple_loss=0.1077, pruned_loss=0.01796, audio_tagging_loss=0.008489, over 15919.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09221, pruned_loss=0.01354, audio_tagging_loss=0.008951, over 3044362.28 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:10:09,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.88 vs. limit=15.0
2023-11-23 22:10:16,259 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.206e+01 9.003e+01 9.537e+01 1.148e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 22:10:16,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384100
2023-11-23 22:10:32,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2560726.6666666665, ans=0.0
2023-11-23 22:10:34,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2560726.6666666665, ans=0.2
2023-11-23 22:10:48,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2560793.3333333335, ans=0.1
2023-11-23 22:11:07,784 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11400, loss[loss=0.05473, simple_loss=0.07098, pruned_loss=0.008858, audio_tagging_loss=0.01039, over 14073.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09174, pruned_loss=0.01337, audio_tagging_loss=0.008919, over 3039106.94 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:11:18,980 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384150
2023-11-23 22:11:37,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.61 vs. limit=6.0
2023-11-23 22:11:42,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2561060.0, ans=0.2
2023-11-23 22:12:00,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2561193.3333333335, ans=0.0
2023-11-23 22:12:10,320 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11450, loss[loss=0.05479, simple_loss=0.07539, pruned_loss=0.009797, audio_tagging_loss=0.007302, over 14257.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09087, pruned_loss=0.01325, audio_tagging_loss=0.008959, over 3035237.08 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:12:10,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2561260.0, ans=0.2
2023-11-23 22:12:17,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2561260.0, ans=0.04949747468305833
2023-11-23 22:12:21,194 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384200
2023-11-23 22:12:22,235 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.398e+01 8.984e+01 9.548e+01 1.185e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 22:12:44,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2561393.3333333335, ans=0.025
2023-11-23 22:13:05,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2561526.6666666665, ans=0.2
2023-11-23 22:13:08,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2561526.6666666665, ans=0.125
2023-11-23 22:13:11,998 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11500, loss[loss=0.0581, simple_loss=0.07574, pruned_loss=0.009231, audio_tagging_loss=0.01099, over 15936.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09107, pruned_loss=0.01317, audio_tagging_loss=0.008948, over 3037638.34 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:13:23,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384250
2023-11-23 22:13:26,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0
2023-11-23 22:13:32,590 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.44 vs. limit=22.5
2023-11-23 22:13:38,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2561726.6666666665, ans=0.125
2023-11-23 22:13:47,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2561726.6666666665, ans=0.0
2023-11-23 22:13:48,939 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.62 vs. limit=10.0
2023-11-23 22:13:50,288 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.59 vs. limit=22.5
2023-11-23 22:14:14,425 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11550, loss[loss=0.06325, simple_loss=0.08411, pruned_loss=0.01293, audio_tagging_loss=0.008264, over 15894.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09127, pruned_loss=0.01319, audio_tagging_loss=0.008918, over 3044523.33 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:14:25,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384300
2023-11-23 22:14:26,803 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.220e+01 8.896e+01 9.591e+01 1.198e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 22:14:50,985 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:14:56,769 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.06 vs. limit=15.0
2023-11-23 22:14:57,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2562126.6666666665, ans=0.125
2023-11-23 22:15:03,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2562193.3333333335, ans=0.125
2023-11-23 22:15:07,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2562193.3333333335, ans=0.125
2023-11-23 22:15:16,204 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11600, loss[loss=0.07734, simple_loss=0.1038, pruned_loss=0.01476, audio_tagging_loss=0.0107, over 15194.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09161, pruned_loss=0.01342, audio_tagging_loss=0.008923, over 3046861.01 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:15:26,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2562260.0, ans=10.0
2023-11-23 22:15:27,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384350
2023-11-23 22:15:50,758 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.84 vs. limit=15.0
2023-11-23 22:16:05,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2562526.6666666665, ans=0.025
2023-11-23 22:16:05,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.13 vs. limit=15.0
2023-11-23 22:16:18,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.41 vs. limit=15.0
2023-11-23 22:16:18,765 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11650, loss[loss=0.06213, simple_loss=0.0838, pruned_loss=0.01088, audio_tagging_loss=0.009345, over 15665.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.091, pruned_loss=0.01336, audio_tagging_loss=0.009006, over 3045522.31 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:16:28,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2562593.3333333335, ans=0.125
2023-11-23 22:16:30,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384400
2023-11-23 22:16:31,039 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.399e+01 8.387e+01 8.924e+01 9.715e+01 1.157e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 22:16:41,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2562660.0, ans=0.0
2023-11-23 22:16:48,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2562726.6666666665, ans=0.125
2023-11-23 22:16:53,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2562726.6666666665, ans=0.0
2023-11-23 22:17:06,146 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.23 vs. limit=15.0
2023-11-23 22:17:22,171 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11700, loss[loss=0.07198, simple_loss=0.1029, pruned_loss=0.0135, audio_tagging_loss=0.007037, over 14481.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09019, pruned_loss=0.01334, audio_tagging_loss=0.009007, over 3046288.72 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:17:23,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2562926.6666666665, ans=0.0
2023-11-23 22:17:33,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384450
2023-11-23 22:17:33,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2562993.3333333335, ans=0.1
2023-11-23 22:17:43,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2562993.3333333335, ans=0.2
2023-11-23 22:17:46,156 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.49 vs. limit=12.0
2023-11-23 22:17:48,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2563060.0, ans=0.125
2023-11-23 22:17:59,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2563126.6666666665, ans=0.1
2023-11-23 22:18:12,698 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.89 vs. limit=22.5
2023-11-23 22:18:24,594 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11750, loss[loss=0.08261, simple_loss=0.1119, pruned_loss=0.01738, audio_tagging_loss=0.009267, over 14943.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09094, pruned_loss=0.01343, audio_tagging_loss=0.009054, over 3048421.45 frames. ], batch size: 55, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:18:30,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2563260.0, ans=0.125
2023-11-23 22:18:35,389 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384500
2023-11-23 22:18:36,427 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.506e+01 8.915e+01 9.648e+01 1.548e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 22:19:03,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2563460.0, ans=0.125
2023-11-23 22:19:04,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2563460.0, ans=0.0
2023-11-23 22:19:26,091 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11800, loss[loss=0.07322, simple_loss=0.08759, pruned_loss=0.01967, audio_tagging_loss=0.009749, over 14337.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.08953, pruned_loss=0.01323, audio_tagging_loss=0.009166, over 3039430.02 frames. ], batch size: 53, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:19:37,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384550
2023-11-23 22:20:08,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2563793.3333333335, ans=0.1
2023-11-23 22:20:13,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2563793.3333333335, ans=0.1
2023-11-23 22:20:20,353 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.34 vs. limit=10.0
2023-11-23 22:20:28,404 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11850, loss[loss=0.08291, simple_loss=0.114, pruned_loss=0.01608, audio_tagging_loss=0.009839, over 15735.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.08977, pruned_loss=0.0134, audio_tagging_loss=0.009259, over 3045711.56 frames. ], batch size: 57, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:20:31,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2563926.6666666665, ans=0.0
2023-11-23 22:20:39,736 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384600
2023-11-23 22:20:41,325 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.384e+01 8.994e+01 9.780e+01 1.224e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 22:21:11,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2564126.6666666665, ans=0.5
2023-11-23 22:21:26,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2564193.3333333335, ans=0.125
2023-11-23 22:21:30,194 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:21:31,109 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11900, loss[loss=0.06021, simple_loss=0.08197, pruned_loss=0.01125, audio_tagging_loss=0.007974, over 15260.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09092, pruned_loss=0.01355, audio_tagging_loss=0.009186, over 3045459.34 frames. ], batch size: 59, lr: 2.10e-03, grad_scale: 16.0
2023-11-23 22:21:36,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2564260.0, ans=0.2
2023-11-23 22:21:41,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384650
2023-11-23 22:21:45,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2564326.6666666665, ans=0.1
2023-11-23 22:21:56,170 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0
2023-11-23 22:22:20,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2564526.6666666665, ans=0.2
2023-11-23 22:22:26,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2564526.6666666665, ans=0.125
2023-11-23 22:22:32,318 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.28 vs. limit=15.0
2023-11-23 22:22:32,792 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 11950, loss[loss=0.05911, simple_loss=0.07596, pruned_loss=0.0109, audio_tagging_loss=0.01024, over 14764.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09135, pruned_loss=0.01372, audio_tagging_loss=0.00927, over 3054142.09 frames. ], batch size: 57, lr: 2.10e-03, grad_scale: 16.0
2023-11-23 22:22:35,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2564593.3333333335, ans=0.125
2023-11-23 22:22:43,262 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384700
2023-11-23 22:22:43,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2564660.0, ans=0.0
2023-11-23 22:22:46,154 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.381e+01 9.044e+01 9.786e+01 1.119e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 22:22:52,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.44 vs. limit=15.0
2023-11-23 22:23:21,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2564860.0, ans=0.125
2023-11-23 22:23:26,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2564860.0, ans=0.09899494936611666
2023-11-23 22:23:32,152 INFO [train_asr.py:1221] (2/4) Epoch 32, batch 12000, loss[loss=0.07497, simple_loss=0.09884, pruned_loss=0.01571, audio_tagging_loss=0.00984, over 16024.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09122, pruned_loss=0.01356, audio_tagging_loss=0.009368, over 3047002.82 frames. ], batch size: 58, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:23:32,153 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 22:24:04,972 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3414, 5.0414, 4.6965, 5.1873], device='cuda:2')
2023-11-23 22:24:14,283 INFO [train_asr.py:1253] (2/4) Epoch 32, validation: loss=0.05848, simple_loss=0.0511, pruned_loss=0.005239, audio_tagging_loss=0.02769, over 4681554.00 frames. 
2023-11-23 22:24:14,284 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 22:24:19,557 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:24:24,967 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384750
2023-11-23 22:25:13,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565086.6666666665, ans=0.1
2023-11-23 22:25:15,127 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 0, loss[loss=0.08871, simple_loss=0.104, pruned_loss=0.01442, audio_tagging_loss=0.02229, over 14819.00 frames. ], tot_loss[loss=0.08871, simple_loss=0.104, pruned_loss=0.01442, audio_tagging_loss=0.02229, over 14819.00 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:25:15,128 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 22:25:50,709 INFO [train_asr.py:1253] (2/4) Epoch 33, validation: loss=0.05781, simple_loss=0.05104, pruned_loss=0.005203, audio_tagging_loss=0.02709, over 4681554.00 frames. 
2023-11-23 22:25:50,710 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 22:25:56,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2565086.6666666665, ans=0.125
2023-11-23 22:25:58,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2565086.6666666665, ans=0.125
2023-11-23 22:26:05,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2565153.3333333335, ans=0.125
2023-11-23 22:26:13,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565153.3333333335, ans=0.1
2023-11-23 22:26:27,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2565286.6666666665, ans=0.125
2023-11-23 22:26:34,524 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384800
2023-11-23 22:26:35,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565286.6666666665, ans=0.1
2023-11-23 22:26:38,167 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.158e+01 8.944e+01 9.729e+01 1.034e+02 1.354e+02, threshold=1.946e+02, percent-clipped=0.0
2023-11-23 22:26:39,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2565353.3333333335, ans=0.125
2023-11-23 22:26:46,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2565353.3333333335, ans=0.0
2023-11-23 22:26:52,489 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 50, loss[loss=0.1049, simple_loss=0.126, pruned_loss=0.02744, audio_tagging_loss=0.01448, over 15308.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09164, pruned_loss=0.01319, audio_tagging_loss=0.01741, over 683562.11 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:26:54,341 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.56 vs. limit=12.0
2023-11-23 22:27:00,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2565420.0, ans=0.125
2023-11-23 22:27:00,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2565420.0, ans=0.125
2023-11-23 22:27:10,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565486.6666666665, ans=0.1
2023-11-23 22:27:36,597 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384850
2023-11-23 22:27:49,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2565686.6666666665, ans=0.125
2023-11-23 22:27:55,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2565753.3333333335, ans=0.125
2023-11-23 22:27:56,907 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 100, loss[loss=0.0892, simple_loss=0.1227, pruned_loss=0.01675, audio_tagging_loss=0.01111, over 15826.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.08941, pruned_loss=0.01282, audio_tagging_loss=0.01665, over 1200542.69 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:27:58,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2565753.3333333335, ans=0.0
2023-11-23 22:28:14,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2565820.0, ans=0.125
2023-11-23 22:28:15,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.51 vs. limit=12.0
2023-11-23 22:28:38,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384900
2023-11-23 22:28:43,527 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.457e+01 8.971e+01 9.700e+01 1.041e+02 1.375e+02, threshold=1.940e+02, percent-clipped=0.0
2023-11-23 22:28:55,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2566020.0, ans=0.07
2023-11-23 22:28:57,722 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 150, loss[loss=0.09462, simple_loss=0.1246, pruned_loss=0.02242, audio_tagging_loss=0.009883, over 14383.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.08944, pruned_loss=0.01293, audio_tagging_loss=0.01484, over 1613459.84 frames. ], batch size: 52, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:28:58,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2566086.6666666665, ans=0.125
2023-11-23 22:28:59,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2566086.6666666665, ans=0.125
2023-11-23 22:29:06,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.77 vs. limit=15.0
2023-11-23 22:29:13,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2566153.3333333335, ans=0.2
2023-11-23 22:29:27,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2566220.0, ans=0.5
2023-11-23 22:29:31,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2566220.0, ans=0.2
2023-11-23 22:29:41,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 384950
2023-11-23 22:29:52,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2566353.3333333335, ans=0.125
2023-11-23 22:29:55,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2566353.3333333335, ans=0.125
2023-11-23 22:29:59,069 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 200, loss[loss=0.07024, simple_loss=0.09673, pruned_loss=0.01241, audio_tagging_loss=0.009461, over 15144.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09138, pruned_loss=0.01324, audio_tagging_loss=0.01307, over 1928362.34 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:30:02,831 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:30:19,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2566486.6666666665, ans=0.125
2023-11-23 22:30:22,898 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.01 vs. limit=15.0
2023-11-23 22:30:37,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2566620.0, ans=0.1
2023-11-23 22:30:42,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385000
2023-11-23 22:30:46,067 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.668e+01 9.124e+01 1.011e+02 1.346e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 22:30:59,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2566753.3333333335, ans=0.125
2023-11-23 22:30:59,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2566753.3333333335, ans=0.05
2023-11-23 22:31:01,363 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 250, loss[loss=0.09184, simple_loss=0.1289, pruned_loss=0.02022, audio_tagging_loss=0.007168, over 14820.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09188, pruned_loss=0.01329, audio_tagging_loss=0.0119, over 2171035.86 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:31:04,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2566753.3333333335, ans=0.125
2023-11-23 22:31:05,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2566753.3333333335, ans=0.125
2023-11-23 22:31:08,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2566753.3333333335, ans=0.125
2023-11-23 22:31:09,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2566753.3333333335, ans=0.035
2023-11-23 22:31:25,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2566886.6666666665, ans=0.0
2023-11-23 22:31:32,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2566886.6666666665, ans=0.125
2023-11-23 22:31:32,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.64 vs. limit=6.0
2023-11-23 22:31:42,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2566953.3333333335, ans=0.0
2023-11-23 22:31:44,956 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385050
2023-11-23 22:31:52,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2567020.0, ans=15.0
2023-11-23 22:32:02,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2567020.0, ans=0.09899494936611666
2023-11-23 22:32:04,230 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 300, loss[loss=0.06614, simple_loss=0.08861, pruned_loss=0.01338, audio_tagging_loss=0.008457, over 14188.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09186, pruned_loss=0.01358, audio_tagging_loss=0.01105, over 2375016.80 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:32:07,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2567086.6666666665, ans=0.125
2023-11-23 22:32:08,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2567086.6666666665, ans=0.125
2023-11-23 22:32:09,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2567086.6666666665, ans=0.125
2023-11-23 22:32:16,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2567153.3333333335, ans=0.125
2023-11-23 22:32:28,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2567220.0, ans=0.0
2023-11-23 22:32:47,986 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385100
2023-11-23 22:32:51,311 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.686e+01 8.804e+01 9.384e+01 1.010e+02 1.261e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-23 22:32:56,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2567353.3333333335, ans=0.1
2023-11-23 22:33:00,455 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.31 vs. limit=15.0
2023-11-23 22:33:04,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=22.5
2023-11-23 22:33:05,396 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 350, loss[loss=0.06545, simple_loss=0.09023, pruned_loss=0.01235, audio_tagging_loss=0.007987, over 14586.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09265, pruned_loss=0.01372, audio_tagging_loss=0.0104, over 2526098.65 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:33:16,047 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.86 vs. limit=10.0
2023-11-23 22:33:25,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2567486.6666666665, ans=0.0
2023-11-23 22:33:31,837 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0
2023-11-23 22:33:39,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2567553.3333333335, ans=0.0
2023-11-23 22:33:49,632 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385150
2023-11-23 22:33:51,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2567620.0, ans=0.0
2023-11-23 22:33:53,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2567620.0, ans=0.125
2023-11-23 22:34:07,547 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 400, loss[loss=0.06774, simple_loss=0.09746, pruned_loss=0.01133, audio_tagging_loss=0.007682, over 16010.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09256, pruned_loss=0.01382, audio_tagging_loss=0.01004, over 2647240.89 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:34:09,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2567753.3333333335, ans=0.0
2023-11-23 22:34:11,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2567753.3333333335, ans=0.125
2023-11-23 22:34:11,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2567753.3333333335, ans=0.2
2023-11-23 22:34:13,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2567753.3333333335, ans=0.2
2023-11-23 22:34:17,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2567753.3333333335, ans=0.125
2023-11-23 22:34:32,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2567886.6666666665, ans=0.125
2023-11-23 22:34:36,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.36 vs. limit=15.0
2023-11-23 22:34:44,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2567953.3333333335, ans=0.1
2023-11-23 22:34:51,219 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385200
2023-11-23 22:34:51,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2567953.3333333335, ans=0.2
2023-11-23 22:34:54,928 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.173e+01 8.325e+01 8.873e+01 9.611e+01 1.385e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 22:34:57,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2568020.0, ans=0.125
2023-11-23 22:34:59,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2568020.0, ans=0.125
2023-11-23 22:35:11,086 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 450, loss[loss=0.07921, simple_loss=0.1094, pruned_loss=0.0171, audio_tagging_loss=0.007433, over 14117.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09201, pruned_loss=0.0137, audio_tagging_loss=0.009822, over 2729932.48 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:35:34,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.12 vs. limit=15.0
2023-11-23 22:35:37,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2568220.0, ans=0.125
2023-11-23 22:35:46,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.49 vs. limit=12.0
2023-11-23 22:35:49,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2568286.6666666665, ans=0.05
2023-11-23 22:35:54,314 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385250
2023-11-23 22:36:11,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.60 vs. limit=15.0
2023-11-23 22:36:12,792 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 500, loss[loss=0.07341, simple_loss=0.1024, pruned_loss=0.01298, audio_tagging_loss=0.009253, over 14892.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09215, pruned_loss=0.01367, audio_tagging_loss=0.009604, over 2804169.61 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:36:17,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.68 vs. limit=15.0
2023-11-23 22:36:57,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385300
2023-11-23 22:37:00,568 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.320e+01 8.409e+01 9.142e+01 9.799e+01 1.482e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 22:37:11,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2568686.6666666665, ans=0.125
2023-11-23 22:37:15,580 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 550, loss[loss=0.07216, simple_loss=0.09776, pruned_loss=0.01375, audio_tagging_loss=0.009532, over 14876.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.0913, pruned_loss=0.01351, audio_tagging_loss=0.009516, over 2859377.03 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:37:36,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2568820.0, ans=0.125
2023-11-23 22:37:58,592 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385350
2023-11-23 22:37:58,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2568953.3333333335, ans=0.125
2023-11-23 22:38:03,531 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.37 vs. limit=8.0
2023-11-23 22:38:04,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2569020.0, ans=0.2
2023-11-23 22:38:17,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2569086.6666666665, ans=0.125
2023-11-23 22:38:17,969 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 600, loss[loss=0.05573, simple_loss=0.0711, pruned_loss=0.008804, audio_tagging_loss=0.01137, over 13611.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09087, pruned_loss=0.01352, audio_tagging_loss=0.009479, over 2894588.04 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:38:21,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2569086.6666666665, ans=0.2
2023-11-23 22:38:55,304 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:39:01,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385400
2023-11-23 22:39:06,580 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.472e+01 9.172e+01 9.611e+01 1.258e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 22:39:19,425 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:39:20,513 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 650, loss[loss=0.06269, simple_loss=0.08226, pruned_loss=0.01212, audio_tagging_loss=0.009438, over 14525.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.0916, pruned_loss=0.01348, audio_tagging_loss=0.00926, over 2927259.24 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:39:20,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2569420.0, ans=0.1
2023-11-23 22:39:31,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2569486.6666666665, ans=0.2
2023-11-23 22:39:48,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2569553.3333333335, ans=0.0
2023-11-23 22:39:50,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2569553.3333333335, ans=0.0
2023-11-23 22:39:55,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2569553.3333333335, ans=0.0
2023-11-23 22:40:02,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2569620.0, ans=0.125
2023-11-23 22:40:02,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2569620.0, ans=0.125
2023-11-23 22:40:04,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385450
2023-11-23 22:40:22,208 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 700, loss[loss=0.06218, simple_loss=0.09033, pruned_loss=0.008096, audio_tagging_loss=0.008916, over 14573.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09084, pruned_loss=0.01322, audio_tagging_loss=0.009244, over 2947309.34 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:40:43,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2569820.0, ans=0.1
2023-11-23 22:40:48,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.57 vs. limit=22.5
2023-11-23 22:40:48,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.57 vs. limit=15.0
2023-11-23 22:41:06,025 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385500
2023-11-23 22:41:06,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2569953.3333333335, ans=0.125
2023-11-23 22:41:11,044 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.418e+01 8.479e+01 9.169e+01 9.632e+01 1.119e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 22:41:25,252 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 750, loss[loss=0.06668, simple_loss=0.09722, pruned_loss=0.01064, audio_tagging_loss=0.007432, over 15546.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09163, pruned_loss=0.01345, audio_tagging_loss=0.009197, over 2970005.65 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:41:30,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2570086.6666666665, ans=0.2
2023-11-23 22:41:55,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2570220.0, ans=0.125
2023-11-23 22:42:04,735 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.82 vs. limit=15.0
2023-11-23 22:42:09,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385550
2023-11-23 22:42:19,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2570353.3333333335, ans=0.125
2023-11-23 22:42:27,207 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 800, loss[loss=0.08145, simple_loss=0.1168, pruned_loss=0.01494, audio_tagging_loss=0.008091, over 14284.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09183, pruned_loss=0.0135, audio_tagging_loss=0.009278, over 2981578.02 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:42:42,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2570486.6666666665, ans=0.125
2023-11-23 22:42:43,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2570486.6666666665, ans=0.0
2023-11-23 22:42:50,909 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=15.0
2023-11-23 22:42:55,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2570553.3333333335, ans=0.0
2023-11-23 22:42:55,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2570553.3333333335, ans=0.125
2023-11-23 22:43:07,935 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:43:10,557 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.16 vs. limit=15.0
2023-11-23 22:43:11,289 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385600
2023-11-23 22:43:16,322 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.424e+01 8.735e+01 9.399e+01 1.001e+02 1.314e+02, threshold=1.880e+02, percent-clipped=0.0
2023-11-23 22:43:30,068 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 850, loss[loss=0.07556, simple_loss=0.1079, pruned_loss=0.01362, audio_tagging_loss=0.008005, over 14965.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09196, pruned_loss=0.01358, audio_tagging_loss=0.009302, over 2998178.21 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:43:43,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2570820.0, ans=0.0
2023-11-23 22:43:53,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2570820.0, ans=0.0
2023-11-23 22:44:11,302 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.93 vs. limit=10.0
2023-11-23 22:44:14,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385650
2023-11-23 22:44:20,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2571020.0, ans=0.0
2023-11-23 22:44:33,815 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 900, loss[loss=0.05951, simple_loss=0.08259, pruned_loss=0.01012, audio_tagging_loss=0.0081, over 14802.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09201, pruned_loss=0.01353, audio_tagging_loss=0.009297, over 3003520.18 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:44:35,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2571086.6666666665, ans=0.1
2023-11-23 22:44:38,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2571086.6666666665, ans=0.09899494936611666
2023-11-23 22:44:45,438 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=15.0
2023-11-23 22:45:03,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2571220.0, ans=0.2
2023-11-23 22:45:17,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385700
2023-11-23 22:45:22,416 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.319e+01 9.000e+01 1.000e+02 1.242e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 22:45:30,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2571353.3333333335, ans=0.125
2023-11-23 22:45:31,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.67 vs. limit=15.0
2023-11-23 22:45:35,492 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 950, loss[loss=0.05109, simple_loss=0.06697, pruned_loss=0.008625, audio_tagging_loss=0.008981, over 14725.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09169, pruned_loss=0.01345, audio_tagging_loss=0.009266, over 3013588.82 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:45:45,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2571420.0, ans=0.125
2023-11-23 22:45:59,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2571553.3333333335, ans=0.125
2023-11-23 22:46:18,637 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.52 vs. limit=15.0
2023-11-23 22:46:19,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385750
2023-11-23 22:46:33,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2571686.6666666665, ans=0.0
2023-11-23 22:46:37,418 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1000, loss[loss=0.05756, simple_loss=0.07614, pruned_loss=0.01119, audio_tagging_loss=0.008301, over 15465.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09205, pruned_loss=0.01353, audio_tagging_loss=0.00912, over 3020406.93 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:47:03,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2571886.6666666665, ans=0.125
2023-11-23 22:47:04,035 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:47:10,556 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.83 vs. limit=22.5
2023-11-23 22:47:20,807 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385800
2023-11-23 22:47:27,680 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.356e+01 8.922e+01 9.562e+01 1.199e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 22:47:40,648 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1050, loss[loss=0.05257, simple_loss=0.06838, pruned_loss=0.007506, audio_tagging_loss=0.01087, over 15189.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09147, pruned_loss=0.01346, audio_tagging_loss=0.009103, over 3025197.41 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:48:05,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2572220.0, ans=0.0
2023-11-23 22:48:24,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385850
2023-11-23 22:48:34,336 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.82 vs. limit=15.0
2023-11-23 22:48:38,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2572353.3333333335, ans=0.125
2023-11-23 22:48:43,142 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1100, loss[loss=0.06587, simple_loss=0.08948, pruned_loss=0.01137, audio_tagging_loss=0.009755, over 15756.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09024, pruned_loss=0.01334, audio_tagging_loss=0.008998, over 3032057.28 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:48:45,568 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:48:53,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2572420.0, ans=0.07
2023-11-23 22:48:55,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2572486.6666666665, ans=0.0
2023-11-23 22:49:00,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.31 vs. limit=15.0
2023-11-23 22:49:02,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2572486.6666666665, ans=0.125
2023-11-23 22:49:22,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2572620.0, ans=0.0
2023-11-23 22:49:23,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2572620.0, ans=0.2
2023-11-23 22:49:27,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385900
2023-11-23 22:49:33,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.297e+01 8.837e+01 9.387e+01 1.175e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 22:49:35,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2572686.6666666665, ans=10.0
2023-11-23 22:49:43,610 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.92 vs. limit=15.0
2023-11-23 22:49:45,374 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1150, loss[loss=0.07319, simple_loss=0.08969, pruned_loss=0.01594, audio_tagging_loss=0.0124, over 16025.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09052, pruned_loss=0.01343, audio_tagging_loss=0.00893, over 3029196.07 frames. ], batch size: 60, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:49:47,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2572753.3333333335, ans=0.0
2023-11-23 22:49:55,661 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.01 vs. limit=12.0
2023-11-23 22:50:14,760 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.04 vs. limit=15.0
2023-11-23 22:50:18,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2572886.6666666665, ans=0.125
2023-11-23 22:50:29,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 385950
2023-11-23 22:50:36,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2573020.0, ans=0.0
2023-11-23 22:50:36,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2573020.0, ans=0.0
2023-11-23 22:50:45,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2573020.0, ans=0.0
2023-11-23 22:50:47,801 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1200, loss[loss=0.07338, simple_loss=0.09807, pruned_loss=0.01687, audio_tagging_loss=0.007474, over 14463.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09005, pruned_loss=0.01344, audio_tagging_loss=0.00889, over 3034035.46 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:50:58,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2573086.6666666665, ans=0.1
2023-11-23 22:51:11,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2573220.0, ans=0.125
2023-11-23 22:51:16,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2573220.0, ans=0.125
2023-11-23 22:51:18,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2573220.0, ans=0.0
2023-11-23 22:51:31,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386000
2023-11-23 22:51:38,145 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.994e+01 8.408e+01 9.187e+01 9.835e+01 1.432e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 22:51:39,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2573353.3333333335, ans=0.0
2023-11-23 22:51:43,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2573353.3333333335, ans=0.1
2023-11-23 22:51:50,526 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1250, loss[loss=0.06224, simple_loss=0.0857, pruned_loss=0.008938, audio_tagging_loss=0.01045, over 13619.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.08977, pruned_loss=0.01345, audio_tagging_loss=0.008922, over 3035958.62 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:51:58,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-23 22:52:17,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2573553.3333333335, ans=0.0
2023-11-23 22:52:27,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2573620.0, ans=0.0
2023-11-23 22:52:29,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2573620.0, ans=0.125
2023-11-23 22:52:35,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386050
2023-11-23 22:52:36,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2573620.0, ans=0.125
2023-11-23 22:52:44,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2573686.6666666665, ans=0.2
2023-11-23 22:52:53,024 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1300, loss[loss=0.075, simple_loss=0.09348, pruned_loss=0.01999, audio_tagging_loss=0.008274, over 14295.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09, pruned_loss=0.01341, audio_tagging_loss=0.00886, over 3035323.40 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:53:20,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2573886.6666666665, ans=15.0
2023-11-23 22:53:36,840 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386100
2023-11-23 22:53:42,576 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.267e+01 8.921e+01 9.505e+01 1.791e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 22:53:54,254 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:53:55,059 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1350, loss[loss=0.06534, simple_loss=0.0809, pruned_loss=0.01166, audio_tagging_loss=0.01323, over 15781.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.08973, pruned_loss=0.01331, audio_tagging_loss=0.008957, over 3037860.62 frames. ], batch size: 61, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:53:59,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2574086.6666666665, ans=0.125
2023-11-23 22:54:03,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2574086.6666666665, ans=0.1
2023-11-23 22:54:13,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2574153.3333333335, ans=0.0
2023-11-23 22:54:25,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2574220.0, ans=0.125
2023-11-23 22:54:31,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2574286.6666666665, ans=0.0
2023-11-23 22:54:32,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2574286.6666666665, ans=0.025
2023-11-23 22:54:35,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2574286.6666666665, ans=0.125
2023-11-23 22:54:37,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2574286.6666666665, ans=0.125
2023-11-23 22:54:38,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386150
2023-11-23 22:54:39,672 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:54:57,758 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1400, loss[loss=0.0623, simple_loss=0.08778, pruned_loss=0.01088, audio_tagging_loss=0.00752, over 14795.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09069, pruned_loss=0.01335, audio_tagging_loss=0.009052, over 3041477.79 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:55:06,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2574420.0, ans=0.0
2023-11-23 22:55:23,973 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.91 vs. limit=15.0
2023-11-23 22:55:41,038 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386200
2023-11-23 22:55:41,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2574620.0, ans=0.125
2023-11-23 22:55:41,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.53 vs. limit=10.0
2023-11-23 22:55:47,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2574686.6666666665, ans=0.125
2023-11-23 22:55:48,295 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.327e+01 9.200e+01 9.811e+01 1.262e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 22:55:58,978 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1450, loss[loss=0.08559, simple_loss=0.1133, pruned_loss=0.01989, audio_tagging_loss=0.009068, over 15753.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09099, pruned_loss=0.01351, audio_tagging_loss=0.009058, over 3042540.66 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:56:01,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2574753.3333333335, ans=0.125
2023-11-23 22:56:13,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2574820.0, ans=0.125
2023-11-23 22:56:26,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2574886.6666666665, ans=0.125
2023-11-23 22:56:39,082 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.65 vs. limit=15.0
2023-11-23 22:56:42,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386250
2023-11-23 22:56:49,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.33 vs. limit=15.0
2023-11-23 22:56:56,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2575020.0, ans=0.125
2023-11-23 22:57:00,702 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1500, loss[loss=0.07438, simple_loss=0.1039, pruned_loss=0.01388, audio_tagging_loss=0.008556, over 14920.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09102, pruned_loss=0.01356, audio_tagging_loss=0.009157, over 3032130.61 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:57:06,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2575086.6666666665, ans=0.0
2023-11-23 22:57:13,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2575153.3333333335, ans=0.125
2023-11-23 22:57:26,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2575220.0, ans=0.0
2023-11-23 22:57:31,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2575220.0, ans=0.125
2023-11-23 22:57:32,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2575220.0, ans=0.0
2023-11-23 22:57:33,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2575220.0, ans=0.125
2023-11-23 22:57:37,785 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.32 vs. limit=15.0
2023-11-23 22:57:44,159 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386300
2023-11-23 22:57:45,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2575286.6666666665, ans=0.04949747468305833
2023-11-23 22:57:47,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.56 vs. limit=15.0
2023-11-23 22:57:49,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2575353.3333333335, ans=0.0
2023-11-23 22:57:51,766 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.322e+01 9.192e+01 9.802e+01 2.224e+02, threshold=1.838e+02, percent-clipped=1.0
2023-11-23 22:58:04,256 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1550, loss[loss=0.04698, simple_loss=0.05974, pruned_loss=0.008299, audio_tagging_loss=0.008809, over 16124.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09248, pruned_loss=0.01379, audio_tagging_loss=0.009058, over 3041348.25 frames. ], batch size: 63, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:58:08,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2575420.0, ans=0.1
2023-11-23 22:58:41,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2575620.0, ans=0.1
2023-11-23 22:58:44,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2575620.0, ans=0.125
2023-11-23 22:58:47,296 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386350
2023-11-23 22:58:56,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2575686.6666666665, ans=0.0
2023-11-23 22:59:02,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2575686.6666666665, ans=0.2
2023-11-23 22:59:05,876 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1600, loss[loss=0.07111, simple_loss=0.09121, pruned_loss=0.01728, audio_tagging_loss=0.008227, over 15421.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09207, pruned_loss=0.0136, audio_tagging_loss=0.009119, over 3039810.54 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:59:24,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2575820.0, ans=0.0
2023-11-23 22:59:36,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2575886.6666666665, ans=0.05
2023-11-23 22:59:46,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2575953.3333333335, ans=0.2
2023-11-23 22:59:49,993 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386400
2023-11-23 22:59:55,563 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.12 vs. limit=22.5
2023-11-23 22:59:57,364 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.368e+01 9.062e+01 9.713e+01 1.291e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 22:59:58,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2576020.0, ans=0.0
2023-11-23 23:00:08,291 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1650, loss[loss=0.06886, simple_loss=0.09089, pruned_loss=0.01442, audio_tagging_loss=0.008988, over 15618.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.0925, pruned_loss=0.01354, audio_tagging_loss=0.00908, over 3047457.35 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:00:31,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2576153.3333333335, ans=0.125
2023-11-23 23:00:37,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2576220.0, ans=0.125
2023-11-23 23:00:46,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2576286.6666666665, ans=0.0
2023-11-23 23:00:52,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386450
2023-11-23 23:00:56,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2576286.6666666665, ans=0.125
2023-11-23 23:01:01,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2576353.3333333335, ans=10.0
2023-11-23 23:01:12,027 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1700, loss[loss=0.08225, simple_loss=0.1186, pruned_loss=0.01522, audio_tagging_loss=0.00772, over 16069.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09204, pruned_loss=0.01371, audio_tagging_loss=0.00915, over 3052868.68 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:01:14,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2576420.0, ans=0.0
2023-11-23 23:01:18,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2576420.0, ans=0.05
2023-11-23 23:01:20,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.50 vs. limit=22.5
2023-11-23 23:01:39,164 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=15.0
2023-11-23 23:01:47,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2576620.0, ans=0.2
2023-11-23 23:01:53,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2576620.0, ans=0.0
2023-11-23 23:01:55,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386500
2023-11-23 23:02:03,012 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.350e+01 8.996e+01 9.717e+01 1.219e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 23:02:14,546 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1750, loss[loss=0.08257, simple_loss=0.1071, pruned_loss=0.01916, audio_tagging_loss=0.009885, over 14370.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09262, pruned_loss=0.01382, audio_tagging_loss=0.00903, over 3045372.82 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:02:40,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2576886.6666666665, ans=0.125
2023-11-23 23:02:42,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2576886.6666666665, ans=0.0
2023-11-23 23:02:43,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2576886.6666666665, ans=0.125
2023-11-23 23:02:47,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2576886.6666666665, ans=0.0
2023-11-23 23:02:50,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2576953.3333333335, ans=0.125
2023-11-23 23:02:58,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386550
2023-11-23 23:03:00,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2576953.3333333335, ans=0.1
2023-11-23 23:03:15,774 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1800, loss[loss=0.08791, simple_loss=0.1149, pruned_loss=0.02268, audio_tagging_loss=0.007784, over 15172.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.0928, pruned_loss=0.01382, audio_tagging_loss=0.008953, over 3046155.08 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:03:17,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2577086.6666666665, ans=0.125
2023-11-23 23:03:46,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2577220.0, ans=0.125
2023-11-23 23:03:52,438 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:03:59,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386600
2023-11-23 23:04:00,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2577286.6666666665, ans=0.125
2023-11-23 23:04:08,237 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.512e+01 8.514e+01 9.080e+01 9.768e+01 1.273e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 23:04:18,922 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1850, loss[loss=0.05315, simple_loss=0.0661, pruned_loss=0.008641, audio_tagging_loss=0.01146, over 16075.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09196, pruned_loss=0.01363, audio_tagging_loss=0.008983, over 3046539.80 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:04:22,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2577420.0, ans=0.1
2023-11-23 23:04:31,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2577486.6666666665, ans=0.125
2023-11-23 23:04:36,881 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.41 vs. limit=15.0
2023-11-23 23:04:37,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2577486.6666666665, ans=0.125
2023-11-23 23:04:40,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.33 vs. limit=15.0
2023-11-23 23:04:41,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2577486.6666666665, ans=0.1
2023-11-23 23:04:50,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2577553.3333333335, ans=0.125
2023-11-23 23:05:02,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386650
2023-11-23 23:05:20,244 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1900, loss[loss=0.07449, simple_loss=0.1042, pruned_loss=0.01172, audio_tagging_loss=0.01068, over 16099.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09123, pruned_loss=0.0134, audio_tagging_loss=0.008927, over 3052464.98 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:05:27,510 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.06 vs. limit=15.0
2023-11-23 23:06:04,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386700
2023-11-23 23:06:07,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2577953.3333333335, ans=0.025
2023-11-23 23:06:13,475 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.280e+01 8.966e+01 9.619e+01 1.252e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 23:06:23,277 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 1950, loss[loss=0.07323, simple_loss=0.09918, pruned_loss=0.01307, audio_tagging_loss=0.01057, over 15191.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09115, pruned_loss=0.01328, audio_tagging_loss=0.008902, over 3047384.79 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:06:36,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2578153.3333333335, ans=0.2
2023-11-23 23:06:47,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.12 vs. limit=15.0
2023-11-23 23:06:57,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2578220.0, ans=0.2
2023-11-23 23:07:07,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386750
2023-11-23 23:07:13,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2578353.3333333335, ans=0.2
2023-11-23 23:07:26,422 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2000, loss[loss=0.09525, simple_loss=0.1283, pruned_loss=0.02348, audio_tagging_loss=0.007642, over 14432.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09119, pruned_loss=0.01346, audio_tagging_loss=0.008982, over 3042363.95 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:07:26,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2578420.0, ans=0.125
2023-11-23 23:07:42,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2578486.6666666665, ans=0.1
2023-11-23 23:07:59,797 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:08:07,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2578620.0, ans=0.1
2023-11-23 23:08:08,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2578620.0, ans=0.125
2023-11-23 23:08:09,843 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386800
2023-11-23 23:08:09,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2578620.0, ans=0.0
2023-11-23 23:08:10,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2578620.0, ans=0.125
2023-11-23 23:08:19,215 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.226e+01 8.902e+01 9.752e+01 1.387e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 23:08:27,876 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:08:28,809 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2050, loss[loss=0.08685, simple_loss=0.1219, pruned_loss=0.01898, audio_tagging_loss=0.006937, over 14640.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09171, pruned_loss=0.01365, audio_tagging_loss=0.008947, over 3037263.20 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:08:46,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2578820.0, ans=0.1
2023-11-23 23:08:47,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2578820.0, ans=0.07
2023-11-23 23:08:57,148 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.98 vs. limit=12.0
2023-11-23 23:09:07,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2578953.3333333335, ans=0.1
2023-11-23 23:09:12,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386850
2023-11-23 23:09:22,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2579020.0, ans=0.125
2023-11-23 23:09:30,957 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2100, loss[loss=0.06943, simple_loss=0.09172, pruned_loss=0.0163, audio_tagging_loss=0.007276, over 15625.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09189, pruned_loss=0.01364, audio_tagging_loss=0.008855, over 3038265.82 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:09:36,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.29 vs. limit=10.0
2023-11-23 23:09:46,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2579153.3333333335, ans=0.2
2023-11-23 23:10:01,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2579220.0, ans=0.0
2023-11-23 23:10:12,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2579286.6666666665, ans=0.125
2023-11-23 23:10:14,342 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386900
2023-11-23 23:10:18,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2579286.6666666665, ans=0.125
2023-11-23 23:10:23,227 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.184e+01 8.426e+01 9.085e+01 9.880e+01 1.258e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 23:10:33,276 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2150, loss[loss=0.06494, simple_loss=0.08581, pruned_loss=0.01469, audio_tagging_loss=0.007336, over 15093.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.092, pruned_loss=0.01344, audio_tagging_loss=0.008801, over 3036939.37 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:10:39,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2579420.0, ans=0.125
2023-11-23 23:10:47,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2579486.6666666665, ans=0.2
2023-11-23 23:10:48,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2579486.6666666665, ans=0.2
2023-11-23 23:10:59,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2579553.3333333335, ans=0.125
2023-11-23 23:11:02,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2579553.3333333335, ans=0.125
2023-11-23 23:11:10,837 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:11:17,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 386950
2023-11-23 23:11:28,836 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.49 vs. limit=22.5
2023-11-23 23:11:36,325 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2200, loss[loss=0.07386, simple_loss=0.1031, pruned_loss=0.01461, audio_tagging_loss=0.007722, over 13893.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09267, pruned_loss=0.01357, audio_tagging_loss=0.008803, over 3037651.42 frames. ], batch size: 52, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:11:47,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2579820.0, ans=0.1
2023-11-23 23:11:49,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.13 vs. limit=22.5
2023-11-23 23:12:04,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.68 vs. limit=15.0
2023-11-23 23:12:08,913 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.78 vs. limit=10.0
2023-11-23 23:12:18,682 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.05 vs. limit=6.0
2023-11-23 23:12:20,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387000
2023-11-23 23:12:24,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0
2023-11-23 23:12:25,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2580020.0, ans=0.07
2023-11-23 23:12:29,890 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.440e+01 8.536e+01 9.005e+01 9.625e+01 2.688e+02, threshold=1.801e+02, percent-clipped=1.0
2023-11-23 23:12:35,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2580020.0, ans=0.125
2023-11-23 23:12:38,248 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2250, loss[loss=0.0683, simple_loss=0.09202, pruned_loss=0.01378, audio_tagging_loss=0.008513, over 16720.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09319, pruned_loss=0.01376, audio_tagging_loss=0.008768, over 3044332.17 frames. ], batch size: 63, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:12:44,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2580086.6666666665, ans=0.0
2023-11-23 23:12:48,001 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.93 vs. limit=10.0
2023-11-23 23:12:48,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2580086.6666666665, ans=0.0
2023-11-23 23:12:49,226 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.50 vs. limit=6.0
2023-11-23 23:12:53,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2580153.3333333335, ans=0.0
2023-11-23 23:13:06,138 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.39 vs. limit=12.0
2023-11-23 23:13:22,559 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387050
2023-11-23 23:13:31,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2580353.3333333335, ans=0.1
2023-11-23 23:13:37,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2580353.3333333335, ans=0.09899494936611666
2023-11-23 23:13:41,177 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2300, loss[loss=0.05259, simple_loss=0.06455, pruned_loss=0.009088, audio_tagging_loss=0.01123, over 14446.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09239, pruned_loss=0.01359, audio_tagging_loss=0.00887, over 3047536.39 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:13:51,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.78 vs. limit=12.0
2023-11-23 23:13:55,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2580486.6666666665, ans=0.1
2023-11-23 23:14:07,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2580553.3333333335, ans=0.125
2023-11-23 23:14:15,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2580553.3333333335, ans=0.125
2023-11-23 23:14:24,327 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387100
2023-11-23 23:14:24,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2580620.0, ans=0.125
2023-11-23 23:14:34,376 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.489e+01 9.011e+01 9.554e+01 1.239e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 23:14:36,334 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:14:37,020 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0
2023-11-23 23:14:37,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2580686.6666666665, ans=0.1
2023-11-23 23:14:43,682 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2350, loss[loss=0.0629, simple_loss=0.08175, pruned_loss=0.01292, audio_tagging_loss=0.009103, over 15472.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09171, pruned_loss=0.01355, audio_tagging_loss=0.008969, over 3039008.10 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:14:51,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2580753.3333333335, ans=0.125
2023-11-23 23:15:01,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2580820.0, ans=0.09899494936611666
2023-11-23 23:15:18,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2580953.3333333335, ans=0.0
2023-11-23 23:15:21,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.95 vs. limit=22.5
2023-11-23 23:15:27,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387150
2023-11-23 23:15:33,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.91 vs. limit=22.5
2023-11-23 23:15:36,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2581020.0, ans=0.1
2023-11-23 23:15:37,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2581020.0, ans=0.125
2023-11-23 23:15:44,720 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2400, loss[loss=0.06146, simple_loss=0.06709, pruned_loss=0.01201, audio_tagging_loss=0.01591, over 13925.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09228, pruned_loss=0.01374, audio_tagging_loss=0.009069, over 3045478.91 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:16:14,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2581220.0, ans=0.125
2023-11-23 23:16:28,342 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387200
2023-11-23 23:16:37,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2581353.3333333335, ans=0.125
2023-11-23 23:16:39,150 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.668e+01 9.147e+01 9.893e+01 1.218e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 23:16:44,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2581353.3333333335, ans=0.125
2023-11-23 23:16:46,719 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2450, loss[loss=0.05736, simple_loss=0.07831, pruned_loss=0.01019, audio_tagging_loss=0.00802, over 15408.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09238, pruned_loss=0.01373, audio_tagging_loss=0.009117, over 3036202.11 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:17:01,893 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-23 23:17:25,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2581620.0, ans=0.125
2023-11-23 23:17:29,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387250
2023-11-23 23:17:31,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2581620.0, ans=0.125
2023-11-23 23:17:50,310 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2500, loss[loss=0.06187, simple_loss=0.08329, pruned_loss=0.01078, audio_tagging_loss=0.009436, over 16187.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09115, pruned_loss=0.01339, audio_tagging_loss=0.009229, over 3042406.37 frames. ], batch size: 63, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:17:57,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2581753.3333333335, ans=0.125
2023-11-23 23:18:07,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2581820.0, ans=0.0
2023-11-23 23:18:08,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2581820.0, ans=0.0
2023-11-23 23:18:09,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2581820.0, ans=0.09899494936611666
2023-11-23 23:18:17,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2581886.6666666665, ans=0.125
2023-11-23 23:18:33,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387300
2023-11-23 23:18:35,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2581953.3333333335, ans=0.1
2023-11-23 23:18:44,135 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.436e+01 9.159e+01 9.960e+01 1.411e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-23 23:18:51,370 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2550, loss[loss=0.05684, simple_loss=0.08137, pruned_loss=0.008237, audio_tagging_loss=0.007922, over 15366.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09071, pruned_loss=0.01325, audio_tagging_loss=0.009229, over 3039865.79 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:18:53,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2582086.6666666665, ans=0.0
2023-11-23 23:18:56,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2582086.6666666665, ans=0.1
2023-11-23 23:19:02,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=2582153.3333333335, ans=0.025
2023-11-23 23:19:06,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2582153.3333333335, ans=0.1
2023-11-23 23:19:19,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2582220.0, ans=0.0
2023-11-23 23:19:22,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=2582220.0, ans=0.1
2023-11-23 23:19:35,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387350
2023-11-23 23:19:35,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2582286.6666666665, ans=0.0
2023-11-23 23:19:52,917 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2600, loss[loss=0.05821, simple_loss=0.08016, pruned_loss=0.01268, audio_tagging_loss=0.005452, over 15608.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.08977, pruned_loss=0.01314, audio_tagging_loss=0.009168, over 3033487.18 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:20:15,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.32 vs. limit=15.0
2023-11-23 23:20:31,253 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0
2023-11-23 23:20:34,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2582620.0, ans=0.125
2023-11-23 23:20:36,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387400
2023-11-23 23:20:45,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2582686.6666666665, ans=0.95
2023-11-23 23:20:48,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2582686.6666666665, ans=0.2
2023-11-23 23:20:48,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2582686.6666666665, ans=0.125
2023-11-23 23:20:48,950 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.392e+01 8.863e+01 9.833e+01 1.286e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 23:20:56,692 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2650, loss[loss=0.06993, simple_loss=0.09738, pruned_loss=0.01465, audio_tagging_loss=0.006584, over 15533.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09049, pruned_loss=0.01324, audio_tagging_loss=0.009154, over 3039211.93 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:21:16,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2582820.0, ans=0.2
2023-11-23 23:21:28,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2582886.6666666665, ans=0.125
2023-11-23 23:21:40,175 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387450
2023-11-23 23:21:58,877 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2700, loss[loss=0.04556, simple_loss=0.06322, pruned_loss=0.006713, audio_tagging_loss=0.007235, over 15908.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09012, pruned_loss=0.0133, audio_tagging_loss=0.00908, over 3040575.62 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:22:14,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2583153.3333333335, ans=0.125
2023-11-23 23:22:16,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2583153.3333333335, ans=0.0
2023-11-23 23:22:28,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2583220.0, ans=0.125
2023-11-23 23:22:30,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2583220.0, ans=0.0
2023-11-23 23:22:42,403 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387500
2023-11-23 23:22:54,186 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.525e+01 8.709e+01 9.550e+01 1.033e+02 1.468e+02, threshold=1.910e+02, percent-clipped=0.0
2023-11-23 23:22:55,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2583353.3333333335, ans=0.2
2023-11-23 23:23:00,138 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2750, loss[loss=0.04613, simple_loss=0.05736, pruned_loss=0.008418, audio_tagging_loss=0.009032, over 14932.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.08949, pruned_loss=0.01326, audio_tagging_loss=0.009107, over 3034215.09 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:23:18,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2583486.6666666665, ans=0.125
2023-11-23 23:23:18,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2583486.6666666665, ans=0.2
2023-11-23 23:23:22,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2583486.6666666665, ans=0.125
2023-11-23 23:23:31,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2583553.3333333335, ans=0.0
2023-11-23 23:23:44,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387550
2023-11-23 23:23:44,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2583620.0, ans=0.125
2023-11-23 23:23:54,120 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:23:57,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2583686.6666666665, ans=0.125
2023-11-23 23:24:02,860 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2800, loss[loss=0.05951, simple_loss=0.08064, pruned_loss=0.008744, audio_tagging_loss=0.01044, over 14852.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09008, pruned_loss=0.01334, audio_tagging_loss=0.009021, over 3043133.01 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:24:34,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2583886.6666666665, ans=0.125
2023-11-23 23:24:46,983 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387600
2023-11-23 23:24:56,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2584020.0, ans=0.0
2023-11-23 23:24:56,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2584020.0, ans=0.125
2023-11-23 23:24:59,634 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.278e+01 8.804e+01 9.483e+01 1.291e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 23:25:06,135 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2850, loss[loss=0.07077, simple_loss=0.09835, pruned_loss=0.01549, audio_tagging_loss=0.006103, over 15723.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.08983, pruned_loss=0.01323, audio_tagging_loss=0.009013, over 3040334.23 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:25:07,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2584086.6666666665, ans=0.125
2023-11-23 23:25:27,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2584153.3333333335, ans=0.125
2023-11-23 23:25:37,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2584220.0, ans=0.1
2023-11-23 23:25:49,593 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387650
2023-11-23 23:26:02,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2584353.3333333335, ans=0.5
2023-11-23 23:26:07,803 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2900, loss[loss=0.0798, simple_loss=0.09985, pruned_loss=0.02075, audio_tagging_loss=0.009126, over 14564.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09021, pruned_loss=0.01329, audio_tagging_loss=0.009055, over 3036488.72 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:26:14,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.64 vs. limit=6.0
2023-11-23 23:26:51,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387700
2023-11-23 23:27:05,508 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.293e+01 9.102e+01 9.710e+01 1.234e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-23 23:27:09,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.35 vs. limit=15.0
2023-11-23 23:27:10,861 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 2950, loss[loss=0.0671, simple_loss=0.08487, pruned_loss=0.01387, audio_tagging_loss=0.0108, over 15127.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09011, pruned_loss=0.01331, audio_tagging_loss=0.009138, over 3039816.50 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:27:22,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2584820.0, ans=0.125
2023-11-23 23:27:43,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2584886.6666666665, ans=0.125
2023-11-23 23:27:46,945 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.70 vs. limit=15.0
2023-11-23 23:27:47,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2584953.3333333335, ans=0.125
2023-11-23 23:27:49,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2584953.3333333335, ans=0.125
2023-11-23 23:27:54,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387750
2023-11-23 23:28:02,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2585020.0, ans=0.125
2023-11-23 23:28:09,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2585020.0, ans=0.125
2023-11-23 23:28:13,248 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3000, loss[loss=0.06024, simple_loss=0.07915, pruned_loss=0.01305, audio_tagging_loss=0.007624, over 15293.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09079, pruned_loss=0.01351, audio_tagging_loss=0.009191, over 3041143.32 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:28:13,249 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-23 23:28:53,118 INFO [train_asr.py:1253] (2/4) Epoch 33, validation: loss=0.05846, simple_loss=0.05103, pruned_loss=0.005194, audio_tagging_loss=0.02775, over 4681554.00 frames. 
2023-11-23 23:28:53,119 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-23 23:28:59,866 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.00 vs. limit=22.5
2023-11-23 23:29:12,381 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.35 vs. limit=15.0
2023-11-23 23:29:23,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2585220.0, ans=0.125
2023-11-23 23:29:37,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387800
2023-11-23 23:29:42,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2585353.3333333335, ans=0.0
2023-11-23 23:29:44,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2585353.3333333335, ans=0.125
2023-11-23 23:29:51,586 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.566e+01 9.149e+01 9.920e+01 1.258e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 23:29:56,967 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3050, loss[loss=0.0852, simple_loss=0.1083, pruned_loss=0.02263, audio_tagging_loss=0.008408, over 15715.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09114, pruned_loss=0.01366, audio_tagging_loss=0.009169, over 3039248.62 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:30:15,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2585486.6666666665, ans=0.125
2023-11-23 23:30:35,035 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:30:42,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387850
2023-11-23 23:30:59,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2585686.6666666665, ans=0.95
2023-11-23 23:31:01,218 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3100, loss[loss=0.06296, simple_loss=0.06973, pruned_loss=0.01326, audio_tagging_loss=0.01483, over 13457.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09203, pruned_loss=0.01376, audio_tagging_loss=0.009345, over 3038958.13 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:31:14,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2585820.0, ans=0.125
2023-11-23 23:31:23,890 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:31:45,694 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387900
2023-11-23 23:31:51,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.27 vs. limit=12.0
2023-11-23 23:31:59,125 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.717e+01 9.122e+01 9.882e+01 2.384e+02, threshold=1.824e+02, percent-clipped=1.0
2023-11-23 23:32:03,858 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3150, loss[loss=0.07497, simple_loss=0.1008, pruned_loss=0.01469, audio_tagging_loss=0.009876, over 15352.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09267, pruned_loss=0.01382, audio_tagging_loss=0.009321, over 3041328.98 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:32:11,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2586086.6666666665, ans=0.0
2023-11-23 23:32:28,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2586220.0, ans=0.125
2023-11-23 23:32:33,508 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.91 vs. limit=22.5
2023-11-23 23:32:34,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2586220.0, ans=0.125
2023-11-23 23:32:42,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2586286.6666666665, ans=0.125
2023-11-23 23:32:47,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 387950
2023-11-23 23:33:01,279 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:33:05,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2586420.0, ans=0.0
2023-11-23 23:33:06,481 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3200, loss[loss=0.06197, simple_loss=0.09278, pruned_loss=0.007018, audio_tagging_loss=0.00856, over 15656.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09303, pruned_loss=0.01375, audio_tagging_loss=0.009261, over 3041328.46 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:33:07,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2586420.0, ans=0.0
2023-11-23 23:33:26,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2586486.6666666665, ans=0.0
2023-11-23 23:33:50,063 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388000
2023-11-23 23:34:04,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2586686.6666666665, ans=0.025
2023-11-23 23:34:06,875 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.288e+01 8.870e+01 9.620e+01 1.279e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 23:34:11,757 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3250, loss[loss=0.04921, simple_loss=0.05675, pruned_loss=0.008143, audio_tagging_loss=0.01269, over 15407.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09266, pruned_loss=0.01355, audio_tagging_loss=0.009299, over 3038571.81 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:34:53,506 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.62 vs. limit=15.0
2023-11-23 23:34:56,641 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388050
2023-11-23 23:35:09,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.81 vs. limit=15.0
2023-11-23 23:35:15,190 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3300, loss[loss=0.04828, simple_loss=0.06001, pruned_loss=0.007771, audio_tagging_loss=0.0105, over 15344.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09149, pruned_loss=0.0133, audio_tagging_loss=0.009335, over 3043632.68 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:35:32,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2587153.3333333335, ans=0.2
2023-11-23 23:35:34,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2587153.3333333335, ans=0.0
2023-11-23 23:35:58,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388100
2023-11-23 23:36:01,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2587286.6666666665, ans=0.2
2023-11-23 23:36:04,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2587353.3333333335, ans=0.0
2023-11-23 23:36:12,198 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.544e+01 9.048e+01 9.852e+01 1.338e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 23:36:12,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2587353.3333333335, ans=0.125
2023-11-23 23:36:14,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2587353.3333333335, ans=0.1
2023-11-23 23:36:18,200 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3350, loss[loss=0.07213, simple_loss=0.09443, pruned_loss=0.01403, audio_tagging_loss=0.01088, over 15143.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09184, pruned_loss=0.0132, audio_tagging_loss=0.009319, over 3051526.48 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:36:29,238 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0
2023-11-23 23:36:34,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2587486.6666666665, ans=0.1
2023-11-23 23:36:40,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2587486.6666666665, ans=0.125
2023-11-23 23:36:56,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2587620.0, ans=0.025
2023-11-23 23:37:02,065 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388150
2023-11-23 23:37:05,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.21 vs. limit=15.0
2023-11-23 23:37:15,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2587686.6666666665, ans=0.2
2023-11-23 23:37:17,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2587686.6666666665, ans=0.5
2023-11-23 23:37:20,747 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3400, loss[loss=0.05752, simple_loss=0.07897, pruned_loss=0.008842, audio_tagging_loss=0.009195, over 16104.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09227, pruned_loss=0.01335, audio_tagging_loss=0.00915, over 3049456.23 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:37:28,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2587753.3333333335, ans=0.0
2023-11-23 23:37:28,443 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.79 vs. limit=10.0
2023-11-23 23:37:45,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2587886.6666666665, ans=0.125
2023-11-23 23:38:04,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388200
2023-11-23 23:38:07,735 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.61 vs. limit=15.0
2023-11-23 23:38:10,054 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.18 vs. limit=15.0
2023-11-23 23:38:16,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2588020.0, ans=0.125
2023-11-23 23:38:17,822 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.375e+01 8.962e+01 9.583e+01 1.424e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 23:38:22,590 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3450, loss[loss=0.07232, simple_loss=0.09951, pruned_loss=0.01604, audio_tagging_loss=0.006536, over 14838.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.0928, pruned_loss=0.01369, audio_tagging_loss=0.009117, over 3054748.36 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:38:33,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2588086.6666666665, ans=0.125
2023-11-23 23:38:56,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2588220.0, ans=0.125
2023-11-23 23:39:05,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2588286.6666666665, ans=0.2
2023-11-23 23:39:07,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388250
2023-11-23 23:39:16,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2588353.3333333335, ans=0.0
2023-11-23 23:39:18,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2588353.3333333335, ans=0.0
2023-11-23 23:39:26,124 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3500, loss[loss=0.07253, simple_loss=0.1021, pruned_loss=0.01233, audio_tagging_loss=0.009148, over 16059.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09277, pruned_loss=0.01376, audio_tagging_loss=0.00898, over 3051913.96 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:39:28,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2588420.0, ans=0.035
2023-11-23 23:39:35,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2588420.0, ans=0.0
2023-11-23 23:39:55,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2588553.3333333335, ans=0.0
2023-11-23 23:39:57,873 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:39:59,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2588553.3333333335, ans=0.1
2023-11-23 23:40:09,071 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388300
2023-11-23 23:40:20,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2588686.6666666665, ans=15.0
2023-11-23 23:40:23,682 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.607e+01 9.289e+01 1.010e+02 1.260e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-23 23:40:25,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2588686.6666666665, ans=0.125
2023-11-23 23:40:28,514 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3550, loss[loss=0.0757, simple_loss=0.1044, pruned_loss=0.01312, audio_tagging_loss=0.0104, over 16042.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.0915, pruned_loss=0.01361, audio_tagging_loss=0.008964, over 3051151.63 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:40:28,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2588753.3333333335, ans=0.05
2023-11-23 23:40:40,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2588820.0, ans=0.0
2023-11-23 23:40:40,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2588820.0, ans=0.0
2023-11-23 23:40:45,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2588820.0, ans=0.2
2023-11-23 23:40:48,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2588820.0, ans=0.09899494936611666
2023-11-23 23:40:54,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2588886.6666666665, ans=0.1
2023-11-23 23:41:12,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388350
2023-11-23 23:41:23,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2589020.0, ans=0.125
2023-11-23 23:41:24,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.13 vs. limit=15.0
2023-11-23 23:41:30,136 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3600, loss[loss=0.06925, simple_loss=0.09324, pruned_loss=0.01381, audio_tagging_loss=0.008812, over 15330.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09154, pruned_loss=0.01362, audio_tagging_loss=0.008918, over 3054768.73 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:41:34,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2589086.6666666665, ans=0.0
2023-11-23 23:41:44,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2589153.3333333335, ans=0.125
2023-11-23 23:41:45,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2589153.3333333335, ans=0.0
2023-11-23 23:41:58,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2589220.0, ans=0.125
2023-11-23 23:42:14,130 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388400
2023-11-23 23:42:28,667 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.372e+01 8.860e+01 9.569e+01 1.357e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 23:42:32,302 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3650, loss[loss=0.08739, simple_loss=0.1277, pruned_loss=0.01856, audio_tagging_loss=0.004995, over 15264.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09186, pruned_loss=0.01368, audio_tagging_loss=0.00883, over 3052961.35 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:42:32,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2589420.0, ans=0.1
2023-11-23 23:42:36,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2589420.0, ans=0.04949747468305833
2023-11-23 23:42:53,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2589486.6666666665, ans=0.2
2023-11-23 23:42:53,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2589486.6666666665, ans=0.125
2023-11-23 23:43:07,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2589553.3333333335, ans=0.95
2023-11-23 23:43:16,787 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388450
2023-11-23 23:43:35,933 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3700, loss[loss=0.07289, simple_loss=0.09661, pruned_loss=0.01442, audio_tagging_loss=0.01016, over 15648.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09192, pruned_loss=0.01362, audio_tagging_loss=0.008846, over 3054061.29 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:43:39,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2589753.3333333335, ans=0.1
2023-11-23 23:43:46,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.95 vs. limit=22.5
2023-11-23 23:43:50,121 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:43:51,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2589820.0, ans=0.125
2023-11-23 23:43:55,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2589820.0, ans=0.0
2023-11-23 23:44:19,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388500
2023-11-23 23:44:27,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2590020.0, ans=0.0
2023-11-23 23:44:33,985 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.486e+01 9.089e+01 9.768e+01 1.177e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 23:44:37,528 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3750, loss[loss=0.05695, simple_loss=0.0845, pruned_loss=0.007318, audio_tagging_loss=0.007378, over 16057.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09238, pruned_loss=0.01361, audio_tagging_loss=0.008896, over 3061725.25 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:44:41,605 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=14.11 vs. limit=15.0
2023-11-23 23:44:50,248 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.87 vs. limit=15.0
2023-11-23 23:44:58,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2590153.3333333335, ans=0.0
2023-11-23 23:45:15,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=2590286.6666666665, ans=0.2
2023-11-23 23:45:21,618 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:45:21,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388550
2023-11-23 23:45:23,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2590286.6666666665, ans=0.025
2023-11-23 23:45:34,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2590353.3333333335, ans=0.0
2023-11-23 23:45:39,698 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3800, loss[loss=0.0851, simple_loss=0.1124, pruned_loss=0.0186, audio_tagging_loss=0.01028, over 15042.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09213, pruned_loss=0.0135, audio_tagging_loss=0.008985, over 3059622.59 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:45:41,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2590420.0, ans=0.125
2023-11-23 23:46:18,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2590620.0, ans=0.125
2023-11-23 23:46:23,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388600
2023-11-23 23:46:39,771 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 8.380e+01 8.887e+01 9.611e+01 1.323e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 23:46:44,011 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3850, loss[loss=0.05054, simple_loss=0.05988, pruned_loss=0.0119, audio_tagging_loss=0.008699, over 15603.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09077, pruned_loss=0.01324, audio_tagging_loss=0.009038, over 3059016.91 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:46:49,567 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.67 vs. limit=15.0
2023-11-23 23:47:04,353 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.68 vs. limit=15.0
2023-11-23 23:47:07,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2590886.6666666665, ans=0.0
2023-11-23 23:47:13,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2590886.6666666665, ans=0.2
2023-11-23 23:47:23,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2590953.3333333335, ans=0.04949747468305833
2023-11-23 23:47:28,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388650
2023-11-23 23:47:30,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.05 vs. limit=10.0
2023-11-23 23:47:33,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2590953.3333333335, ans=0.0
2023-11-23 23:47:37,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2591020.0, ans=0.125
2023-11-23 23:47:42,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2591020.0, ans=0.125
2023-11-23 23:47:47,379 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3900, loss[loss=0.08419, simple_loss=0.1195, pruned_loss=0.01363, audio_tagging_loss=0.0108, over 15104.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09105, pruned_loss=0.01322, audio_tagging_loss=0.009184, over 3058618.18 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:47:59,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2591153.3333333335, ans=0.05
2023-11-23 23:48:12,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2591220.0, ans=0.0
2023-11-23 23:48:15,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2591220.0, ans=0.0
2023-11-23 23:48:32,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388700
2023-11-23 23:48:32,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2591286.6666666665, ans=0.0
2023-11-23 23:48:38,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2591353.3333333335, ans=0.125
2023-11-23 23:48:46,588 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.330e+01 8.934e+01 9.660e+01 1.290e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 23:48:50,234 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 3950, loss[loss=0.05393, simple_loss=0.0668, pruned_loss=0.0106, audio_tagging_loss=0.009922, over 14683.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09043, pruned_loss=0.01305, audio_tagging_loss=0.009339, over 3054717.84 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:48:57,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2591420.0, ans=0.125
2023-11-23 23:49:29,965 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.40 vs. limit=22.5
2023-11-23 23:49:34,227 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388750
2023-11-23 23:49:34,766 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.29 vs. limit=15.0
2023-11-23 23:49:39,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2591686.6666666665, ans=0.125
2023-11-23 23:49:41,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2591686.6666666665, ans=0.125
2023-11-23 23:49:53,534 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4000, loss[loss=0.07593, simple_loss=0.09541, pruned_loss=0.01825, audio_tagging_loss=0.009975, over 14548.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09119, pruned_loss=0.01345, audio_tagging_loss=0.009368, over 3048965.13 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:50:17,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2591886.6666666665, ans=0.0
2023-11-23 23:50:30,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2591953.3333333335, ans=0.1
2023-11-23 23:50:36,376 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.73 vs. limit=22.5
2023-11-23 23:50:37,127 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388800
2023-11-23 23:50:54,120 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.337e+01 9.138e+01 9.888e+01 1.432e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 23:50:56,665 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4050, loss[loss=0.08968, simple_loss=0.1132, pruned_loss=0.02393, audio_tagging_loss=0.009164, over 14792.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09145, pruned_loss=0.01347, audio_tagging_loss=0.009369, over 3046791.60 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:51:00,284 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:51:10,481 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.37 vs. limit=15.0
2023-11-23 23:51:35,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2592286.6666666665, ans=0.125
2023-11-23 23:51:40,568 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388850
2023-11-23 23:51:43,384 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.72 vs. limit=15.0
2023-11-23 23:51:45,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2592353.3333333335, ans=0.1
2023-11-23 23:51:58,738 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4100, loss[loss=0.05955, simple_loss=0.08239, pruned_loss=0.009953, audio_tagging_loss=0.008404, over 16144.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09189, pruned_loss=0.01356, audio_tagging_loss=0.009379, over 3048304.93 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:52:03,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2592420.0, ans=10.0
2023-11-23 23:52:22,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2592486.6666666665, ans=0.125
2023-11-23 23:52:43,277 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388900
2023-11-23 23:52:44,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2592620.0, ans=0.125
2023-11-23 23:52:59,651 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.379e+01 9.260e+01 9.829e+01 1.268e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-23 23:52:59,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2592686.6666666665, ans=0.05
2023-11-23 23:53:02,088 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4150, loss[loss=0.05925, simple_loss=0.07907, pruned_loss=0.01051, audio_tagging_loss=0.00921, over 15724.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09241, pruned_loss=0.01363, audio_tagging_loss=0.009198, over 3050953.94 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:53:02,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2592753.3333333335, ans=0.2
2023-11-23 23:53:12,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2592753.3333333335, ans=0.0
2023-11-23 23:53:24,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2592820.0, ans=0.125
2023-11-23 23:53:30,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2592886.6666666665, ans=0.125
2023-11-23 23:53:43,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2592953.3333333335, ans=0.0
2023-11-23 23:53:45,372 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 388950
2023-11-23 23:53:48,196 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:54:03,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2593086.6666666665, ans=0.1
2023-11-23 23:54:04,244 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4200, loss[loss=0.06157, simple_loss=0.07205, pruned_loss=0.01166, audio_tagging_loss=0.01389, over 16121.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09283, pruned_loss=0.01377, audio_tagging_loss=0.009142, over 3047774.30 frames. ], batch size: 63, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:54:04,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.02 vs. limit=15.0
2023-11-23 23:54:20,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2593153.3333333335, ans=0.125
2023-11-23 23:54:30,273 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.52 vs. limit=15.0
2023-11-23 23:54:39,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2593220.0, ans=0.125
2023-11-23 23:54:44,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2593286.6666666665, ans=0.125
2023-11-23 23:54:48,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389000
2023-11-23 23:54:50,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2593286.6666666665, ans=0.0
2023-11-23 23:54:58,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2593353.3333333335, ans=0.125
2023-11-23 23:55:04,545 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.440e+01 9.074e+01 9.920e+01 1.262e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 23:55:07,028 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4250, loss[loss=0.0844, simple_loss=0.1137, pruned_loss=0.0191, audio_tagging_loss=0.008459, over 15344.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09321, pruned_loss=0.01379, audio_tagging_loss=0.009065, over 3045048.05 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:55:15,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2593420.0, ans=0.1
2023-11-23 23:55:18,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2593486.6666666665, ans=0.125
2023-11-23 23:55:20,153 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.52 vs. limit=15.0
2023-11-23 23:55:47,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2593620.0, ans=0.0
2023-11-23 23:55:50,181 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.48 vs. limit=15.0
2023-11-23 23:55:51,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389050
2023-11-23 23:56:10,050 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4300, loss[loss=0.06083, simple_loss=0.07399, pruned_loss=0.01181, audio_tagging_loss=0.01203, over 14261.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09347, pruned_loss=0.014, audio_tagging_loss=0.008924, over 3045723.17 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:56:38,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2593886.6666666665, ans=0.07
2023-11-23 23:56:53,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389100
2023-11-23 23:57:10,080 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.164e+01 8.627e+01 9.104e+01 9.923e+01 1.149e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-23 23:57:12,439 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4350, loss[loss=0.05801, simple_loss=0.07806, pruned_loss=0.01063, audio_tagging_loss=0.008353, over 15122.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09291, pruned_loss=0.01401, audio_tagging_loss=0.008922, over 3043915.34 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:57:18,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2594086.6666666665, ans=0.125
2023-11-23 23:57:56,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389150
2023-11-23 23:57:58,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2594286.6666666665, ans=0.0
2023-11-23 23:58:06,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2594353.3333333335, ans=0.125
2023-11-23 23:58:15,039 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4400, loss[loss=0.057, simple_loss=0.08306, pruned_loss=0.00769, audio_tagging_loss=0.007779, over 14930.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09259, pruned_loss=0.01391, audio_tagging_loss=0.008836, over 3043753.87 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0
2023-11-23 23:58:15,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2594420.0, ans=0.1
2023-11-23 23:58:36,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2594486.6666666665, ans=0.2
2023-11-23 23:58:38,339 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.76 vs. limit=15.0
2023-11-23 23:58:58,454 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389200
2023-11-23 23:59:13,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2594686.6666666665, ans=0.0
2023-11-23 23:59:14,485 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.468e+01 8.522e+01 9.303e+01 9.983e+01 1.286e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-23 23:59:16,887 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4450, loss[loss=0.06063, simple_loss=0.07913, pruned_loss=0.009609, audio_tagging_loss=0.01146, over 15979.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09239, pruned_loss=0.01384, audio_tagging_loss=0.008823, over 3049942.77 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 32.0
2023-11-23 23:59:48,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2594886.6666666665, ans=0.125
2023-11-23 23:59:55,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2594953.3333333335, ans=0.125
2023-11-24 00:00:00,979 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389250
2023-11-24 00:00:08,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0
2023-11-24 00:00:20,336 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4500, loss[loss=0.06675, simple_loss=0.08421, pruned_loss=0.01433, audio_tagging_loss=0.01031, over 14377.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09271, pruned_loss=0.01375, audio_tagging_loss=0.008776, over 3055292.49 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:00:36,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2595153.3333333335, ans=0.2
2023-11-24 00:01:04,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389300
2023-11-24 00:01:08,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2595286.6666666665, ans=0.125
2023-11-24 00:01:19,570 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.819e+01 8.069e+01 8.788e+01 9.879e+01 1.178e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-24 00:01:21,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2595420.0, ans=0.0
2023-11-24 00:01:22,013 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4550, loss[loss=0.07882, simple_loss=0.1004, pruned_loss=0.02026, audio_tagging_loss=0.008352, over 14934.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09247, pruned_loss=0.01373, audio_tagging_loss=0.008805, over 3046866.39 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:01:43,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2595486.6666666665, ans=0.0
2023-11-24 00:01:58,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2595620.0, ans=0.125
2023-11-24 00:02:03,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2595620.0, ans=0.1
2023-11-24 00:02:05,550 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389350
2023-11-24 00:02:10,186 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:02:20,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2595686.6666666665, ans=0.0
2023-11-24 00:02:23,778 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4600, loss[loss=0.06921, simple_loss=0.1042, pruned_loss=0.01096, audio_tagging_loss=0.006166, over 15174.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09209, pruned_loss=0.01353, audio_tagging_loss=0.008959, over 3043518.69 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:02:30,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2595753.3333333335, ans=0.0
2023-11-24 00:02:49,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2595886.6666666665, ans=0.2
2023-11-24 00:02:50,523 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.53 vs. limit=15.0
2023-11-24 00:03:02,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2595953.3333333335, ans=10.0
2023-11-24 00:03:06,744 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389400
2023-11-24 00:03:10,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2595953.3333333335, ans=0.0
2023-11-24 00:03:24,567 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.634e+01 9.201e+01 1.007e+02 1.274e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 00:03:26,980 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4650, loss[loss=0.07883, simple_loss=0.1071, pruned_loss=0.01664, audio_tagging_loss=0.008618, over 15607.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09102, pruned_loss=0.01334, audio_tagging_loss=0.009039, over 3047109.77 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:03:28,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=2596086.6666666665, ans=0.02
2023-11-24 00:03:37,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2596086.6666666665, ans=15.0
2023-11-24 00:03:37,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2596153.3333333335, ans=0.0
2023-11-24 00:03:57,009 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.41 vs. limit=15.0
2023-11-24 00:04:06,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.71 vs. limit=12.0
2023-11-24 00:04:10,453 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389450
2023-11-24 00:04:17,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2596353.3333333335, ans=0.125
2023-11-24 00:04:17,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.62 vs. limit=15.0
2023-11-24 00:04:19,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2596353.3333333335, ans=0.0
2023-11-24 00:04:28,870 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4700, loss[loss=0.05457, simple_loss=0.06948, pruned_loss=0.01022, audio_tagging_loss=0.009606, over 15706.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09043, pruned_loss=0.01338, audio_tagging_loss=0.009125, over 3043582.18 frames. ], batch size: 62, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:04:55,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2596553.3333333335, ans=0.125
2023-11-24 00:05:01,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2596553.3333333335, ans=0.125
2023-11-24 00:05:04,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2596553.3333333335, ans=0.125
2023-11-24 00:05:12,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389500
2023-11-24 00:05:28,428 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.415e+01 9.088e+01 1.001e+02 1.234e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 00:05:30,855 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4750, loss[loss=0.07942, simple_loss=0.09981, pruned_loss=0.01986, audio_tagging_loss=0.009654, over 14421.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09133, pruned_loss=0.01339, audio_tagging_loss=0.009194, over 3043459.17 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:06:07,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2596886.6666666665, ans=0.0
2023-11-24 00:06:14,939 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389550
2023-11-24 00:06:35,073 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4800, loss[loss=0.0738, simple_loss=0.1029, pruned_loss=0.01541, audio_tagging_loss=0.006967, over 15274.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09056, pruned_loss=0.01325, audio_tagging_loss=0.009343, over 3048091.65 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:06:35,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2597086.6666666665, ans=0.0
2023-11-24 00:06:35,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2597086.6666666665, ans=0.125
2023-11-24 00:06:45,979 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:06:46,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2597153.3333333335, ans=0.125
2023-11-24 00:06:48,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=2597153.3333333335, ans=0.02
2023-11-24 00:06:52,776 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.67 vs. limit=22.5
2023-11-24 00:07:10,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2597286.6666666665, ans=0.2
2023-11-24 00:07:19,037 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389600
2023-11-24 00:07:25,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.75 vs. limit=12.0
2023-11-24 00:07:34,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2597353.3333333335, ans=0.0
2023-11-24 00:07:37,707 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.181e+01 8.337e+01 8.951e+01 9.611e+01 1.135e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-24 00:07:37,753 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4850, loss[loss=0.07007, simple_loss=0.09962, pruned_loss=0.01203, audio_tagging_loss=0.008227, over 15737.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09113, pruned_loss=0.01328, audio_tagging_loss=0.009332, over 3048497.47 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:07:47,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2597420.0, ans=0.125
2023-11-24 00:08:06,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2597553.3333333335, ans=0.04949747468305833
2023-11-24 00:08:15,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2597620.0, ans=0.125
2023-11-24 00:08:18,556 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=12.0
2023-11-24 00:08:21,543 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389650
2023-11-24 00:08:23,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2597620.0, ans=0.0
2023-11-24 00:08:39,297 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4900, loss[loss=0.06156, simple_loss=0.07819, pruned_loss=0.01193, audio_tagging_loss=0.01054, over 14786.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09059, pruned_loss=0.01321, audio_tagging_loss=0.009307, over 3050156.20 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:09:02,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2597820.0, ans=0.95
2023-11-24 00:09:11,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.34 vs. limit=10.0
2023-11-24 00:09:23,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389700
2023-11-24 00:09:31,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=12.0
2023-11-24 00:09:43,035 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.493e+01 8.944e+01 9.853e+01 1.226e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-24 00:09:43,080 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 4950, loss[loss=0.0773, simple_loss=0.1068, pruned_loss=0.01466, audio_tagging_loss=0.009244, over 15145.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.0916, pruned_loss=0.0134, audio_tagging_loss=0.00918, over 3053299.94 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:10:13,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2598220.0, ans=0.2
2023-11-24 00:10:26,119 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389750
2023-11-24 00:10:28,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2598286.6666666665, ans=0.0
2023-11-24 00:10:40,610 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.09 vs. limit=22.5
2023-11-24 00:10:45,889 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5000, loss[loss=0.09127, simple_loss=0.1265, pruned_loss=0.02045, audio_tagging_loss=0.007552, over 14925.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09183, pruned_loss=0.01348, audio_tagging_loss=0.009018, over 3052076.95 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:11:10,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2598553.3333333335, ans=0.1
2023-11-24 00:11:29,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389800
2023-11-24 00:11:29,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2598620.0, ans=0.125
2023-11-24 00:11:35,265 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.33 vs. limit=22.5
2023-11-24 00:11:43,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.68 vs. limit=10.0
2023-11-24 00:11:47,899 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.099e+01 8.775e+01 9.573e+01 1.290e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-24 00:11:47,944 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5050, loss[loss=0.08366, simple_loss=0.1021, pruned_loss=0.02113, audio_tagging_loss=0.01148, over 15880.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09091, pruned_loss=0.0134, audio_tagging_loss=0.00899, over 3042961.55 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:12:01,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2598820.0, ans=0.0
2023-11-24 00:12:25,870 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.38 vs. limit=22.5
2023-11-24 00:12:32,487 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389850
2023-11-24 00:12:48,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2599020.0, ans=0.125
2023-11-24 00:12:50,836 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5100, loss[loss=0.07014, simple_loss=0.09114, pruned_loss=0.01508, audio_tagging_loss=0.009491, over 13970.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.0908, pruned_loss=0.01348, audio_tagging_loss=0.008993, over 3043912.70 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:12:53,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.whiten.whitening_limit, batch_count=2599086.6666666665, ans=12.0
2023-11-24 00:12:54,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2599086.6666666665, ans=0.125
2023-11-24 00:13:07,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2599153.3333333335, ans=0.125
2023-11-24 00:13:20,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2599220.0, ans=0.09899494936611666
2023-11-24 00:13:26,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2599220.0, ans=0.0
2023-11-24 00:13:34,786 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389900
2023-11-24 00:13:35,422 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.26 vs. limit=15.0
2023-11-24 00:13:46,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2599353.3333333335, ans=0.0
2023-11-24 00:13:48,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.66 vs. limit=8.0
2023-11-24 00:13:51,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2599353.3333333335, ans=0.0
2023-11-24 00:13:54,280 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.310e+01 9.019e+01 9.639e+01 1.666e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 00:13:54,324 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5150, loss[loss=0.05897, simple_loss=0.08617, pruned_loss=0.008886, audio_tagging_loss=0.006994, over 15718.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09126, pruned_loss=0.01368, audio_tagging_loss=0.008871, over 3049777.44 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:14:05,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2599486.6666666665, ans=0.2
2023-11-24 00:14:09,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2599486.6666666665, ans=0.125
2023-11-24 00:14:13,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2599486.6666666665, ans=0.0
2023-11-24 00:14:38,582 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 389950
2023-11-24 00:14:38,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2599620.0, ans=0.1
2023-11-24 00:14:38,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2599620.0, ans=0.09899494936611666
2023-11-24 00:14:43,969 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.74 vs. limit=15.0
2023-11-24 00:14:49,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2599686.6666666665, ans=0.5
2023-11-24 00:14:56,549 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5200, loss[loss=0.07476, simple_loss=0.09675, pruned_loss=0.01716, audio_tagging_loss=0.009231, over 14959.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09119, pruned_loss=0.0136, audio_tagging_loss=0.008878, over 3043780.18 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:15:05,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2599753.3333333335, ans=0.1
2023-11-24 00:15:18,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2599820.0, ans=0.2
2023-11-24 00:15:41,033 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.91 vs. limit=15.0
2023-11-24 00:15:41,663 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390000
2023-11-24 00:16:00,316 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.701e+01 8.431e+01 8.968e+01 9.666e+01 1.343e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-24 00:16:00,363 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5250, loss[loss=0.07243, simple_loss=0.09923, pruned_loss=0.01344, audio_tagging_loss=0.009369, over 14688.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.0922, pruned_loss=0.01378, audio_tagging_loss=0.008785, over 3046705.68 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:16:21,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2600153.3333333335, ans=0.125
2023-11-24 00:16:26,446 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.70 vs. limit=15.0
2023-11-24 00:16:32,339 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=15.0
2023-11-24 00:16:43,593 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390050
2023-11-24 00:16:55,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2600353.3333333335, ans=0.125
2023-11-24 00:16:58,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2600353.3333333335, ans=0.2
2023-11-24 00:16:58,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2600353.3333333335, ans=0.0
2023-11-24 00:17:03,034 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5300, loss[loss=0.08416, simple_loss=0.1144, pruned_loss=0.01969, audio_tagging_loss=0.007279, over 14612.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09255, pruned_loss=0.0137, audio_tagging_loss=0.008696, over 3043546.91 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:17:07,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2600420.0, ans=0.1
2023-11-24 00:17:36,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2600553.3333333335, ans=0.1
2023-11-24 00:17:41,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2600620.0, ans=0.2
2023-11-24 00:17:46,785 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390100
2023-11-24 00:17:50,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2600620.0, ans=0.5
2023-11-24 00:17:56,983 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:18:04,809 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5350, loss[loss=0.07463, simple_loss=0.1084, pruned_loss=0.01287, audio_tagging_loss=0.007542, over 14735.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09211, pruned_loss=0.01368, audio_tagging_loss=0.0088, over 3041865.51 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:18:05,954 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.377e+01 8.739e+01 9.630e+01 1.201e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-24 00:18:10,224 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.32 vs. limit=15.0
2023-11-24 00:18:16,115 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.15 vs. limit=10.0
2023-11-24 00:18:16,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2600820.0, ans=0.09899494936611666
2023-11-24 00:18:34,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2600886.6666666665, ans=0.1
2023-11-24 00:18:40,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2600886.6666666665, ans=0.125
2023-11-24 00:18:48,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390150
2023-11-24 00:18:57,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2601020.0, ans=0.125
2023-11-24 00:19:06,226 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5400, loss[loss=0.05631, simple_loss=0.0752, pruned_loss=0.007872, audio_tagging_loss=0.01083, over 15106.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09173, pruned_loss=0.01355, audio_tagging_loss=0.008911, over 3039163.42 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:19:08,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2601086.6666666665, ans=0.125
2023-11-24 00:19:11,058 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.32 vs. limit=10.0
2023-11-24 00:19:14,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2601086.6666666665, ans=0.125
2023-11-24 00:19:23,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff2.min_abs, batch_count=2601153.3333333335, ans=0.1
2023-11-24 00:19:25,126 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0
2023-11-24 00:19:50,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390200
2023-11-24 00:20:09,849 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5450, loss[loss=0.05797, simple_loss=0.07523, pruned_loss=0.009584, audio_tagging_loss=0.01077, over 15424.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09134, pruned_loss=0.01341, audio_tagging_loss=0.008973, over 3034751.99 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:20:10,928 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.432e+01 8.413e+01 8.965e+01 9.765e+01 1.201e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 00:20:25,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2601486.6666666665, ans=0.0
2023-11-24 00:20:40,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.whiten.whitening_limit, batch_count=2601553.3333333335, ans=15.0
2023-11-24 00:20:52,460 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390250
2023-11-24 00:21:11,526 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5500, loss[loss=0.07975, simple_loss=0.1024, pruned_loss=0.01998, audio_tagging_loss=0.00858, over 15798.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09188, pruned_loss=0.0135, audio_tagging_loss=0.009053, over 3042529.91 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:21:17,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2601753.3333333335, ans=0.125
2023-11-24 00:21:44,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2601886.6666666665, ans=0.125
2023-11-24 00:21:55,016 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390300
2023-11-24 00:22:03,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2602020.0, ans=0.2
2023-11-24 00:22:13,307 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5550, loss[loss=0.06154, simple_loss=0.07629, pruned_loss=0.01221, audio_tagging_loss=0.01118, over 15770.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.0915, pruned_loss=0.01347, audio_tagging_loss=0.009211, over 3042396.44 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:22:14,449 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.193e+01 8.280e+01 9.193e+01 9.915e+01 1.422e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 00:22:31,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2602153.3333333335, ans=0.0
2023-11-24 00:22:52,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2602286.6666666665, ans=0.1
2023-11-24 00:22:57,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390350
2023-11-24 00:23:02,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2602353.3333333335, ans=0.125
2023-11-24 00:23:08,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2602353.3333333335, ans=0.125
2023-11-24 00:23:10,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2602353.3333333335, ans=0.2
2023-11-24 00:23:15,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2602420.0, ans=0.125
2023-11-24 00:23:16,743 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5600, loss[loss=0.06627, simple_loss=0.09386, pruned_loss=0.01294, audio_tagging_loss=0.006404, over 15947.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09141, pruned_loss=0.01336, audio_tagging_loss=0.009237, over 3037316.44 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:23:22,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2602420.0, ans=15.0
2023-11-24 00:23:32,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2602486.6666666665, ans=0.1
2023-11-24 00:23:36,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.73 vs. limit=15.0
2023-11-24 00:23:43,970 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0
2023-11-24 00:23:47,413 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.88 vs. limit=12.0
2023-11-24 00:23:59,831 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390400
2023-11-24 00:24:00,931 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:24:02,981 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.88 vs. limit=22.5
2023-11-24 00:24:18,476 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5650, loss[loss=0.04882, simple_loss=0.06716, pruned_loss=0.006249, audio_tagging_loss=0.008989, over 15221.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09154, pruned_loss=0.0135, audio_tagging_loss=0.009221, over 3043436.38 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:24:19,630 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.276e+01 8.461e+01 8.950e+01 9.725e+01 1.309e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-24 00:25:02,275 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390450
2023-11-24 00:25:04,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2602953.3333333335, ans=0.125
2023-11-24 00:25:08,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2603020.0, ans=0.0
2023-11-24 00:25:15,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2603020.0, ans=0.2
2023-11-24 00:25:17,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2603020.0, ans=0.2
2023-11-24 00:25:20,371 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5700, loss[loss=0.0638, simple_loss=0.08782, pruned_loss=0.01177, audio_tagging_loss=0.00812, over 15688.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.0911, pruned_loss=0.0135, audio_tagging_loss=0.009274, over 3047077.81 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:25:24,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2603086.6666666665, ans=0.0
2023-11-24 00:25:30,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2603086.6666666665, ans=0.125
2023-11-24 00:26:04,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390500
2023-11-24 00:26:23,067 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5750, loss[loss=0.06809, simple_loss=0.09524, pruned_loss=0.01242, audio_tagging_loss=0.008047, over 15059.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09053, pruned_loss=0.01335, audio_tagging_loss=0.009154, over 3046176.40 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:26:24,161 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.481e+01 9.036e+01 9.796e+01 1.280e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 00:26:53,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2603553.3333333335, ans=0.125
2023-11-24 00:26:56,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2603553.3333333335, ans=0.125
2023-11-24 00:26:58,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2603620.0, ans=0.95
2023-11-24 00:27:04,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-24 00:27:06,333 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390550
2023-11-24 00:27:09,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2603620.0, ans=0.2
2023-11-24 00:27:25,131 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5800, loss[loss=0.07753, simple_loss=0.1098, pruned_loss=0.01425, audio_tagging_loss=0.008371, over 14000.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.08958, pruned_loss=0.0131, audio_tagging_loss=0.009064, over 3042329.44 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:27:29,471 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.92 vs. limit=22.5
2023-11-24 00:27:38,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2603820.0, ans=0.0
2023-11-24 00:27:38,821 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.38 vs. limit=15.0
2023-11-24 00:28:08,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390600
2023-11-24 00:28:19,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2604020.0, ans=0.1
2023-11-24 00:28:26,601 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5850, loss[loss=0.06495, simple_loss=0.07686, pruned_loss=0.01536, audio_tagging_loss=0.01116, over 14541.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09011, pruned_loss=0.01316, audio_tagging_loss=0.009038, over 3039713.56 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:28:30,556 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.379e+01 8.883e+01 9.652e+01 2.888e+02, threshold=1.777e+02, percent-clipped=1.0
2023-11-24 00:28:53,267 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-24 00:28:53,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2604220.0, ans=0.2
2023-11-24 00:28:56,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2604220.0, ans=0.125
2023-11-24 00:29:04,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.88 vs. limit=15.0
2023-11-24 00:29:06,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2604286.6666666665, ans=0.1
2023-11-24 00:29:10,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390650
2023-11-24 00:29:29,339 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5900, loss[loss=0.05152, simple_loss=0.06757, pruned_loss=0.008164, audio_tagging_loss=0.009565, over 14484.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09112, pruned_loss=0.01322, audio_tagging_loss=0.008963, over 3042844.34 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:29:47,452 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:29:48,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2604486.6666666665, ans=0.125
2023-11-24 00:29:52,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2604486.6666666665, ans=0.125
2023-11-24 00:30:13,490 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390700
2023-11-24 00:30:14,151 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.24 vs. limit=12.0
2023-11-24 00:30:32,174 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 5950, loss[loss=0.0696, simple_loss=0.09005, pruned_loss=0.01512, audio_tagging_loss=0.009456, over 15498.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09152, pruned_loss=0.01323, audio_tagging_loss=0.008944, over 3045519.93 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:30:35,908 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.451e+01 9.116e+01 9.981e+01 1.115e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 00:31:15,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390750
2023-11-24 00:31:17,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2604953.3333333335, ans=0.05
2023-11-24 00:31:31,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2605020.0, ans=0.125
2023-11-24 00:31:33,654 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6000, loss[loss=0.06428, simple_loss=0.08887, pruned_loss=0.01091, audio_tagging_loss=0.008933, over 14302.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.0915, pruned_loss=0.01324, audio_tagging_loss=0.008922, over 3048287.21 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:31:33,655 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 00:32:09,909 INFO [train_asr.py:1253] (2/4) Epoch 33, validation: loss=0.05769, simple_loss=0.05098, pruned_loss=0.005124, audio_tagging_loss=0.02707, over 4681554.00 frames. 
2023-11-24 00:32:09,910 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 00:32:28,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2605153.3333333335, ans=0.2
2023-11-24 00:32:44,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2605286.6666666665, ans=0.2
2023-11-24 00:32:52,237 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390800
2023-11-24 00:32:55,875 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:32:56,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2605286.6666666665, ans=0.0
2023-11-24 00:33:01,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2605353.3333333335, ans=0.95
2023-11-24 00:33:07,431 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2605353.3333333335, ans=0.0
2023-11-24 00:33:08,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2605353.3333333335, ans=0.1
2023-11-24 00:33:11,952 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6050, loss[loss=0.05533, simple_loss=0.07385, pruned_loss=0.009606, audio_tagging_loss=0.008795, over 14886.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09143, pruned_loss=0.01333, audio_tagging_loss=0.008857, over 3047337.72 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:33:12,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2605420.0, ans=0.0
2023-11-24 00:33:15,470 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.390e+01 8.432e+01 8.885e+01 9.962e+01 1.302e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 00:33:26,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2605486.6666666665, ans=0.125
2023-11-24 00:33:50,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2605620.0, ans=0.04949747468305833
2023-11-24 00:33:54,283 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.88 vs. limit=15.0
2023-11-24 00:33:54,991 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390850
2023-11-24 00:33:58,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2605620.0, ans=0.125
2023-11-24 00:34:12,719 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6100, loss[loss=0.05413, simple_loss=0.06674, pruned_loss=0.009362, audio_tagging_loss=0.0114, over 14970.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09157, pruned_loss=0.01338, audio_tagging_loss=0.008858, over 3052305.51 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:34:19,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2605753.3333333335, ans=0.125
2023-11-24 00:34:39,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2605886.6666666665, ans=0.2
2023-11-24 00:34:39,478 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.28 vs. limit=12.0
2023-11-24 00:34:42,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.82 vs. limit=15.0
2023-11-24 00:34:56,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390900
2023-11-24 00:35:14,632 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6150, loss[loss=0.09729, simple_loss=0.1221, pruned_loss=0.02771, audio_tagging_loss=0.008508, over 15073.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.0915, pruned_loss=0.01339, audio_tagging_loss=0.008869, over 3050153.22 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:35:19,340 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.208e+01 8.886e+01 9.597e+01 1.503e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 00:35:24,908 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.45 vs. limit=6.0
2023-11-24 00:35:31,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2606153.3333333335, ans=0.125
2023-11-24 00:35:57,635 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 390950
2023-11-24 00:36:17,596 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6200, loss[loss=0.05873, simple_loss=0.0765, pruned_loss=0.01006, audio_tagging_loss=0.01042, over 15435.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09075, pruned_loss=0.01331, audio_tagging_loss=0.00896, over 3050875.12 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:36:17,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2606420.0, ans=0.125
2023-11-24 00:36:27,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2606420.0, ans=0.1
2023-11-24 00:36:33,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2606486.6666666665, ans=0.05
2023-11-24 00:36:34,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2606486.6666666665, ans=10.0
2023-11-24 00:36:40,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2606553.3333333335, ans=0.125
2023-11-24 00:37:02,071 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391000
2023-11-24 00:37:07,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2606686.6666666665, ans=0.125
2023-11-24 00:37:07,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2606686.6666666665, ans=0.0
2023-11-24 00:37:20,039 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6250, loss[loss=0.06517, simple_loss=0.09209, pruned_loss=0.01067, audio_tagging_loss=0.00845, over 15312.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09025, pruned_loss=0.01334, audio_tagging_loss=0.009047, over 3051318.28 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:37:21,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2606753.3333333335, ans=0.0
2023-11-24 00:37:23,526 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.315e+01 8.981e+01 9.556e+01 1.931e+02, threshold=1.796e+02, percent-clipped=1.0
2023-11-24 00:37:35,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2606820.0, ans=0.1
2023-11-24 00:37:46,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2606886.6666666665, ans=0.1
2023-11-24 00:38:04,287 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391050
2023-11-24 00:38:16,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.59 vs. limit=10.0
2023-11-24 00:38:17,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2607020.0, ans=0.1
2023-11-24 00:38:22,587 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6300, loss[loss=0.05829, simple_loss=0.07327, pruned_loss=0.008906, audio_tagging_loss=0.01275, over 15428.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09052, pruned_loss=0.01322, audio_tagging_loss=0.009154, over 3057146.71 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:39:00,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.89 vs. limit=10.0
2023-11-24 00:39:05,769 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391100
2023-11-24 00:39:05,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2607286.6666666665, ans=0.125
2023-11-24 00:39:06,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2607286.6666666665, ans=0.04949747468305833
2023-11-24 00:39:08,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2607286.6666666665, ans=0.125
2023-11-24 00:39:17,509 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.05 vs. limit=22.5
2023-11-24 00:39:25,694 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6350, loss[loss=0.05548, simple_loss=0.06506, pruned_loss=0.009564, audio_tagging_loss=0.01338, over 13635.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09138, pruned_loss=0.01321, audio_tagging_loss=0.009184, over 3050745.45 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:39:29,229 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.017e+01 8.323e+01 9.041e+01 9.620e+01 1.215e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 00:39:34,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2607420.0, ans=0.125
2023-11-24 00:39:38,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2607486.6666666665, ans=0.125
2023-11-24 00:39:40,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2607486.6666666665, ans=0.2
2023-11-24 00:39:53,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2607553.3333333335, ans=0.125
2023-11-24 00:40:09,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391150
2023-11-24 00:40:20,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2607686.6666666665, ans=0.125
2023-11-24 00:40:21,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2607686.6666666665, ans=0.07
2023-11-24 00:40:27,280 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6400, loss[loss=0.09143, simple_loss=0.1253, pruned_loss=0.0207, audio_tagging_loss=0.008111, over 15548.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.0909, pruned_loss=0.01305, audio_tagging_loss=0.009286, over 3050855.02 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 00:40:32,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2607753.3333333335, ans=0.0
2023-11-24 00:40:44,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2607820.0, ans=0.0
2023-11-24 00:40:53,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2607886.6666666665, ans=0.125
2023-11-24 00:40:58,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.30 vs. limit=15.0
2023-11-24 00:41:11,011 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391200
2023-11-24 00:41:20,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2608020.0, ans=0.0
2023-11-24 00:41:20,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.21 vs. limit=22.5
2023-11-24 00:41:29,323 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6450, loss[loss=0.06739, simple_loss=0.09115, pruned_loss=0.01327, audio_tagging_loss=0.008544, over 15018.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09117, pruned_loss=0.01319, audio_tagging_loss=0.009302, over 3049899.81 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:41:34,718 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.349e+01 8.985e+01 9.691e+01 1.713e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 00:42:14,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391250
2023-11-24 00:42:32,636 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:42:33,542 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6500, loss[loss=0.05412, simple_loss=0.07917, pruned_loss=0.005342, audio_tagging_loss=0.009194, over 15687.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09096, pruned_loss=0.01317, audio_tagging_loss=0.009286, over 3050440.40 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:42:34,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2608420.0, ans=0.125
2023-11-24 00:42:50,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2608486.6666666665, ans=0.05
2023-11-24 00:42:56,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2608553.3333333335, ans=0.1
2023-11-24 00:43:04,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2608553.3333333335, ans=0.0
2023-11-24 00:43:06,849 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.39 vs. limit=15.0
2023-11-24 00:43:09,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2608620.0, ans=0.1
2023-11-24 00:43:17,559 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391300
2023-11-24 00:43:32,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2608686.6666666665, ans=0.125
2023-11-24 00:43:35,736 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6550, loss[loss=0.05742, simple_loss=0.07554, pruned_loss=0.01009, audio_tagging_loss=0.009561, over 13936.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09108, pruned_loss=0.01334, audio_tagging_loss=0.009147, over 3053890.82 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:43:40,545 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.079e+01 8.397e+01 9.072e+01 9.736e+01 1.207e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 00:43:42,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2608753.3333333335, ans=0.125
2023-11-24 00:43:47,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.94 vs. limit=15.0
2023-11-24 00:44:02,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2608886.6666666665, ans=0.125
2023-11-24 00:44:12,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2608953.3333333335, ans=0.05
2023-11-24 00:44:19,800 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391350
2023-11-24 00:44:37,629 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6600, loss[loss=0.06296, simple_loss=0.08232, pruned_loss=0.01028, audio_tagging_loss=0.01151, over 14472.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09106, pruned_loss=0.01327, audio_tagging_loss=0.009038, over 3051836.29 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:44:44,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2609086.6666666665, ans=0.125
2023-11-24 00:45:21,717 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391400
2023-11-24 00:45:41,440 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6650, loss[loss=0.04996, simple_loss=0.06267, pruned_loss=0.008921, audio_tagging_loss=0.009706, over 15807.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09157, pruned_loss=0.01338, audio_tagging_loss=0.009068, over 3052765.29 frames. ], batch size: 64, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:45:46,135 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.357e+01 8.351e+01 8.932e+01 9.624e+01 1.119e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 00:46:06,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2609553.3333333335, ans=0.125
2023-11-24 00:46:06,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2609553.3333333335, ans=0.2
2023-11-24 00:46:08,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2609553.3333333335, ans=0.95
2023-11-24 00:46:14,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2609553.3333333335, ans=0.125
2023-11-24 00:46:25,033 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391450
2023-11-24 00:46:25,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2609620.0, ans=0.125
2023-11-24 00:46:28,003 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.98 vs. limit=22.5
2023-11-24 00:46:28,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2609620.0, ans=0.04949747468305833
2023-11-24 00:46:31,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2609686.6666666665, ans=0.0
2023-11-24 00:46:32,652 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.90 vs. limit=6.0
2023-11-24 00:46:42,720 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6700, loss[loss=0.07161, simple_loss=0.09819, pruned_loss=0.01501, audio_tagging_loss=0.007508, over 15391.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09129, pruned_loss=0.01341, audio_tagging_loss=0.009093, over 3047988.79 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:46:59,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2609820.0, ans=0.125
2023-11-24 00:47:13,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2609886.6666666665, ans=0.0
2023-11-24 00:47:26,655 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391500
2023-11-24 00:47:28,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2609953.3333333335, ans=0.0
2023-11-24 00:47:37,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2610020.0, ans=0.1
2023-11-24 00:47:42,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2610020.0, ans=0.1
2023-11-24 00:47:45,025 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6750, loss[loss=0.07317, simple_loss=0.1032, pruned_loss=0.01448, audio_tagging_loss=0.007102, over 14435.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09144, pruned_loss=0.01354, audio_tagging_loss=0.009035, over 3038695.54 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:47:49,704 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.433e+01 8.920e+01 9.710e+01 1.340e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 00:48:01,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2610153.3333333335, ans=0.04949747468305833
2023-11-24 00:48:05,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2610153.3333333335, ans=0.125
2023-11-24 00:48:22,944 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.38 vs. limit=12.0
2023-11-24 00:48:28,843 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391550
2023-11-24 00:48:30,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2610286.6666666665, ans=0.0
2023-11-24 00:48:32,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2610286.6666666665, ans=0.015
2023-11-24 00:48:48,395 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6800, loss[loss=0.0649, simple_loss=0.08845, pruned_loss=0.01186, audio_tagging_loss=0.008812, over 16329.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09075, pruned_loss=0.01348, audio_tagging_loss=0.008989, over 3033908.63 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 00:48:58,189 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.29 vs. limit=22.5
2023-11-24 00:49:00,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2610486.6666666665, ans=0.0
2023-11-24 00:49:02,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2610486.6666666665, ans=0.125
2023-11-24 00:49:06,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2610486.6666666665, ans=0.1
2023-11-24 00:49:09,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2610486.6666666665, ans=0.125
2023-11-24 00:49:10,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2610486.6666666665, ans=0.125
2023-11-24 00:49:11,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2610553.3333333335, ans=0.04949747468305833
2023-11-24 00:49:21,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2610553.3333333335, ans=0.0
2023-11-24 00:49:30,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2610620.0, ans=0.0
2023-11-24 00:49:31,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391600
2023-11-24 00:49:39,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2610686.6666666665, ans=0.0
2023-11-24 00:49:50,166 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6850, loss[loss=0.06197, simple_loss=0.08429, pruned_loss=0.01069, audio_tagging_loss=0.009127, over 14501.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09008, pruned_loss=0.01331, audio_tagging_loss=0.0091, over 3033117.41 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:49:56,144 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.434e+01 9.033e+01 9.858e+01 1.368e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 00:49:56,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2610753.3333333335, ans=0.0
2023-11-24 00:50:10,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2610820.0, ans=0.2
2023-11-24 00:50:12,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2610820.0, ans=0.2
2023-11-24 00:50:34,006 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391650
2023-11-24 00:50:35,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2610953.3333333335, ans=0.125
2023-11-24 00:50:37,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2610953.3333333335, ans=0.0
2023-11-24 00:50:42,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2611020.0, ans=0.125
2023-11-24 00:50:48,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2611020.0, ans=0.125
2023-11-24 00:50:52,617 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6900, loss[loss=0.06434, simple_loss=0.08682, pruned_loss=0.01258, audio_tagging_loss=0.008353, over 14718.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09119, pruned_loss=0.01336, audio_tagging_loss=0.009022, over 3043388.83 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:50:56,366 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:51:16,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2611153.3333333335, ans=0.125
2023-11-24 00:51:24,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2611220.0, ans=0.2
2023-11-24 00:51:30,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2611286.6666666665, ans=0.2
2023-11-24 00:51:36,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391700
2023-11-24 00:51:40,220 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.62 vs. limit=22.5
2023-11-24 00:51:41,364 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:51:55,498 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 6950, loss[loss=0.05298, simple_loss=0.07234, pruned_loss=0.008773, audio_tagging_loss=0.008034, over 14381.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09239, pruned_loss=0.01358, audio_tagging_loss=0.008923, over 3043663.49 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:51:57,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2611420.0, ans=0.125
2023-11-24 00:51:59,369 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.52 vs. limit=15.0
2023-11-24 00:52:03,204 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.470e+01 9.036e+01 9.958e+01 1.197e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 00:52:04,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2611420.0, ans=0.125
2023-11-24 00:52:38,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391750
2023-11-24 00:52:47,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2611686.6666666665, ans=0.125
2023-11-24 00:52:49,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2611686.6666666665, ans=0.0
2023-11-24 00:52:51,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2611686.6666666665, ans=0.125
2023-11-24 00:52:51,791 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:52:57,555 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7000, loss[loss=0.07404, simple_loss=0.1029, pruned_loss=0.01675, audio_tagging_loss=0.005829, over 14398.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09187, pruned_loss=0.01356, audio_tagging_loss=0.008936, over 3039771.87 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:53:03,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2611753.3333333335, ans=0.125
2023-11-24 00:53:03,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2611753.3333333335, ans=0.09899494936611666
2023-11-24 00:53:12,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2611820.0, ans=0.125
2023-11-24 00:53:41,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391800
2023-11-24 00:53:59,736 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7050, loss[loss=0.06462, simple_loss=0.09757, pruned_loss=0.008094, audio_tagging_loss=0.007743, over 15233.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09027, pruned_loss=0.01345, audio_tagging_loss=0.00901, over 3034859.68 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:54:01,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.99 vs. limit=22.5
2023-11-24 00:54:06,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2612086.6666666665, ans=0.125
2023-11-24 00:54:07,536 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.402e+01 9.074e+01 1.006e+02 1.351e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 00:54:09,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2612086.6666666665, ans=0.0
2023-11-24 00:54:29,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.34 vs. limit=15.0
2023-11-24 00:54:43,870 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391850
2023-11-24 00:54:47,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2612286.6666666665, ans=0.125
2023-11-24 00:55:02,808 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7100, loss[loss=0.04693, simple_loss=0.06461, pruned_loss=0.008028, audio_tagging_loss=0.006599, over 13763.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09049, pruned_loss=0.01341, audio_tagging_loss=0.009041, over 3035797.04 frames. ], batch size: 52, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:55:04,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2612420.0, ans=0.0
2023-11-24 00:55:13,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2612420.0, ans=0.125
2023-11-24 00:55:24,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2612486.6666666665, ans=0.0
2023-11-24 00:55:25,844 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.28 vs. limit=15.0
2023-11-24 00:55:26,808 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.96 vs. limit=10.0
2023-11-24 00:55:37,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2612553.3333333335, ans=0.125
2023-11-24 00:55:39,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2612620.0, ans=0.2
2023-11-24 00:55:45,716 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391900
2023-11-24 00:55:54,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2612686.6666666665, ans=0.125
2023-11-24 00:55:56,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2612686.6666666665, ans=0.125
2023-11-24 00:55:57,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2612686.6666666665, ans=0.125
2023-11-24 00:56:05,206 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7150, loss[loss=0.06378, simple_loss=0.07746, pruned_loss=0.0127, audio_tagging_loss=0.01235, over 14460.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09056, pruned_loss=0.01333, audio_tagging_loss=0.009101, over 3043251.43 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:56:12,274 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.300e+01 8.358e+01 9.047e+01 9.923e+01 1.303e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 00:56:22,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2612820.0, ans=0.125
2023-11-24 00:56:27,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2612886.6666666665, ans=0.125
2023-11-24 00:56:37,737 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.20 vs. limit=12.0
2023-11-24 00:56:49,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 391950
2023-11-24 00:56:51,028 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.81 vs. limit=15.0
2023-11-24 00:56:51,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2612953.3333333335, ans=0.125
2023-11-24 00:56:52,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2612953.3333333335, ans=0.0
2023-11-24 00:57:06,673 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7200, loss[loss=0.07758, simple_loss=0.1044, pruned_loss=0.01787, audio_tagging_loss=0.007518, over 14756.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09042, pruned_loss=0.01324, audio_tagging_loss=0.009174, over 3048230.51 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:57:35,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2613220.0, ans=0.0
2023-11-24 00:57:41,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-24 00:57:50,686 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392000
2023-11-24 00:58:12,145 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7250, loss[loss=0.06888, simple_loss=0.08558, pruned_loss=0.01486, audio_tagging_loss=0.01124, over 14608.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09135, pruned_loss=0.01341, audio_tagging_loss=0.009244, over 3049402.93 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:58:20,980 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.420e+01 8.844e+01 9.320e+01 1.006e+02 1.273e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 00:58:24,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2613486.6666666665, ans=0.125
2023-11-24 00:58:35,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2613486.6666666665, ans=0.0
2023-11-24 00:58:55,247 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392050
2023-11-24 00:59:06,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2613686.6666666665, ans=0.0
2023-11-24 00:59:07,042 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.06 vs. limit=22.5
2023-11-24 00:59:15,267 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7300, loss[loss=0.0787, simple_loss=0.1134, pruned_loss=0.01535, audio_tagging_loss=0.006643, over 15799.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09213, pruned_loss=0.01357, audio_tagging_loss=0.009177, over 3047643.69 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:59:18,259 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.42 vs. limit=15.0
2023-11-24 00:59:21,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2613753.3333333335, ans=0.0
2023-11-24 00:59:22,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2613753.3333333335, ans=0.0
2023-11-24 00:59:22,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2613753.3333333335, ans=0.125
2023-11-24 00:59:23,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2613753.3333333335, ans=0.0
2023-11-24 00:59:26,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2613820.0, ans=0.125
2023-11-24 00:59:28,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2613820.0, ans=0.2
2023-11-24 00:59:52,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2613953.3333333335, ans=0.07
2023-11-24 00:59:53,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2613953.3333333335, ans=0.2
2023-11-24 00:59:55,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2613953.3333333335, ans=0.0
2023-11-24 00:59:58,579 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392100
2023-11-24 01:00:03,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2614020.0, ans=0.1
2023-11-24 01:00:10,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2614020.0, ans=0.2
2023-11-24 01:00:16,223 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7350, loss[loss=0.07689, simple_loss=0.1027, pruned_loss=0.01835, audio_tagging_loss=0.007194, over 15428.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09188, pruned_loss=0.01358, audio_tagging_loss=0.009028, over 3050277.82 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:00:20,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2614086.6666666665, ans=0.0
2023-11-24 01:00:23,327 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 8.518e+01 9.030e+01 9.706e+01 1.263e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 01:00:48,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2614220.0, ans=0.0
2023-11-24 01:01:00,277 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392150
2023-11-24 01:01:09,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2614353.3333333335, ans=0.125
2023-11-24 01:01:18,051 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7400, loss[loss=0.0764, simple_loss=0.1038, pruned_loss=0.01592, audio_tagging_loss=0.0086, over 15912.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.092, pruned_loss=0.01343, audio_tagging_loss=0.008942, over 3046211.84 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:01:26,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2614420.0, ans=0.1
2023-11-24 01:01:28,600 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=14.98 vs. limit=15.0
2023-11-24 01:01:45,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2614553.3333333335, ans=0.1
2023-11-24 01:01:46,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2614553.3333333335, ans=0.125
2023-11-24 01:01:51,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2614553.3333333335, ans=0.0
2023-11-24 01:02:02,042 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392200
2023-11-24 01:02:09,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2614686.6666666665, ans=0.0
2023-11-24 01:02:11,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2614686.6666666665, ans=0.125
2023-11-24 01:02:17,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2614686.6666666665, ans=0.125
2023-11-24 01:02:21,662 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7450, loss[loss=0.07051, simple_loss=0.0959, pruned_loss=0.01516, audio_tagging_loss=0.007398, over 14965.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09213, pruned_loss=0.01357, audio_tagging_loss=0.008927, over 3033645.90 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:02:28,726 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.459e+01 9.049e+01 9.753e+01 1.412e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 01:03:05,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392250
2023-11-24 01:03:18,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2615020.0, ans=0.0
2023-11-24 01:03:23,816 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7500, loss[loss=0.08064, simple_loss=0.1038, pruned_loss=0.02053, audio_tagging_loss=0.008208, over 14650.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09165, pruned_loss=0.01362, audio_tagging_loss=0.008865, over 3031381.84 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:03:36,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.26 vs. limit=12.0
2023-11-24 01:03:53,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2615220.0, ans=0.2
2023-11-24 01:04:07,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392300
2023-11-24 01:04:25,646 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7550, loss[loss=0.0738, simple_loss=0.09654, pruned_loss=0.01645, audio_tagging_loss=0.009084, over 13358.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09186, pruned_loss=0.01349, audio_tagging_loss=0.008843, over 3039755.47 frames. ], batch size: 50, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:04:32,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1.whitening_limit, batch_count=2615420.0, ans=10.0
2023-11-24 01:04:33,439 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.602e+01 9.110e+01 9.644e+01 1.333e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 01:04:33,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2615420.0, ans=0.125
2023-11-24 01:04:33,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2615420.0, ans=0.125
2023-11-24 01:04:43,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2615486.6666666665, ans=0.0
2023-11-24 01:04:44,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2615486.6666666665, ans=0.125
2023-11-24 01:04:44,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2615486.6666666665, ans=0.1
2023-11-24 01:04:48,686 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=12.0
2023-11-24 01:05:04,517 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.77 vs. limit=15.0
2023-11-24 01:05:08,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2615620.0, ans=0.125
2023-11-24 01:05:09,870 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392350
2023-11-24 01:05:22,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2615686.6666666665, ans=0.0
2023-11-24 01:05:25,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2615686.6666666665, ans=0.125
2023-11-24 01:05:29,042 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7600, loss[loss=0.07204, simple_loss=0.1012, pruned_loss=0.01323, audio_tagging_loss=0.008204, over 14635.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09158, pruned_loss=0.01349, audio_tagging_loss=0.00893, over 3044132.19 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 01:05:31,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2615753.3333333335, ans=0.1
2023-11-24 01:05:44,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2615820.0, ans=0.125
2023-11-24 01:05:48,532 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.22 vs. limit=15.0
2023-11-24 01:06:12,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392400
2023-11-24 01:06:12,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2615953.3333333335, ans=0.04949747468305833
2023-11-24 01:06:19,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2616020.0, ans=0.07
2023-11-24 01:06:31,602 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7650, loss[loss=0.05346, simple_loss=0.07162, pruned_loss=0.00826, audio_tagging_loss=0.00939, over 15706.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09127, pruned_loss=0.01328, audio_tagging_loss=0.008885, over 3046574.64 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:06:39,747 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.599e+01 8.392e+01 8.875e+01 9.417e+01 1.557e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 01:06:50,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2616153.3333333335, ans=0.125
2023-11-24 01:06:51,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.16 vs. limit=22.5
2023-11-24 01:06:57,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2616220.0, ans=10.0
2023-11-24 01:07:04,826 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.41 vs. limit=15.0
2023-11-24 01:07:10,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2616286.6666666665, ans=0.125
2023-11-24 01:07:15,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392450
2023-11-24 01:07:16,001 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.85 vs. limit=10.0
2023-11-24 01:07:23,984 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:07:28,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2616353.3333333335, ans=0.0
2023-11-24 01:07:33,368 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7700, loss[loss=0.06974, simple_loss=0.09947, pruned_loss=0.01313, audio_tagging_loss=0.00687, over 15270.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09102, pruned_loss=0.01323, audio_tagging_loss=0.008919, over 3046497.20 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:07:34,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2616420.0, ans=0.125
2023-11-24 01:07:52,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.32 vs. limit=15.0
2023-11-24 01:07:52,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2616486.6666666665, ans=0.125
2023-11-24 01:07:59,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2616553.3333333335, ans=0.125
2023-11-24 01:08:09,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2616553.3333333335, ans=0.1
2023-11-24 01:08:09,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2616553.3333333335, ans=0.0
2023-11-24 01:08:17,288 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392500
2023-11-24 01:08:17,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2616620.0, ans=0.1
2023-11-24 01:08:25,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2616686.6666666665, ans=0.1
2023-11-24 01:08:36,710 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7750, loss[loss=0.09549, simple_loss=0.1328, pruned_loss=0.01971, audio_tagging_loss=0.00937, over 15252.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09133, pruned_loss=0.01321, audio_tagging_loss=0.008991, over 3038558.82 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:08:38,557 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.90 vs. limit=10.0
2023-11-24 01:08:45,017 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.197e+01 8.416e+01 8.954e+01 9.884e+01 1.252e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 01:08:53,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2616820.0, ans=0.0
2023-11-24 01:08:54,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2616820.0, ans=0.2
2023-11-24 01:09:04,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2616886.6666666665, ans=0.1
2023-11-24 01:09:19,620 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392550
2023-11-24 01:09:20,265 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.99 vs. limit=15.0
2023-11-24 01:09:28,257 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-24 01:09:37,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2617086.6666666665, ans=0.09899494936611666
2023-11-24 01:09:38,495 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7800, loss[loss=0.06801, simple_loss=0.08568, pruned_loss=0.01632, audio_tagging_loss=0.008858, over 15080.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09262, pruned_loss=0.01353, audio_tagging_loss=0.008989, over 3043569.65 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:09:52,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2617153.3333333335, ans=0.125
2023-11-24 01:09:52,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2617153.3333333335, ans=0.125
2023-11-24 01:10:22,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392600
2023-11-24 01:10:38,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2617353.3333333335, ans=0.125
2023-11-24 01:10:41,592 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7850, loss[loss=0.04827, simple_loss=0.06441, pruned_loss=0.007923, audio_tagging_loss=0.008146, over 15611.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09261, pruned_loss=0.01359, audio_tagging_loss=0.008978, over 3040112.88 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:10:45,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2617420.0, ans=0.2
2023-11-24 01:10:50,001 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.631e+01 9.206e+01 9.904e+01 1.275e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 01:11:25,091 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392650
2023-11-24 01:11:30,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=2617686.6666666665, ans=0.1
2023-11-24 01:11:43,257 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7900, loss[loss=0.04192, simple_loss=0.05103, pruned_loss=0.006051, audio_tagging_loss=0.01035, over 14555.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09247, pruned_loss=0.01368, audio_tagging_loss=0.009011, over 3044271.60 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:12:26,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392700
2023-11-24 01:12:28,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2617953.3333333335, ans=0.0
2023-11-24 01:12:41,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2618020.0, ans=0.95
2023-11-24 01:12:46,551 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 7950, loss[loss=0.06025, simple_loss=0.07811, pruned_loss=0.009941, audio_tagging_loss=0.01125, over 14763.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09268, pruned_loss=0.01376, audio_tagging_loss=0.009087, over 3053627.27 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:12:46,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2618086.6666666665, ans=0.125
2023-11-24 01:12:54,800 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.531e+01 8.932e+01 9.571e+01 1.197e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 01:13:03,041 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.11 vs. limit=6.0
2023-11-24 01:13:03,665 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:13:29,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2618286.6666666665, ans=0.125
2023-11-24 01:13:30,103 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392750
2023-11-24 01:13:30,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2618286.6666666665, ans=0.1
2023-11-24 01:13:41,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2618353.3333333335, ans=0.1
2023-11-24 01:13:48,858 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8000, loss[loss=0.07669, simple_loss=0.1049, pruned_loss=0.01729, audio_tagging_loss=0.006962, over 15384.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09214, pruned_loss=0.01373, audio_tagging_loss=0.009192, over 3053762.58 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:13:59,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2618486.6666666665, ans=0.0
2023-11-24 01:14:05,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2618486.6666666665, ans=0.0
2023-11-24 01:14:05,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2618486.6666666665, ans=0.1
2023-11-24 01:14:07,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2618486.6666666665, ans=0.125
2023-11-24 01:14:25,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.02 vs. limit=15.0
2023-11-24 01:14:31,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0
2023-11-24 01:14:32,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392800
2023-11-24 01:14:33,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2618620.0, ans=0.125
2023-11-24 01:14:50,743 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8050, loss[loss=0.07528, simple_loss=0.1017, pruned_loss=0.01521, audio_tagging_loss=0.009205, over 16073.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09326, pruned_loss=0.01386, audio_tagging_loss=0.009102, over 3050581.56 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:14:51,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2618753.3333333335, ans=0.025
2023-11-24 01:14:53,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2618753.3333333335, ans=0.125
2023-11-24 01:15:01,891 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.420e+01 8.885e+01 9.707e+01 1.213e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 01:15:03,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2618820.0, ans=0.1
2023-11-24 01:15:08,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.67 vs. limit=15.0
2023-11-24 01:15:12,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2618820.0, ans=0.125
2023-11-24 01:15:20,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2618886.6666666665, ans=0.125
2023-11-24 01:15:26,857 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.27 vs. limit=15.0
2023-11-24 01:15:34,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392850
2023-11-24 01:15:45,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2619020.0, ans=0.125
2023-11-24 01:15:54,259 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8100, loss[loss=0.06715, simple_loss=0.07945, pruned_loss=0.01614, audio_tagging_loss=0.01128, over 14301.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.0924, pruned_loss=0.01371, audio_tagging_loss=0.008997, over 3047214.09 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:16:01,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2619086.6666666665, ans=0.125
2023-11-24 01:16:05,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2619153.3333333335, ans=0.125
2023-11-24 01:16:38,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392900
2023-11-24 01:16:43,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2619353.3333333335, ans=0.1
2023-11-24 01:16:50,796 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.37 vs. limit=15.0
2023-11-24 01:16:54,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2619353.3333333335, ans=0.125
2023-11-24 01:16:56,167 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8150, loss[loss=0.04853, simple_loss=0.06053, pruned_loss=0.008375, audio_tagging_loss=0.009891, over 14428.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09269, pruned_loss=0.01367, audio_tagging_loss=0.008906, over 3047662.44 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:16:59,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2619420.0, ans=0.125
2023-11-24 01:17:06,605 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.428e+01 9.309e+01 1.006e+02 1.505e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 01:17:40,909 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 392950
2023-11-24 01:17:43,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2619620.0, ans=0.125
2023-11-24 01:17:44,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2619620.0, ans=0.0
2023-11-24 01:17:51,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2619686.6666666665, ans=0.2
2023-11-24 01:17:59,141 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8200, loss[loss=0.07245, simple_loss=0.09604, pruned_loss=0.01694, audio_tagging_loss=0.007497, over 15884.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09143, pruned_loss=0.01336, audio_tagging_loss=0.008927, over 3043741.69 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:18:01,539 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:18:05,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2619753.3333333335, ans=0.1
2023-11-24 01:18:13,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2619820.0, ans=0.125
2023-11-24 01:18:15,919 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.81 vs. limit=15.0
2023-11-24 01:18:20,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2619820.0, ans=0.1
2023-11-24 01:18:32,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2619886.6666666665, ans=0.0
2023-11-24 01:18:43,383 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393000
2023-11-24 01:19:03,423 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8250, loss[loss=0.07573, simple_loss=0.09503, pruned_loss=0.01937, audio_tagging_loss=0.008849, over 15351.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09152, pruned_loss=0.01329, audio_tagging_loss=0.008963, over 3042008.74 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:19:06,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2620086.6666666665, ans=0.2
2023-11-24 01:19:13,064 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.535e+01 8.227e+01 9.050e+01 1.002e+02 1.257e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 01:19:15,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2620153.3333333335, ans=0.125
2023-11-24 01:19:21,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2620153.3333333335, ans=0.2
2023-11-24 01:19:27,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2620220.0, ans=0.125
2023-11-24 01:19:47,613 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393050
2023-11-24 01:19:50,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2620286.6666666665, ans=0.2
2023-11-24 01:20:05,498 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8300, loss[loss=0.06818, simple_loss=0.09205, pruned_loss=0.01435, audio_tagging_loss=0.007801, over 15207.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09226, pruned_loss=0.01352, audio_tagging_loss=0.008797, over 3040145.53 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:20:09,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2620420.0, ans=0.125
2023-11-24 01:20:13,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2620420.0, ans=0.0
2023-11-24 01:20:16,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2620486.6666666665, ans=0.0
2023-11-24 01:20:49,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393100
2023-11-24 01:21:07,459 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8350, loss[loss=0.06537, simple_loss=0.08625, pruned_loss=0.0148, audio_tagging_loss=0.007441, over 14791.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09221, pruned_loss=0.01357, audio_tagging_loss=0.008818, over 3039361.01 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:21:09,297 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.30 vs. limit=15.0
2023-11-24 01:21:20,001 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.568e+01 9.324e+01 1.020e+02 1.487e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 01:21:25,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.66 vs. limit=22.5
2023-11-24 01:21:27,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2620820.0, ans=0.1
2023-11-24 01:21:46,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2620953.3333333335, ans=0.125
2023-11-24 01:21:51,028 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393150
2023-11-24 01:21:51,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2620953.3333333335, ans=0.125
2023-11-24 01:21:53,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2620953.3333333335, ans=0.125
2023-11-24 01:22:11,137 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8400, loss[loss=0.07615, simple_loss=0.1133, pruned_loss=0.0119, audio_tagging_loss=0.007624, over 14647.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09225, pruned_loss=0.01346, audio_tagging_loss=0.008806, over 3040914.32 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:22:31,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2621153.3333333335, ans=0.125
2023-11-24 01:22:40,530 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.90 vs. limit=22.5
2023-11-24 01:22:41,638 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-24 01:22:42,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2621220.0, ans=0.0
2023-11-24 01:22:47,368 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0
2023-11-24 01:22:53,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2621286.6666666665, ans=0.125
2023-11-24 01:22:54,556 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393200
2023-11-24 01:23:12,773 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8450, loss[loss=0.06188, simple_loss=0.087, pruned_loss=0.009948, audio_tagging_loss=0.008433, over 15535.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09318, pruned_loss=0.0136, audio_tagging_loss=0.008867, over 3044604.00 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:23:23,339 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.428e+01 8.989e+01 9.532e+01 1.176e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 01:23:37,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2621553.3333333335, ans=0.04949747468305833
2023-11-24 01:23:38,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.79 vs. limit=15.0
2023-11-24 01:23:44,753 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.81 vs. limit=15.0
2023-11-24 01:23:45,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2621553.3333333335, ans=0.125
2023-11-24 01:23:49,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2621620.0, ans=0.125
2023-11-24 01:23:51,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2621620.0, ans=0.125
2023-11-24 01:23:56,083 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393250
2023-11-24 01:24:04,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2621686.6666666665, ans=0.0
2023-11-24 01:24:08,720 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.08 vs. limit=15.0
2023-11-24 01:24:13,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten.whitening_limit, batch_count=2621753.3333333335, ans=22.5
2023-11-24 01:24:13,902 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8500, loss[loss=0.08221, simple_loss=0.1092, pruned_loss=0.01865, audio_tagging_loss=0.008972, over 16112.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09188, pruned_loss=0.01339, audio_tagging_loss=0.008937, over 3042681.95 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:24:18,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2621753.3333333335, ans=0.0
2023-11-24 01:24:26,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2621820.0, ans=10.0
2023-11-24 01:24:34,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2621820.0, ans=0.0
2023-11-24 01:24:45,768 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-24 01:24:57,404 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393300
2023-11-24 01:25:01,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2621953.3333333335, ans=0.0
2023-11-24 01:25:05,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2622020.0, ans=0.125
2023-11-24 01:25:13,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2622020.0, ans=0.125
2023-11-24 01:25:15,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.25 vs. limit=15.0
2023-11-24 01:25:17,339 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8550, loss[loss=0.06451, simple_loss=0.08064, pruned_loss=0.01399, audio_tagging_loss=0.0102, over 15589.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09243, pruned_loss=0.01345, audio_tagging_loss=0.009007, over 3042825.35 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:25:29,171 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.811e+01 9.309e+01 9.853e+01 1.247e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 01:25:37,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2622153.3333333335, ans=0.04949747468305833
2023-11-24 01:25:40,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.63 vs. limit=10.0
2023-11-24 01:25:49,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2622220.0, ans=0.125
2023-11-24 01:25:53,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2622286.6666666665, ans=0.1
2023-11-24 01:25:57,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2622286.6666666665, ans=0.1
2023-11-24 01:25:59,962 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393350
2023-11-24 01:26:16,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2622353.3333333335, ans=0.1
2023-11-24 01:26:17,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2622420.0, ans=0.0
2023-11-24 01:26:18,351 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8600, loss[loss=0.06996, simple_loss=0.09834, pruned_loss=0.01258, audio_tagging_loss=0.008203, over 14681.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09397, pruned_loss=0.01383, audio_tagging_loss=0.008897, over 3050958.54 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:26:42,422 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:26:44,018 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2622553.3333333335, ans=0.125
2023-11-24 01:26:47,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2622553.3333333335, ans=0.125
2023-11-24 01:27:01,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393400
2023-11-24 01:27:09,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2622686.6666666665, ans=0.0
2023-11-24 01:27:16,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2622686.6666666665, ans=0.0
2023-11-24 01:27:19,493 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8650, loss[loss=0.04968, simple_loss=0.06411, pruned_loss=0.01022, audio_tagging_loss=0.007414, over 14895.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09355, pruned_loss=0.0138, audio_tagging_loss=0.008966, over 3051981.78 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:27:32,823 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.465e+01 9.032e+01 9.917e+01 1.291e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 01:27:47,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2622886.6666666665, ans=0.125
2023-11-24 01:27:50,144 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.60 vs. limit=15.0
2023-11-24 01:27:56,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2622953.3333333335, ans=0.125
2023-11-24 01:27:59,194 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:28:02,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2622953.3333333335, ans=0.2
2023-11-24 01:28:03,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393450
2023-11-24 01:28:09,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2623020.0, ans=0.125
2023-11-24 01:28:13,799 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.48 vs. limit=12.0
2023-11-24 01:28:14,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2623020.0, ans=0.0
2023-11-24 01:28:15,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2623020.0, ans=0.125
2023-11-24 01:28:22,591 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8700, loss[loss=0.05102, simple_loss=0.0668, pruned_loss=0.008009, audio_tagging_loss=0.009609, over 16007.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09297, pruned_loss=0.01378, audio_tagging_loss=0.009051, over 3052805.82 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:28:25,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2623086.6666666665, ans=0.125
2023-11-24 01:29:00,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2623286.6666666665, ans=0.125
2023-11-24 01:29:05,294 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393500
2023-11-24 01:29:09,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2623286.6666666665, ans=0.2
2023-11-24 01:29:14,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2623353.3333333335, ans=0.1
2023-11-24 01:29:24,775 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8750, loss[loss=0.07384, simple_loss=0.09045, pruned_loss=0.01696, audio_tagging_loss=0.01166, over 14741.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09257, pruned_loss=0.0137, audio_tagging_loss=0.00911, over 3055354.40 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:29:36,658 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.231e+01 8.417e+01 9.143e+01 1.010e+02 1.352e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 01:29:37,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2623486.6666666665, ans=0.125
2023-11-24 01:29:45,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2623486.6666666665, ans=0.125
2023-11-24 01:30:07,473 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393550
2023-11-24 01:30:25,614 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8800, loss[loss=0.07763, simple_loss=0.1122, pruned_loss=0.01352, audio_tagging_loss=0.008007, over 15496.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.0929, pruned_loss=0.01372, audio_tagging_loss=0.009138, over 3043743.28 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:30:50,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2623886.6666666665, ans=0.125
2023-11-24 01:30:54,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2623886.6666666665, ans=0.125
2023-11-24 01:30:55,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2623886.6666666665, ans=0.125
2023-11-24 01:30:58,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2623886.6666666665, ans=0.2
2023-11-24 01:31:03,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2623953.3333333335, ans=0.125
2023-11-24 01:31:08,771 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393600
2023-11-24 01:31:27,896 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8850, loss[loss=0.07019, simple_loss=0.09312, pruned_loss=0.01203, audio_tagging_loss=0.0116, over 17057.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09294, pruned_loss=0.01375, audio_tagging_loss=0.009171, over 3042460.76 frames. ], batch size: 64, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:31:39,634 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.345e+01 8.328e+01 9.022e+01 9.725e+01 1.238e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 01:31:40,846 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:31:43,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2624153.3333333335, ans=0.125
2023-11-24 01:31:55,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2624220.0, ans=0.2
2023-11-24 01:32:01,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2624220.0, ans=0.0
2023-11-24 01:32:09,894 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393650
2023-11-24 01:32:25,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2624353.3333333335, ans=0.125
2023-11-24 01:32:28,677 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8900, loss[loss=0.08019, simple_loss=0.1187, pruned_loss=0.01463, audio_tagging_loss=0.006212, over 16112.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09407, pruned_loss=0.01392, audio_tagging_loss=0.009012, over 3050596.26 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:33:12,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393700
2023-11-24 01:33:19,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2624686.6666666665, ans=0.125
2023-11-24 01:33:22,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2624686.6666666665, ans=0.125
2023-11-24 01:33:23,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2624686.6666666665, ans=0.07
2023-11-24 01:33:28,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2624686.6666666665, ans=0.125
2023-11-24 01:33:30,359 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 8950, loss[loss=0.07269, simple_loss=0.09493, pruned_loss=0.01686, audio_tagging_loss=0.008361, over 14642.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09336, pruned_loss=0.01373, audio_tagging_loss=0.00892, over 3053129.23 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:33:42,260 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.467e+01 9.115e+01 9.992e+01 1.363e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 01:34:02,175 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.45 vs. limit=15.0
2023-11-24 01:34:13,842 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393750
2023-11-24 01:34:15,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.52 vs. limit=15.0
2023-11-24 01:34:21,470 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.33 vs. limit=15.0
2023-11-24 01:34:27,587 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.99 vs. limit=22.5
2023-11-24 01:34:32,139 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9000, loss[loss=0.09386, simple_loss=0.1266, pruned_loss=0.02471, audio_tagging_loss=0.005842, over 14507.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09396, pruned_loss=0.01382, audio_tagging_loss=0.008796, over 3061059.32 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:34:32,139 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 01:35:10,899 INFO [train_asr.py:1253] (2/4) Epoch 33, validation: loss=0.05892, simple_loss=0.05094, pruned_loss=0.005119, audio_tagging_loss=0.02833, over 4681554.00 frames. 
2023-11-24 01:35:10,899 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 01:35:17,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.27 vs. limit=22.5
2023-11-24 01:35:20,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.42 vs. limit=15.0
2023-11-24 01:35:23,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2625153.3333333335, ans=0.0
2023-11-24 01:35:29,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2625153.3333333335, ans=0.5
2023-11-24 01:35:31,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2625153.3333333335, ans=0.0
2023-11-24 01:35:49,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2625286.6666666665, ans=0.0
2023-11-24 01:35:53,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393800
2023-11-24 01:36:03,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2625353.3333333335, ans=0.125
2023-11-24 01:36:12,537 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9050, loss[loss=0.06837, simple_loss=0.09486, pruned_loss=0.01138, audio_tagging_loss=0.009561, over 15906.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09307, pruned_loss=0.01362, audio_tagging_loss=0.008786, over 3064904.28 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:36:12,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2625420.0, ans=0.035
2023-11-24 01:36:25,590 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.819e+01 9.377e+01 1.005e+02 1.265e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 01:36:35,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2625486.6666666665, ans=0.2
2023-11-24 01:36:40,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2625553.3333333335, ans=0.0
2023-11-24 01:36:55,937 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393850
2023-11-24 01:37:06,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2625686.6666666665, ans=0.0
2023-11-24 01:37:14,627 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9100, loss[loss=0.06973, simple_loss=0.09211, pruned_loss=0.01399, audio_tagging_loss=0.009679, over 14588.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.0931, pruned_loss=0.01339, audio_tagging_loss=0.008772, over 3056115.45 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:37:20,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2625753.3333333335, ans=0.5
2023-11-24 01:37:42,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2625886.6666666665, ans=0.125
2023-11-24 01:37:43,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2625886.6666666665, ans=0.0
2023-11-24 01:37:57,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393900
2023-11-24 01:37:59,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2625953.3333333335, ans=0.125
2023-11-24 01:38:01,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0
2023-11-24 01:38:15,236 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9150, loss[loss=0.06775, simple_loss=0.08758, pruned_loss=0.01329, audio_tagging_loss=0.01067, over 15212.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09358, pruned_loss=0.01352, audio_tagging_loss=0.008717, over 3047513.56 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:38:18,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2626086.6666666665, ans=0.125
2023-11-24 01:38:27,504 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 7.991e+01 8.782e+01 9.725e+01 1.593e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-24 01:38:50,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2626220.0, ans=0.125
2023-11-24 01:38:54,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.87 vs. limit=15.0
2023-11-24 01:38:58,269 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 393950
2023-11-24 01:39:03,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.65 vs. limit=10.0
2023-11-24 01:39:16,551 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9200, loss[loss=0.07796, simple_loss=0.1003, pruned_loss=0.01831, audio_tagging_loss=0.009507, over 14443.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09271, pruned_loss=0.01353, audio_tagging_loss=0.008835, over 3054356.89 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 01:39:16,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2626420.0, ans=0.125
2023-11-24 01:39:18,522 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=22.5
2023-11-24 01:39:30,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2626486.6666666665, ans=0.0
2023-11-24 01:39:33,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2626486.6666666665, ans=0.125
2023-11-24 01:39:40,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2626553.3333333335, ans=0.1
2023-11-24 01:39:46,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2626553.3333333335, ans=0.0
2023-11-24 01:39:50,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2626553.3333333335, ans=0.125
2023-11-24 01:39:58,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394000
2023-11-24 01:40:12,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2626686.6666666665, ans=0.1
2023-11-24 01:40:18,490 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9250, loss[loss=0.07554, simple_loss=0.1081, pruned_loss=0.01349, audio_tagging_loss=0.007979, over 16679.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09249, pruned_loss=0.01348, audio_tagging_loss=0.008898, over 3053394.17 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:40:22,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2626753.3333333335, ans=0.07
2023-11-24 01:40:31,440 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.331e+01 9.082e+01 9.914e+01 1.295e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 01:40:38,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2626820.0, ans=0.125
2023-11-24 01:40:41,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.96 vs. limit=15.0
2023-11-24 01:40:49,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2626886.6666666665, ans=0.0
2023-11-24 01:40:58,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2626953.3333333335, ans=0.2
2023-11-24 01:41:00,542 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394050
2023-11-24 01:41:10,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2627020.0, ans=0.125
2023-11-24 01:41:16,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2627020.0, ans=0.0
2023-11-24 01:41:16,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2627020.0, ans=0.125
2023-11-24 01:41:19,512 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9300, loss[loss=0.05039, simple_loss=0.06741, pruned_loss=0.008512, audio_tagging_loss=0.008169, over 14742.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09291, pruned_loss=0.01374, audio_tagging_loss=0.008863, over 3058688.82 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:41:37,907 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:42:03,193 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394100
2023-11-24 01:42:14,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2627353.3333333335, ans=0.125
2023-11-24 01:42:17,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2627353.3333333335, ans=0.2
2023-11-24 01:42:21,253 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9350, loss[loss=0.05096, simple_loss=0.06298, pruned_loss=0.007655, audio_tagging_loss=0.01181, over 14733.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09274, pruned_loss=0.01377, audio_tagging_loss=0.008875, over 3062345.62 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:42:32,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2627486.6666666665, ans=0.1
2023-11-24 01:42:36,052 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.369e+01 8.431e+01 9.003e+01 9.569e+01 1.110e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 01:42:57,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2627620.0, ans=0.125
2023-11-24 01:42:57,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.95 vs. limit=15.0
2023-11-24 01:42:59,018 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.74 vs. limit=22.5
2023-11-24 01:43:04,306 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394150
2023-11-24 01:43:15,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2627686.6666666665, ans=0.125
2023-11-24 01:43:22,663 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9400, loss[loss=0.08095, simple_loss=0.1072, pruned_loss=0.01804, audio_tagging_loss=0.009294, over 14423.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09258, pruned_loss=0.01369, audio_tagging_loss=0.009009, over 3058865.16 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:43:22,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2627753.3333333335, ans=0.125
2023-11-24 01:43:31,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2627753.3333333335, ans=0.125
2023-11-24 01:43:53,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2627886.6666666665, ans=0.125
2023-11-24 01:44:03,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2627953.3333333335, ans=0.125
2023-11-24 01:44:03,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2627953.3333333335, ans=0.0
2023-11-24 01:44:06,273 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394200
2023-11-24 01:44:19,202 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.38 vs. limit=15.0
2023-11-24 01:44:25,391 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9450, loss[loss=0.05412, simple_loss=0.06844, pruned_loss=0.007341, audio_tagging_loss=0.01256, over 13546.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.0929, pruned_loss=0.01366, audio_tagging_loss=0.009101, over 3055229.49 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:44:25,415 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:44:36,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2628153.3333333335, ans=0.125
2023-11-24 01:44:39,847 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.278e+01 8.363e+01 8.998e+01 9.818e+01 1.304e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 01:44:57,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2628220.0, ans=0.2
2023-11-24 01:45:02,873 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.85 vs. limit=15.0
2023-11-24 01:45:09,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394250
2023-11-24 01:45:12,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2628286.6666666665, ans=0.0
2023-11-24 01:45:26,604 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9500, loss[loss=0.06833, simple_loss=0.09063, pruned_loss=0.01286, audio_tagging_loss=0.01015, over 15665.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09288, pruned_loss=0.01386, audio_tagging_loss=0.009184, over 3060775.69 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:45:26,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2628420.0, ans=0.125
2023-11-24 01:45:28,677 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.20 vs. limit=15.0
2023-11-24 01:45:34,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2628420.0, ans=0.1
2023-11-24 01:45:35,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2628420.0, ans=0.1
2023-11-24 01:45:45,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2628486.6666666665, ans=0.125
2023-11-24 01:46:09,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394300
2023-11-24 01:46:27,670 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9550, loss[loss=0.06148, simple_loss=0.07272, pruned_loss=0.0137, audio_tagging_loss=0.01142, over 14757.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09335, pruned_loss=0.01397, audio_tagging_loss=0.0092, over 3055690.74 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:46:40,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2628820.0, ans=0.125
2023-11-24 01:46:42,914 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.519e+01 9.047e+01 9.704e+01 1.211e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 01:46:48,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2628820.0, ans=0.125
2023-11-24 01:46:58,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2628886.6666666665, ans=0.125
2023-11-24 01:47:03,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2628953.3333333335, ans=0.0
2023-11-24 01:47:10,260 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394350
2023-11-24 01:47:15,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2629020.0, ans=0.125
2023-11-24 01:47:18,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2629020.0, ans=0.1
2023-11-24 01:47:29,314 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9600, loss[loss=0.05614, simple_loss=0.07381, pruned_loss=0.009824, audio_tagging_loss=0.009408, over 14943.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.0921, pruned_loss=0.01373, audio_tagging_loss=0.009308, over 3053853.73 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:47:55,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2629220.0, ans=0.125
2023-11-24 01:48:13,548 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394400
2023-11-24 01:48:18,757 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:48:24,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2629353.3333333335, ans=0.2
2023-11-24 01:48:31,237 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9650, loss[loss=0.07987, simple_loss=0.1082, pruned_loss=0.01747, audio_tagging_loss=0.008273, over 15324.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09156, pruned_loss=0.01363, audio_tagging_loss=0.009256, over 3053744.21 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:48:41,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2629420.0, ans=0.125
2023-11-24 01:48:43,271 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:48:45,251 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.217e+01 8.903e+01 9.505e+01 1.138e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-24 01:48:47,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2629486.6666666665, ans=0.125
2023-11-24 01:49:12,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2629620.0, ans=0.125
2023-11-24 01:49:14,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394450
2023-11-24 01:49:18,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2629620.0, ans=0.2
2023-11-24 01:49:18,755 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.25 vs. limit=22.5
2023-11-24 01:49:32,178 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9700, loss[loss=0.08165, simple_loss=0.1077, pruned_loss=0.01924, audio_tagging_loss=0.008556, over 15506.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09288, pruned_loss=0.01387, audio_tagging_loss=0.009055, over 3050986.01 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:49:33,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2629753.3333333335, ans=0.125
2023-11-24 01:49:41,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2629753.3333333335, ans=0.125
2023-11-24 01:49:41,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2629753.3333333335, ans=10.0
2023-11-24 01:49:51,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2629820.0, ans=0.0
2023-11-24 01:50:00,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2629886.6666666665, ans=0.0
2023-11-24 01:50:15,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394500
2023-11-24 01:50:27,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2630020.0, ans=0.125
2023-11-24 01:50:34,851 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9750, loss[loss=0.05855, simple_loss=0.06723, pruned_loss=0.009138, audio_tagging_loss=0.01579, over 14684.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09188, pruned_loss=0.0136, audio_tagging_loss=0.009013, over 3052193.38 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:50:41,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2630086.6666666665, ans=0.2
2023-11-24 01:50:49,143 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.192e+01 8.620e+01 9.194e+01 9.972e+01 1.344e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 01:50:56,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2630153.3333333335, ans=0.125
2023-11-24 01:50:57,033 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.20 vs. limit=15.0
2023-11-24 01:51:17,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394550
2023-11-24 01:51:36,065 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9800, loss[loss=0.08624, simple_loss=0.1243, pruned_loss=0.02011, audio_tagging_loss=0.003994, over 15202.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09218, pruned_loss=0.01372, audio_tagging_loss=0.008983, over 3049424.52 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:51:51,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2630486.6666666665, ans=10.0
2023-11-24 01:52:19,667 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394600
2023-11-24 01:52:32,707 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.19 vs. limit=5.0
2023-11-24 01:52:33,029 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:52:37,796 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9850, loss[loss=0.1026, simple_loss=0.1456, pruned_loss=0.02332, audio_tagging_loss=0.006535, over 16756.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09188, pruned_loss=0.01363, audio_tagging_loss=0.009039, over 3051678.24 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:52:53,142 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.642e+01 9.395e+01 1.002e+02 1.404e+02, threshold=1.879e+02, percent-clipped=0.0
2023-11-24 01:53:03,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2630886.6666666665, ans=0.0
2023-11-24 01:53:14,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2630953.3333333335, ans=0.125
2023-11-24 01:53:18,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2630953.3333333335, ans=0.125
2023-11-24 01:53:19,654 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0
2023-11-24 01:53:21,227 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394650
2023-11-24 01:53:40,122 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9900, loss[loss=0.05335, simple_loss=0.07213, pruned_loss=0.008642, audio_tagging_loss=0.008644, over 14141.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09104, pruned_loss=0.01357, audio_tagging_loss=0.009108, over 3050501.24 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:54:08,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2631220.0, ans=0.0
2023-11-24 01:54:08,253 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:54:17,248 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.63 vs. limit=15.0
2023-11-24 01:54:20,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2631286.6666666665, ans=0.1
2023-11-24 01:54:20,691 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0
2023-11-24 01:54:23,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394700
2023-11-24 01:54:25,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2631286.6666666665, ans=0.125
2023-11-24 01:54:32,624 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.75 vs. limit=15.0
2023-11-24 01:54:42,050 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 9950, loss[loss=0.07142, simple_loss=0.09055, pruned_loss=0.01452, audio_tagging_loss=0.01163, over 15141.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09114, pruned_loss=0.01363, audio_tagging_loss=0.009006, over 3050665.17 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:54:52,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2631486.6666666665, ans=0.0
2023-11-24 01:54:56,127 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.637e+01 9.131e+01 9.854e+01 1.265e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 01:55:02,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2631486.6666666665, ans=0.125
2023-11-24 01:55:09,879 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.42 vs. limit=15.0
2023-11-24 01:55:18,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2631620.0, ans=0.125
2023-11-24 01:55:24,946 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394750
2023-11-24 01:55:35,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2631686.6666666665, ans=0.2
2023-11-24 01:55:42,555 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10000, loss[loss=0.06403, simple_loss=0.08984, pruned_loss=0.01184, audio_tagging_loss=0.00727, over 14606.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09111, pruned_loss=0.01341, audio_tagging_loss=0.008892, over 3054201.46 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 01:55:50,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2631753.3333333335, ans=0.1
2023-11-24 01:56:04,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2631820.0, ans=0.125
2023-11-24 01:56:04,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2631820.0, ans=0.125
2023-11-24 01:56:25,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394800
2023-11-24 01:56:25,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2631953.3333333335, ans=0.125
2023-11-24 01:56:36,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2632020.0, ans=0.125
2023-11-24 01:56:45,124 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10050, loss[loss=0.06942, simple_loss=0.1031, pruned_loss=0.01197, audio_tagging_loss=0.005896, over 15443.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09139, pruned_loss=0.0133, audio_tagging_loss=0.0089, over 3054930.47 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 01:56:45,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2632086.6666666665, ans=0.1
2023-11-24 01:56:59,415 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.081e+01 8.389e+01 9.133e+01 9.609e+01 1.226e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 01:57:02,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2632153.3333333335, ans=0.125
2023-11-24 01:57:27,569 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394850
2023-11-24 01:57:37,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2632353.3333333335, ans=0.0
2023-11-24 01:57:46,263 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10100, loss[loss=0.0746, simple_loss=0.1072, pruned_loss=0.01288, audio_tagging_loss=0.008104, over 15644.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.0909, pruned_loss=0.01324, audio_tagging_loss=0.009043, over 3050222.68 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 01:57:52,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2632420.0, ans=0.035
2023-11-24 01:58:29,536 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394900
2023-11-24 01:58:36,496 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:58:47,634 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10150, loss[loss=0.04312, simple_loss=0.06142, pruned_loss=0.00497, audio_tagging_loss=0.007443, over 16742.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09134, pruned_loss=0.01343, audio_tagging_loss=0.009051, over 3058732.98 frames. ], batch size: 66, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 01:59:02,460 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.176e+01 8.517e+01 9.184e+01 1.003e+02 1.404e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 01:59:17,265 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:59:30,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 394950
2023-11-24 01:59:40,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2633020.0, ans=0.125
2023-11-24 01:59:49,447 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10200, loss[loss=0.06186, simple_loss=0.07865, pruned_loss=0.009951, audio_tagging_loss=0.01259, over 14126.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09086, pruned_loss=0.0134, audio_tagging_loss=0.009159, over 3055137.85 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 01:59:55,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2633086.6666666665, ans=0.125
2023-11-24 02:00:11,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2633153.3333333335, ans=0.0
2023-11-24 02:00:14,013 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:00:14,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2633220.0, ans=0.125
2023-11-24 02:00:23,422 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:00:32,828 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395000
2023-11-24 02:00:52,229 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10250, loss[loss=0.06809, simple_loss=0.08674, pruned_loss=0.0129, audio_tagging_loss=0.01183, over 15352.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09087, pruned_loss=0.01334, audio_tagging_loss=0.009167, over 3052768.34 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:00:53,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2633420.0, ans=0.125
2023-11-24 02:01:07,145 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.621e+01 8.620e+01 9.148e+01 9.659e+01 1.254e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 02:01:30,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2633620.0, ans=0.0
2023-11-24 02:01:36,554 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395050
2023-11-24 02:01:48,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2633686.6666666665, ans=0.125
2023-11-24 02:01:48,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2633686.6666666665, ans=0.125
2023-11-24 02:01:48,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2633686.6666666665, ans=0.125
2023-11-24 02:01:53,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2633753.3333333335, ans=0.0
2023-11-24 02:01:54,959 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10300, loss[loss=0.06379, simple_loss=0.08346, pruned_loss=0.01121, audio_tagging_loss=0.01085, over 16078.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09222, pruned_loss=0.01359, audio_tagging_loss=0.0091, over 3054480.96 frames. ], batch size: 61, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:02:07,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2633820.0, ans=0.125
2023-11-24 02:02:07,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2633820.0, ans=0.1
2023-11-24 02:02:11,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2633820.0, ans=0.125
2023-11-24 02:02:14,709 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:02:14,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2633820.0, ans=0.125
2023-11-24 02:02:17,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2633820.0, ans=0.125
2023-11-24 02:02:23,313 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.53 vs. limit=15.0
2023-11-24 02:02:34,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2633953.3333333335, ans=0.1
2023-11-24 02:02:38,283 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395100
2023-11-24 02:02:56,671 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10350, loss[loss=0.06266, simple_loss=0.08202, pruned_loss=0.01003, audio_tagging_loss=0.01162, over 14837.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09144, pruned_loss=0.0134, audio_tagging_loss=0.009244, over 3053151.32 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:03:04,449 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.72 vs. limit=22.5
2023-11-24 02:03:12,161 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.449e+01 8.937e+01 9.719e+01 1.290e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-24 02:03:39,982 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395150
2023-11-24 02:03:53,402 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.26 vs. limit=22.5
2023-11-24 02:03:58,865 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10400, loss[loss=0.05443, simple_loss=0.07816, pruned_loss=0.008036, audio_tagging_loss=0.007313, over 15521.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09078, pruned_loss=0.01324, audio_tagging_loss=0.009435, over 3047559.32 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:04:06,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2634420.0, ans=0.125
2023-11-24 02:04:08,846 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.39 vs. limit=6.0
2023-11-24 02:04:29,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2634553.3333333335, ans=0.0
2023-11-24 02:04:36,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2634620.0, ans=0.125
2023-11-24 02:04:41,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395200
2023-11-24 02:04:54,298 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:04:59,974 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10450, loss[loss=0.07407, simple_loss=0.09673, pruned_loss=0.01616, audio_tagging_loss=0.009542, over 14730.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.08956, pruned_loss=0.01311, audio_tagging_loss=0.009506, over 3040398.42 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:05:14,722 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.359e+01 8.851e+01 9.617e+01 1.233e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-24 02:05:37,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2634953.3333333335, ans=0.0
2023-11-24 02:05:37,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.76 vs. limit=12.0
2023-11-24 02:05:42,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2634953.3333333335, ans=0.125
2023-11-24 02:05:43,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395250
2023-11-24 02:05:44,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2634953.3333333335, ans=0.125
2023-11-24 02:05:51,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2635020.0, ans=0.2
2023-11-24 02:05:55,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2635020.0, ans=0.125
2023-11-24 02:06:01,537 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10500, loss[loss=0.06601, simple_loss=0.09342, pruned_loss=0.01396, audio_tagging_loss=0.005336, over 15958.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.08967, pruned_loss=0.01312, audio_tagging_loss=0.009291, over 3041661.70 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:06:15,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2635153.3333333335, ans=0.125
2023-11-24 02:06:21,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.86 vs. limit=10.0
2023-11-24 02:06:24,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2635153.3333333335, ans=0.0
2023-11-24 02:06:34,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2635220.0, ans=0.05
2023-11-24 02:06:38,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2635286.6666666665, ans=0.0
2023-11-24 02:06:44,658 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395300
2023-11-24 02:06:51,669 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.86 vs. limit=15.0
2023-11-24 02:07:04,422 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10550, loss[loss=0.05733, simple_loss=0.07416, pruned_loss=0.01276, audio_tagging_loss=0.007484, over 14934.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.0904, pruned_loss=0.01329, audio_tagging_loss=0.009087, over 3041616.98 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:07:05,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2635420.0, ans=0.2
2023-11-24 02:07:19,125 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.732e+01 9.283e+01 1.036e+02 1.632e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 02:07:32,712 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.09 vs. limit=10.0
2023-11-24 02:07:47,051 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:07:48,228 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395350
2023-11-24 02:08:05,708 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10600, loss[loss=0.06312, simple_loss=0.08733, pruned_loss=0.009511, audio_tagging_loss=0.009948, over 15825.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09059, pruned_loss=0.01339, audio_tagging_loss=0.008974, over 3049197.39 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:08:18,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2635820.0, ans=0.125
2023-11-24 02:08:25,468 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:08:48,929 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395400
2023-11-24 02:09:06,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2636086.6666666665, ans=0.125
2023-11-24 02:09:07,227 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10650, loss[loss=0.05994, simple_loss=0.08413, pruned_loss=0.008895, audio_tagging_loss=0.008981, over 15636.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.0905, pruned_loss=0.01337, audio_tagging_loss=0.009017, over 3042870.59 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:09:23,873 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.426e+01 9.214e+01 9.689e+01 1.168e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 02:09:33,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2636220.0, ans=0.1
2023-11-24 02:09:51,000 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395450
2023-11-24 02:10:06,794 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.19 vs. limit=15.0
2023-11-24 02:10:10,657 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10700, loss[loss=0.07157, simple_loss=0.09733, pruned_loss=0.01269, audio_tagging_loss=0.01021, over 16019.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.0894, pruned_loss=0.01305, audio_tagging_loss=0.009003, over 3042044.37 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:10:15,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2636420.0, ans=0.0
2023-11-24 02:10:20,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2636420.0, ans=0.0
2023-11-24 02:10:32,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2636486.6666666665, ans=0.035
2023-11-24 02:10:35,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2636553.3333333335, ans=0.0
2023-11-24 02:10:49,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2636620.0, ans=0.2
2023-11-24 02:10:53,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395500
2023-11-24 02:11:06,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2636686.6666666665, ans=0.0
2023-11-24 02:11:06,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2636686.6666666665, ans=0.0
2023-11-24 02:11:11,246 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10750, loss[loss=0.06796, simple_loss=0.09257, pruned_loss=0.01422, audio_tagging_loss=0.007452, over 16663.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.0899, pruned_loss=0.01323, audio_tagging_loss=0.008956, over 3045909.94 frames. ], batch size: 63, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:11:11,806 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0
2023-11-24 02:11:26,613 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.296e+01 8.958e+01 9.992e+01 1.267e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 02:11:27,321 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.26 vs. limit=22.5
2023-11-24 02:11:30,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2636820.0, ans=0.025
2023-11-24 02:11:34,883 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.45 vs. limit=22.5
2023-11-24 02:11:37,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2636886.6666666665, ans=0.0
2023-11-24 02:11:55,185 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395550
2023-11-24 02:12:12,796 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10800, loss[loss=0.07659, simple_loss=0.1022, pruned_loss=0.0167, audio_tagging_loss=0.008803, over 16136.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.08983, pruned_loss=0.0133, audio_tagging_loss=0.008874, over 3050225.54 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:12:25,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2637153.3333333335, ans=0.125
2023-11-24 02:12:56,731 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395600
2023-11-24 02:13:16,932 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10850, loss[loss=0.05768, simple_loss=0.0757, pruned_loss=0.01005, audio_tagging_loss=0.00978, over 14256.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08952, pruned_loss=0.01317, audio_tagging_loss=0.008977, over 3046656.20 frames. ], batch size: 53, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:13:33,527 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.225e+01 8.902e+01 9.397e+01 1.304e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-24 02:13:41,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2637553.3333333335, ans=0.1
2023-11-24 02:14:00,252 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395650
2023-11-24 02:14:10,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2637686.6666666665, ans=0.125
2023-11-24 02:14:15,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2637686.6666666665, ans=0.125
2023-11-24 02:14:16,152 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:14:18,577 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10900, loss[loss=0.07206, simple_loss=0.1008, pruned_loss=0.01244, audio_tagging_loss=0.009193, over 15666.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09035, pruned_loss=0.01328, audio_tagging_loss=0.008987, over 3054505.65 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:14:26,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2637753.3333333335, ans=0.07
2023-11-24 02:14:37,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2637820.0, ans=0.125
2023-11-24 02:14:59,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2637953.3333333335, ans=0.125
2023-11-24 02:15:01,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395700
2023-11-24 02:15:19,294 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 10950, loss[loss=0.08342, simple_loss=0.1155, pruned_loss=0.01823, audio_tagging_loss=0.007468, over 15319.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09163, pruned_loss=0.01342, audio_tagging_loss=0.008992, over 3051728.02 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:15:36,935 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.249e+01 9.167e+01 9.838e+01 1.256e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 02:15:37,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2638153.3333333335, ans=0.125
2023-11-24 02:15:42,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2638153.3333333335, ans=0.1
2023-11-24 02:16:02,769 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395750
2023-11-24 02:16:02,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2638286.6666666665, ans=0.035
2023-11-24 02:16:02,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2638286.6666666665, ans=0.1
2023-11-24 02:16:05,279 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:16:08,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2638353.3333333335, ans=0.125
2023-11-24 02:16:11,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2638353.3333333335, ans=0.125
2023-11-24 02:16:12,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2638353.3333333335, ans=0.2
2023-11-24 02:16:19,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2638353.3333333335, ans=0.1
2023-11-24 02:16:22,042 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11000, loss[loss=0.07293, simple_loss=0.09324, pruned_loss=0.01778, audio_tagging_loss=0.008537, over 15219.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09118, pruned_loss=0.01332, audio_tagging_loss=0.008991, over 3048812.65 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:16:33,482 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:16:34,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2638486.6666666665, ans=0.125
2023-11-24 02:16:35,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.47 vs. limit=12.0
2023-11-24 02:16:44,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2638486.6666666665, ans=0.125
2023-11-24 02:16:54,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2638553.3333333335, ans=0.125
2023-11-24 02:17:00,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2638620.0, ans=0.125
2023-11-24 02:17:04,755 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395800
2023-11-24 02:17:24,173 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11050, loss[loss=0.05604, simple_loss=0.08037, pruned_loss=0.007029, audio_tagging_loss=0.008823, over 15397.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09086, pruned_loss=0.01321, audio_tagging_loss=0.009037, over 3046039.93 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:17:26,018 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.06 vs. limit=22.5
2023-11-24 02:17:35,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2638820.0, ans=0.0
2023-11-24 02:17:40,567 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.311e+01 8.882e+01 9.606e+01 1.257e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-24 02:17:40,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2638820.0, ans=0.2
2023-11-24 02:17:42,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2638820.0, ans=0.2
2023-11-24 02:17:44,816 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.20 vs. limit=15.0
2023-11-24 02:17:53,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2638886.6666666665, ans=0.0
2023-11-24 02:17:59,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.24 vs. limit=6.0
2023-11-24 02:18:00,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2638953.3333333335, ans=0.95
2023-11-24 02:18:02,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2638953.3333333335, ans=0.125
2023-11-24 02:18:07,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395850
2023-11-24 02:18:07,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2638953.3333333335, ans=0.05
2023-11-24 02:18:25,475 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11100, loss[loss=0.05613, simple_loss=0.07368, pruned_loss=0.01012, audio_tagging_loss=0.009177, over 15572.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09089, pruned_loss=0.01338, audio_tagging_loss=0.009125, over 3053498.45 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:18:41,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2639153.3333333335, ans=0.0
2023-11-24 02:18:49,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2639220.0, ans=0.125
2023-11-24 02:18:57,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2639220.0, ans=0.125
2023-11-24 02:18:57,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2639220.0, ans=0.1
2023-11-24 02:19:08,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395900
2023-11-24 02:19:20,861 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:19:22,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2639353.3333333335, ans=0.0
2023-11-24 02:19:27,078 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11150, loss[loss=0.07864, simple_loss=0.1109, pruned_loss=0.01507, audio_tagging_loss=0.008103, over 15289.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.0903, pruned_loss=0.01328, audio_tagging_loss=0.009321, over 3055676.19 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:19:27,910 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:19:31,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2639420.0, ans=0.125
2023-11-24 02:19:36,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2639420.0, ans=0.0
2023-11-24 02:19:43,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2639486.6666666665, ans=0.0
2023-11-24 02:19:45,342 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.458e+01 9.119e+01 9.750e+01 1.333e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 02:20:10,018 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 395950
2023-11-24 02:20:10,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2639620.0, ans=0.0
2023-11-24 02:20:29,213 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11200, loss[loss=0.06881, simple_loss=0.08588, pruned_loss=0.01318, audio_tagging_loss=0.01268, over 13385.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.0901, pruned_loss=0.01335, audio_tagging_loss=0.009366, over 3056828.67 frames. ], batch size: 50, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:20:31,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2639753.3333333335, ans=0.2
2023-11-24 02:20:39,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2639753.3333333335, ans=0.125
2023-11-24 02:21:12,770 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396000
2023-11-24 02:21:14,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2639953.3333333335, ans=0.0
2023-11-24 02:21:19,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2639953.3333333335, ans=0.125
2023-11-24 02:21:26,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2640020.0, ans=0.0
2023-11-24 02:21:34,665 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11250, loss[loss=0.0678, simple_loss=0.09378, pruned_loss=0.01325, audio_tagging_loss=0.007664, over 16433.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09001, pruned_loss=0.01336, audio_tagging_loss=0.009281, over 3058853.53 frames. ], batch size: 62, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:21:34,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2640086.6666666665, ans=0.125
2023-11-24 02:21:53,431 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.556e+01 8.444e+01 8.848e+01 9.459e+01 1.220e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-24 02:22:01,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2640220.0, ans=0.0
2023-11-24 02:22:07,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2640220.0, ans=0.2
2023-11-24 02:22:13,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2640286.6666666665, ans=0.125
2023-11-24 02:22:18,377 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396050
2023-11-24 02:22:30,506 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-24 02:22:36,428 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11300, loss[loss=0.05831, simple_loss=0.07757, pruned_loss=0.01436, audio_tagging_loss=0.005165, over 14820.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.0898, pruned_loss=0.01316, audio_tagging_loss=0.009164, over 3052274.13 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:22:43,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2640420.0, ans=0.1
2023-11-24 02:22:54,783 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.41 vs. limit=22.5
2023-11-24 02:23:19,456 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396100
2023-11-24 02:23:34,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2640686.6666666665, ans=0.125
2023-11-24 02:23:38,686 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11350, loss[loss=0.06382, simple_loss=0.08398, pruned_loss=0.01405, audio_tagging_loss=0.007774, over 15142.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09009, pruned_loss=0.01323, audio_tagging_loss=0.00911, over 3045224.07 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:23:52,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2640820.0, ans=0.0
2023-11-24 02:23:57,199 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.374e+01 8.369e+01 8.906e+01 9.576e+01 1.384e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-24 02:23:57,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2640820.0, ans=0.0
2023-11-24 02:24:03,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.56 vs. limit=15.0
2023-11-24 02:24:18,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2640953.3333333335, ans=0.1
2023-11-24 02:24:22,829 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396150
2023-11-24 02:24:30,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2641020.0, ans=0.125
2023-11-24 02:24:41,105 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11400, loss[loss=0.07707, simple_loss=0.09965, pruned_loss=0.01645, audio_tagging_loss=0.01079, over 14476.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09114, pruned_loss=0.01318, audio_tagging_loss=0.008991, over 3047095.51 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:25:21,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2641286.6666666665, ans=0.0
2023-11-24 02:25:23,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396200
2023-11-24 02:25:35,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2641353.3333333335, ans=0.1
2023-11-24 02:25:42,736 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11450, loss[loss=0.06165, simple_loss=0.083, pruned_loss=0.01232, audio_tagging_loss=0.00783, over 15450.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09184, pruned_loss=0.01332, audio_tagging_loss=0.008919, over 3049853.15 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:25:43,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2641420.0, ans=0.0
2023-11-24 02:25:48,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.19 vs. limit=15.0
2023-11-24 02:25:59,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.97 vs. limit=15.0
2023-11-24 02:26:02,134 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.524e+01 9.029e+01 9.681e+01 1.161e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 02:26:04,751 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:26:13,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2641553.3333333335, ans=0.1
2023-11-24 02:26:23,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2641620.0, ans=0.0
2023-11-24 02:26:26,327 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396250
2023-11-24 02:26:45,829 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11500, loss[loss=0.05065, simple_loss=0.06979, pruned_loss=0.007329, audio_tagging_loss=0.008429, over 15045.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09206, pruned_loss=0.01349, audio_tagging_loss=0.008873, over 3044352.16 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:27:01,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2641820.0, ans=0.0
2023-11-24 02:27:14,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2641886.6666666665, ans=0.2
2023-11-24 02:27:27,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2641953.3333333335, ans=0.125
2023-11-24 02:27:27,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2641953.3333333335, ans=0.125
2023-11-24 02:27:30,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396300
2023-11-24 02:27:30,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2641953.3333333335, ans=0.0
2023-11-24 02:27:41,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2642020.0, ans=0.125
2023-11-24 02:27:47,703 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11550, loss[loss=0.06914, simple_loss=0.09583, pruned_loss=0.01396, audio_tagging_loss=0.007267, over 15918.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09146, pruned_loss=0.01337, audio_tagging_loss=0.008901, over 3047369.60 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:27:47,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2642086.6666666665, ans=0.125
2023-11-24 02:28:06,102 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.401e+01 9.028e+01 9.629e+01 1.407e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 02:28:13,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2642220.0, ans=0.0
2023-11-24 02:28:26,653 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:28:31,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396350
2023-11-24 02:28:32,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.23 vs. limit=15.0
2023-11-24 02:28:38,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2642353.3333333335, ans=0.025
2023-11-24 02:28:48,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.57 vs. limit=15.0
2023-11-24 02:28:49,972 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11600, loss[loss=0.06875, simple_loss=0.09401, pruned_loss=0.01314, audio_tagging_loss=0.008611, over 15687.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09164, pruned_loss=0.01337, audio_tagging_loss=0.00883, over 3059314.96 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:29:10,123 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.41 vs. limit=22.5
2023-11-24 02:29:14,723 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.52 vs. limit=22.5
2023-11-24 02:29:21,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2642553.3333333335, ans=0.0
2023-11-24 02:29:25,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2642553.3333333335, ans=0.09899494936611666
2023-11-24 02:29:25,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2642553.3333333335, ans=0.125
2023-11-24 02:29:34,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396400
2023-11-24 02:29:34,783 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.46 vs. limit=15.0
2023-11-24 02:29:53,922 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11650, loss[loss=0.07114, simple_loss=0.1054, pruned_loss=0.01364, audio_tagging_loss=0.004826, over 15381.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09184, pruned_loss=0.01342, audio_tagging_loss=0.008878, over 3057554.88 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:30:03,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2642753.3333333335, ans=0.125
2023-11-24 02:30:12,886 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.278e+01 8.940e+01 9.530e+01 1.250e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 02:30:29,426 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:30:38,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396450
2023-11-24 02:30:55,491 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11700, loss[loss=0.05457, simple_loss=0.07298, pruned_loss=0.009945, audio_tagging_loss=0.008138, over 15337.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09158, pruned_loss=0.01331, audio_tagging_loss=0.0089, over 3060249.51 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:31:06,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2643153.3333333335, ans=0.0
2023-11-24 02:31:14,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.01 vs. limit=15.0
2023-11-24 02:31:20,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.26 vs. limit=15.0
2023-11-24 02:31:39,042 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396500
2023-11-24 02:31:48,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2643353.3333333335, ans=0.125
2023-11-24 02:31:56,461 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11750, loss[loss=0.0724, simple_loss=0.1027, pruned_loss=0.01173, audio_tagging_loss=0.00931, over 15919.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09119, pruned_loss=0.01334, audio_tagging_loss=0.008908, over 3054700.44 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:32:04,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2643420.0, ans=0.0
2023-11-24 02:32:17,386 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.547e+01 9.165e+01 9.829e+01 1.276e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 02:32:32,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2643553.3333333335, ans=0.125
2023-11-24 02:32:33,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2643620.0, ans=0.2
2023-11-24 02:32:40,275 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396550
2023-11-24 02:32:50,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2643686.6666666665, ans=0.125
2023-11-24 02:32:56,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2643686.6666666665, ans=0.2
2023-11-24 02:33:00,490 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11800, loss[loss=0.08325, simple_loss=0.1166, pruned_loss=0.01824, audio_tagging_loss=0.006723, over 15959.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09154, pruned_loss=0.01346, audio_tagging_loss=0.008866, over 3048204.45 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:33:31,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2643886.6666666665, ans=0.0
2023-11-24 02:33:32,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2643886.6666666665, ans=0.1
2023-11-24 02:33:43,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396600
2023-11-24 02:33:51,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2644020.0, ans=0.2
2023-11-24 02:34:01,628 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11850, loss[loss=0.05104, simple_loss=0.07064, pruned_loss=0.008407, audio_tagging_loss=0.007311, over 15067.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09194, pruned_loss=0.01357, audio_tagging_loss=0.00896, over 3046905.04 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:34:03,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2644086.6666666665, ans=0.0
2023-11-24 02:34:06,007 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.98 vs. limit=22.5
2023-11-24 02:34:19,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2644153.3333333335, ans=0.125
2023-11-24 02:34:20,344 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.605e+01 8.557e+01 9.128e+01 9.993e+01 1.182e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 02:34:22,386 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.86 vs. limit=22.5
2023-11-24 02:34:30,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2644220.0, ans=0.125
2023-11-24 02:34:45,198 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396650
2023-11-24 02:35:02,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2644420.0, ans=0.0
2023-11-24 02:35:02,954 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11900, loss[loss=0.05638, simple_loss=0.07271, pruned_loss=0.007727, audio_tagging_loss=0.01229, over 15508.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09182, pruned_loss=0.01357, audio_tagging_loss=0.009092, over 3048612.64 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:35:04,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2644420.0, ans=0.1
2023-11-24 02:35:18,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2644486.6666666665, ans=0.125
2023-11-24 02:35:29,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2644553.3333333335, ans=0.125
2023-11-24 02:35:30,691 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.25 vs. limit=22.5
2023-11-24 02:35:34,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2644553.3333333335, ans=0.0
2023-11-24 02:35:46,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396700
2023-11-24 02:35:58,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.52 vs. limit=15.0
2023-11-24 02:35:59,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.46 vs. limit=6.0
2023-11-24 02:36:00,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2644686.6666666665, ans=0.125
2023-11-24 02:36:05,989 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 11950, loss[loss=0.05628, simple_loss=0.06756, pruned_loss=0.01185, audio_tagging_loss=0.01065, over 15748.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09107, pruned_loss=0.01332, audio_tagging_loss=0.009206, over 3056158.46 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:36:06,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.01 vs. limit=15.0
2023-11-24 02:36:21,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2644820.0, ans=0.05
2023-11-24 02:36:25,401 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.152e+01 8.882e+01 9.494e+01 1.160e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-24 02:36:26,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2644820.0, ans=0.09899494936611666
2023-11-24 02:36:34,353 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.77 vs. limit=6.0
2023-11-24 02:36:38,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2644886.6666666665, ans=0.125
2023-11-24 02:36:38,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2644886.6666666665, ans=0.125
2023-11-24 02:36:39,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2644886.6666666665, ans=0.0
2023-11-24 02:36:47,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396750
2023-11-24 02:36:54,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2645020.0, ans=0.1
2023-11-24 02:37:06,006 INFO [train_asr.py:1221] (2/4) Epoch 33, batch 12000, loss[loss=0.07154, simple_loss=0.1053, pruned_loss=0.01252, audio_tagging_loss=0.006358, over 14947.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09202, pruned_loss=0.01364, audio_tagging_loss=0.009214, over 3052682.91 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:37:06,007 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 02:37:42,972 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.9993, 5.8862, 5.6665, 5.5739], device='cuda:2')
2023-11-24 02:37:46,452 INFO [train_asr.py:1253] (2/4) Epoch 33, validation: loss=0.05829, simple_loss=0.05098, pruned_loss=0.005164, audio_tagging_loss=0.02763, over 4681554.00 frames. 
2023-11-24 02:37:46,453 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 02:37:46,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2645086.6666666665, ans=0.125
2023-11-24 02:37:51,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2645086.6666666665, ans=0.1
2023-11-24 02:38:12,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2645220.0, ans=0.2
2023-11-24 02:38:48,862 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 0, loss[loss=0.09437, simple_loss=0.1159, pruned_loss=0.01808, audio_tagging_loss=0.01833, over 15433.00 frames. ], tot_loss[loss=0.09437, simple_loss=0.1159, pruned_loss=0.01808, audio_tagging_loss=0.01833, over 15433.00 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:38:48,863 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 02:39:24,525 INFO [train_asr.py:1253] (2/4) Epoch 34, validation: loss=0.058, simple_loss=0.05102, pruned_loss=0.005202, audio_tagging_loss=0.02729, over 4681554.00 frames. 
2023-11-24 02:39:24,525 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 02:39:29,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2645253.3333333335, ans=0.0
2023-11-24 02:39:37,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396800
2023-11-24 02:39:49,947 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.28 vs. limit=10.0
2023-11-24 02:39:50,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2645386.6666666665, ans=0.125
2023-11-24 02:39:56,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2645386.6666666665, ans=0.125
2023-11-24 02:40:16,360 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.450e+01 9.137e+01 9.827e+01 1.054e+02 1.431e+02, threshold=1.965e+02, percent-clipped=0.0
2023-11-24 02:40:27,157 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 50, loss[loss=0.09515, simple_loss=0.1166, pruned_loss=0.02175, audio_tagging_loss=0.01509, over 15714.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.09099, pruned_loss=0.01337, audio_tagging_loss=0.0174, over 691243.45 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:40:28,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2645586.6666666665, ans=0.1
2023-11-24 02:40:36,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2645586.6666666665, ans=0.125
2023-11-24 02:40:39,329 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396850
2023-11-24 02:40:44,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2645653.3333333335, ans=0.0
2023-11-24 02:40:51,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2645720.0, ans=0.125
2023-11-24 02:40:56,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2645720.0, ans=0.0
2023-11-24 02:41:14,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2645786.6666666665, ans=0.07
2023-11-24 02:41:20,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2645853.3333333335, ans=0.0
2023-11-24 02:41:22,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2645853.3333333335, ans=0.125
2023-11-24 02:41:29,487 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 100, loss[loss=0.08359, simple_loss=0.1136, pruned_loss=0.01521, audio_tagging_loss=0.0116, over 14548.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.09276, pruned_loss=0.0134, audio_tagging_loss=0.01655, over 1205632.78 frames. ], batch size: 54, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:41:36,147 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.62 vs. limit=22.5
2023-11-24 02:41:41,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396900
2023-11-24 02:42:08,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2646120.0, ans=0.125
2023-11-24 02:42:20,411 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.325e+01 8.964e+01 9.568e+01 1.027e+02 1.310e+02, threshold=1.914e+02, percent-clipped=0.0
2023-11-24 02:42:31,558 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 150, loss[loss=0.04846, simple_loss=0.05364, pruned_loss=0.008384, audio_tagging_loss=0.01326, over 13745.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09059, pruned_loss=0.01303, audio_tagging_loss=0.01506, over 1604328.95 frames. ], batch size: 53, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:42:32,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2646253.3333333335, ans=0.125
2023-11-24 02:42:44,479 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 396950
2023-11-24 02:43:01,776 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.77 vs. limit=12.0
2023-11-24 02:43:08,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2646453.3333333335, ans=0.0
2023-11-24 02:43:27,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2646520.0, ans=0.0
2023-11-24 02:43:30,914 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:43:34,085 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 200, loss[loss=0.08553, simple_loss=0.1113, pruned_loss=0.02081, audio_tagging_loss=0.009051, over 13449.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09194, pruned_loss=0.01327, audio_tagging_loss=0.01319, over 1920877.86 frames. ], batch size: 55, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:43:36,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2646586.6666666665, ans=0.0
2023-11-24 02:43:42,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2646586.6666666665, ans=0.125
2023-11-24 02:43:46,028 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397000
2023-11-24 02:43:58,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2646720.0, ans=0.1
2023-11-24 02:44:20,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2646786.6666666665, ans=0.125
2023-11-24 02:44:22,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2646853.3333333335, ans=0.0
2023-11-24 02:44:23,144 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.96 vs. limit=15.0
2023-11-24 02:44:24,864 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.517e+01 9.117e+01 1.003e+02 1.656e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 02:44:30,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2646853.3333333335, ans=0.125
2023-11-24 02:44:35,765 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 250, loss[loss=0.07658, simple_loss=0.09852, pruned_loss=0.01871, audio_tagging_loss=0.008617, over 14904.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.0925, pruned_loss=0.01355, audio_tagging_loss=0.01194, over 2171175.44 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:44:36,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2646920.0, ans=0.0
2023-11-24 02:44:37,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2646920.0, ans=0.1
2023-11-24 02:44:40,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2646920.0, ans=0.0
2023-11-24 02:44:47,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397050
2023-11-24 02:44:58,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2646986.6666666665, ans=0.125
2023-11-24 02:45:07,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2647053.3333333335, ans=0.125
2023-11-24 02:45:12,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2647120.0, ans=0.0
2023-11-24 02:45:14,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2647120.0, ans=0.125
2023-11-24 02:45:36,990 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 300, loss[loss=0.03999, simple_loss=0.05058, pruned_loss=0.004515, audio_tagging_loss=0.01019, over 16563.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09249, pruned_loss=0.01364, audio_tagging_loss=0.01107, over 2363450.19 frames. ], batch size: 64, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:45:50,869 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397100
2023-11-24 02:46:03,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2647386.6666666665, ans=0.1
2023-11-24 02:46:10,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2647386.6666666665, ans=0.0
2023-11-24 02:46:24,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2647453.3333333335, ans=0.0
2023-11-24 02:46:30,121 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.278e+01 8.816e+01 9.876e+01 1.220e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-24 02:46:33,767 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.61 vs. limit=15.0
2023-11-24 02:46:40,769 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 350, loss[loss=0.08382, simple_loss=0.1249, pruned_loss=0.01445, audio_tagging_loss=0.006941, over 16331.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09191, pruned_loss=0.01335, audio_tagging_loss=0.0105, over 2517275.68 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:46:52,711 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397150
2023-11-24 02:46:56,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2647653.3333333335, ans=0.1
2023-11-24 02:47:12,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2647720.0, ans=0.125
2023-11-24 02:47:31,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2647853.3333333335, ans=0.1
2023-11-24 02:47:33,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2647853.3333333335, ans=0.1
2023-11-24 02:47:42,074 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 400, loss[loss=0.06046, simple_loss=0.07757, pruned_loss=0.01073, audio_tagging_loss=0.01094, over 15513.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09138, pruned_loss=0.01312, audio_tagging_loss=0.01018, over 2636092.20 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:47:49,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2647920.0, ans=0.125
2023-11-24 02:47:54,090 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397200
2023-11-24 02:47:54,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2647986.6666666665, ans=0.0
2023-11-24 02:48:06,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2648053.3333333335, ans=0.2
2023-11-24 02:48:08,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2648053.3333333335, ans=0.125
2023-11-24 02:48:10,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2648053.3333333335, ans=0.2
2023-11-24 02:48:35,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.562e+01 9.066e+01 9.786e+01 1.375e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 02:48:44,724 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 450, loss[loss=0.07058, simple_loss=0.08737, pruned_loss=0.01686, audio_tagging_loss=0.01004, over 13950.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09158, pruned_loss=0.01348, audio_tagging_loss=0.00986, over 2723758.45 frames. ], batch size: 53, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:48:55,486 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:48:57,559 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397250
2023-11-24 02:49:13,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2648386.6666666665, ans=0.125
2023-11-24 02:49:19,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2648386.6666666665, ans=0.125
2023-11-24 02:49:39,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2648520.0, ans=0.125
2023-11-24 02:49:46,798 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 500, loss[loss=0.08547, simple_loss=0.1186, pruned_loss=0.0195, audio_tagging_loss=0.006678, over 14531.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09089, pruned_loss=0.0134, audio_tagging_loss=0.009708, over 2795090.27 frames. ], batch size: 54, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:49:50,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2648586.6666666665, ans=0.125
2023-11-24 02:49:51,026 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.96 vs. limit=6.0
2023-11-24 02:49:53,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=2648586.6666666665, ans=0.02
2023-11-24 02:49:53,664 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.05 vs. limit=15.0
2023-11-24 02:49:59,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397300
2023-11-24 02:50:38,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.347e+01 9.082e+01 9.817e+01 1.239e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 02:50:48,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2648920.0, ans=0.125
2023-11-24 02:50:48,950 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 550, loss[loss=0.07968, simple_loss=0.1095, pruned_loss=0.01527, audio_tagging_loss=0.009667, over 15894.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09034, pruned_loss=0.01327, audio_tagging_loss=0.009671, over 2848113.02 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:50:56,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2648920.0, ans=0.125
2023-11-24 02:50:57,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.08 vs. limit=15.0
2023-11-24 02:51:00,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397350
2023-11-24 02:51:13,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2649053.3333333335, ans=0.0
2023-11-24 02:51:43,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2649186.6666666665, ans=0.125
2023-11-24 02:51:49,953 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 600, loss[loss=0.06587, simple_loss=0.08809, pruned_loss=0.01523, audio_tagging_loss=0.006599, over 15937.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09015, pruned_loss=0.01319, audio_tagging_loss=0.009539, over 2894288.14 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:52:03,022 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397400
2023-11-24 02:52:09,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2649320.0, ans=0.0
2023-11-24 02:52:19,515 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.62 vs. limit=10.0
2023-11-24 02:52:29,144 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.30 vs. limit=22.5
2023-11-24 02:52:31,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2649453.3333333335, ans=0.0
2023-11-24 02:52:40,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2649520.0, ans=0.0
2023-11-24 02:52:42,870 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.375e+01 9.177e+01 9.802e+01 2.402e+02, threshold=1.835e+02, percent-clipped=1.0
2023-11-24 02:52:52,940 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 650, loss[loss=0.06693, simple_loss=0.09509, pruned_loss=0.01227, audio_tagging_loss=0.007123, over 15226.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.08979, pruned_loss=0.01315, audio_tagging_loss=0.009429, over 2917133.43 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:53:04,697 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397450
2023-11-24 02:53:04,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2649653.3333333335, ans=0.2
2023-11-24 02:53:07,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2649653.3333333335, ans=0.125
2023-11-24 02:53:19,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2649720.0, ans=0.95
2023-11-24 02:53:22,421 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.10 vs. limit=22.5
2023-11-24 02:53:47,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2649853.3333333335, ans=0.1
2023-11-24 02:53:54,369 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 700, loss[loss=0.07294, simple_loss=0.08659, pruned_loss=0.01784, audio_tagging_loss=0.01181, over 14443.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.0904, pruned_loss=0.01325, audio_tagging_loss=0.009411, over 2946903.23 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:53:54,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2649920.0, ans=0.0
2023-11-24 02:54:06,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2649986.6666666665, ans=0.125
2023-11-24 02:54:06,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397500
2023-11-24 02:54:30,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2650120.0, ans=0.125
2023-11-24 02:54:45,332 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.74 vs. limit=12.0
2023-11-24 02:54:47,547 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.698e+01 9.188e+01 9.937e+01 1.605e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-24 02:54:55,818 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 750, loss[loss=0.08883, simple_loss=0.1295, pruned_loss=0.01618, audio_tagging_loss=0.0079, over 15486.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09108, pruned_loss=0.01355, audio_tagging_loss=0.009313, over 2971349.82 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:54:57,316 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:55:09,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397550
2023-11-24 02:55:11,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2650320.0, ans=0.0
2023-11-24 02:55:40,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2650453.3333333335, ans=0.2
2023-11-24 02:55:44,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2650520.0, ans=0.0
2023-11-24 02:55:53,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2650520.0, ans=0.1
2023-11-24 02:55:58,172 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 800, loss[loss=0.06365, simple_loss=0.08407, pruned_loss=0.01116, audio_tagging_loss=0.01046, over 16096.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09149, pruned_loss=0.01359, audio_tagging_loss=0.009248, over 2988885.43 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:56:10,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397600
2023-11-24 02:56:11,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2650653.3333333335, ans=0.125
2023-11-24 02:56:11,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.41 vs. limit=15.0
2023-11-24 02:56:12,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2650653.3333333335, ans=0.125
2023-11-24 02:56:47,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2650853.3333333335, ans=0.125
2023-11-24 02:56:51,289 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 8.448e+01 9.134e+01 9.872e+01 1.389e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 02:56:59,545 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 850, loss[loss=0.06403, simple_loss=0.08427, pruned_loss=0.01326, audio_tagging_loss=0.008639, over 14750.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.0916, pruned_loss=0.01369, audio_tagging_loss=0.009259, over 3004020.33 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:57:09,651 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.69 vs. limit=15.0
2023-11-24 02:57:12,029 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397650
2023-11-24 02:57:16,933 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:57:32,321 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.62 vs. limit=10.0
2023-11-24 02:57:44,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.88 vs. limit=6.0
2023-11-24 02:57:53,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2651186.6666666665, ans=0.125
2023-11-24 02:58:02,060 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 900, loss[loss=0.06161, simple_loss=0.08142, pruned_loss=0.01301, audio_tagging_loss=0.007892, over 15887.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09156, pruned_loss=0.01362, audio_tagging_loss=0.009311, over 3021060.62 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:58:14,482 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397700
2023-11-24 02:58:25,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2651320.0, ans=0.125
2023-11-24 02:58:39,856 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.98 vs. limit=22.5
2023-11-24 02:58:45,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2651453.3333333335, ans=0.035
2023-11-24 02:58:47,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2651453.3333333335, ans=0.0
2023-11-24 02:58:56,280 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.194e+01 8.559e+01 9.218e+01 1.010e+02 1.214e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 02:58:58,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2651520.0, ans=0.125
2023-11-24 02:59:00,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2651520.0, ans=0.0
2023-11-24 02:59:04,533 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 950, loss[loss=0.06764, simple_loss=0.08755, pruned_loss=0.01366, audio_tagging_loss=0.0102, over 14807.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09077, pruned_loss=0.01353, audio_tagging_loss=0.009343, over 3024151.88 frames. ], batch size: 54, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:59:17,035 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397750
2023-11-24 02:59:24,590 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.35 vs. limit=15.0
2023-11-24 02:59:30,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2651720.0, ans=0.0
2023-11-24 02:59:32,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2651720.0, ans=0.125
2023-11-24 02:59:40,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.31 vs. limit=22.5
2023-11-24 02:59:54,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2651853.3333333335, ans=0.125
2023-11-24 02:59:54,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2651853.3333333335, ans=0.0
2023-11-24 02:59:56,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2651853.3333333335, ans=0.1
2023-11-24 02:59:56,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.53 vs. limit=22.5
2023-11-24 03:00:06,445 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1000, loss[loss=0.05906, simple_loss=0.08005, pruned_loss=0.01079, audio_tagging_loss=0.008245, over 15000.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09123, pruned_loss=0.01347, audio_tagging_loss=0.009208, over 3032936.81 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:00:18,402 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397800
2023-11-24 03:00:30,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2652053.3333333335, ans=0.0
2023-11-24 03:00:32,250 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:00:38,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=15.0
2023-11-24 03:00:42,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-24 03:00:45,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2652120.0, ans=0.125
2023-11-24 03:01:01,760 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.415e+01 8.930e+01 9.689e+01 1.161e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 03:01:09,029 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1050, loss[loss=0.06987, simple_loss=0.09771, pruned_loss=0.01512, audio_tagging_loss=0.005897, over 14469.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09064, pruned_loss=0.01336, audio_tagging_loss=0.009095, over 3030879.24 frames. ], batch size: 53, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:01:21,498 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397850
2023-11-24 03:01:52,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=14.04 vs. limit=15.0
2023-11-24 03:02:06,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2652520.0, ans=0.125
2023-11-24 03:02:06,967 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.73 vs. limit=15.0
2023-11-24 03:02:10,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2652586.6666666665, ans=0.09899494936611666
2023-11-24 03:02:11,742 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1100, loss[loss=0.07013, simple_loss=0.08844, pruned_loss=0.01387, audio_tagging_loss=0.01203, over 14818.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09045, pruned_loss=0.01328, audio_tagging_loss=0.008981, over 3036114.54 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:02:15,966 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:02:16,854 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.68 vs. limit=22.5
2023-11-24 03:02:24,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397900
2023-11-24 03:02:27,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2652653.3333333335, ans=0.125
2023-11-24 03:02:29,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2652653.3333333335, ans=0.0
2023-11-24 03:02:36,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2652720.0, ans=0.2
2023-11-24 03:02:39,902 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:02:40,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2652720.0, ans=0.125
2023-11-24 03:02:46,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2652786.6666666665, ans=0.125
2023-11-24 03:03:06,342 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.355e+01 8.539e+01 9.041e+01 9.654e+01 1.220e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 03:03:13,515 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1150, loss[loss=0.06157, simple_loss=0.08396, pruned_loss=0.01008, audio_tagging_loss=0.009505, over 15913.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09052, pruned_loss=0.0133, audio_tagging_loss=0.008983, over 3036233.53 frames. ], batch size: 60, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:03:25,490 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 397950
2023-11-24 03:03:27,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2652986.6666666665, ans=0.125
2023-11-24 03:03:32,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2652986.6666666665, ans=0.0
2023-11-24 03:03:57,409 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.10 vs. limit=15.0
2023-11-24 03:03:59,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2653120.0, ans=0.125
2023-11-24 03:03:59,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2653120.0, ans=0.125
2023-11-24 03:04:10,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2653186.6666666665, ans=0.0
2023-11-24 03:04:14,287 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1200, loss[loss=0.08464, simple_loss=0.1092, pruned_loss=0.02316, audio_tagging_loss=0.006872, over 15692.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09005, pruned_loss=0.01331, audio_tagging_loss=0.008972, over 3033956.37 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:04:16,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2653253.3333333335, ans=0.125
2023-11-24 03:04:25,882 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.0
2023-11-24 03:04:26,331 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398000
2023-11-24 03:04:39,480 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.72 vs. limit=15.0
2023-11-24 03:04:54,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2653453.3333333335, ans=0.2
2023-11-24 03:05:09,787 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.593e+01 9.162e+01 9.902e+01 1.327e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 03:05:16,277 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1250, loss[loss=0.08816, simple_loss=0.1194, pruned_loss=0.02066, audio_tagging_loss=0.007793, over 15043.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09062, pruned_loss=0.01346, audio_tagging_loss=0.008897, over 3035403.59 frames. ], batch size: 55, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:05:25,188 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.71 vs. limit=15.0
2023-11-24 03:05:29,453 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398050
2023-11-24 03:05:42,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2653720.0, ans=0.0
2023-11-24 03:05:49,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.91 vs. limit=15.0
2023-11-24 03:06:11,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2653853.3333333335, ans=0.125
2023-11-24 03:06:12,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2653853.3333333335, ans=0.125
2023-11-24 03:06:18,462 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1300, loss[loss=0.0738, simple_loss=0.105, pruned_loss=0.01418, audio_tagging_loss=0.007136, over 15218.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09092, pruned_loss=0.01345, audio_tagging_loss=0.008853, over 3031865.70 frames. ], batch size: 55, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:06:24,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2653920.0, ans=0.5
2023-11-24 03:06:30,330 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398100
2023-11-24 03:06:38,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2653986.6666666665, ans=0.2
2023-11-24 03:06:41,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2654053.3333333335, ans=0.125
2023-11-24 03:07:13,594 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.353e+01 8.205e+01 8.973e+01 9.705e+01 1.641e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-24 03:07:19,489 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1350, loss[loss=0.05549, simple_loss=0.0691, pruned_loss=0.01322, audio_tagging_loss=0.007712, over 13863.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.08979, pruned_loss=0.01327, audio_tagging_loss=0.00901, over 3032298.03 frames. ], batch size: 53, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:07:22,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2654253.3333333335, ans=0.0
2023-11-24 03:07:30,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2654320.0, ans=10.0
2023-11-24 03:07:31,285 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398150
2023-11-24 03:07:36,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2654320.0, ans=0.0
2023-11-24 03:07:45,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2654386.6666666665, ans=0.07
2023-11-24 03:08:02,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.64 vs. limit=15.0
2023-11-24 03:08:03,981 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:08:06,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2654453.3333333335, ans=10.0
2023-11-24 03:08:18,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2654520.0, ans=0.125
2023-11-24 03:08:20,640 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1400, loss[loss=0.05698, simple_loss=0.07688, pruned_loss=0.009044, audio_tagging_loss=0.009493, over 15016.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.08969, pruned_loss=0.01331, audio_tagging_loss=0.009061, over 3033636.42 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:08:20,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2654586.6666666665, ans=0.125
2023-11-24 03:08:34,322 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398200
2023-11-24 03:09:17,338 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.476e+01 9.016e+01 9.845e+01 1.286e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 03:09:23,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2654920.0, ans=0.1
2023-11-24 03:09:24,009 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1450, loss[loss=0.06747, simple_loss=0.08745, pruned_loss=0.01437, audio_tagging_loss=0.009373, over 15540.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.08994, pruned_loss=0.01318, audio_tagging_loss=0.009101, over 3040045.47 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:09:27,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.20 vs. limit=15.0
2023-11-24 03:09:28,442 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.27 vs. limit=22.5
2023-11-24 03:09:35,957 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398250
2023-11-24 03:09:36,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2654986.6666666665, ans=0.125
2023-11-24 03:09:40,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2654986.6666666665, ans=0.0
2023-11-24 03:09:53,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2655053.3333333335, ans=0.09899494936611666
2023-11-24 03:09:54,010 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0
2023-11-24 03:10:11,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2655120.0, ans=0.1
2023-11-24 03:10:11,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2655120.0, ans=0.0
2023-11-24 03:10:21,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2655186.6666666665, ans=0.125
2023-11-24 03:10:25,534 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1500, loss[loss=0.08362, simple_loss=0.1115, pruned_loss=0.01913, audio_tagging_loss=0.008734, over 16002.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09016, pruned_loss=0.0133, audio_tagging_loss=0.0092, over 3041750.47 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:10:29,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2655253.3333333335, ans=0.95
2023-11-24 03:10:37,545 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398300
2023-11-24 03:10:45,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2655320.0, ans=0.125
2023-11-24 03:10:56,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2655386.6666666665, ans=0.0
2023-11-24 03:10:58,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2655386.6666666665, ans=0.1
2023-11-24 03:11:09,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2655453.3333333335, ans=15.0
2023-11-24 03:11:18,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2655520.0, ans=0.07
2023-11-24 03:11:21,389 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.576e+01 9.222e+01 1.014e+02 1.240e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 03:11:27,356 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1550, loss[loss=0.06392, simple_loss=0.08077, pruned_loss=0.01283, audio_tagging_loss=0.0107, over 14895.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.08933, pruned_loss=0.01319, audio_tagging_loss=0.009337, over 3043294.49 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:11:31,520 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.34 vs. limit=15.0
2023-11-24 03:11:38,374 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:11:41,054 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398350
2023-11-24 03:11:46,197 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:11:51,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2655653.3333333335, ans=0.125
2023-11-24 03:12:03,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2655720.0, ans=0.0
2023-11-24 03:12:31,483 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1600, loss[loss=0.0498, simple_loss=0.05784, pruned_loss=0.009366, audio_tagging_loss=0.01151, over 13838.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.08979, pruned_loss=0.01326, audio_tagging_loss=0.009331, over 3041825.98 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:12:43,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398400
2023-11-24 03:12:46,857 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:12:57,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2656053.3333333335, ans=0.1
2023-11-24 03:13:15,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2656120.0, ans=0.0
2023-11-24 03:13:21,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2656186.6666666665, ans=0.2
2023-11-24 03:13:24,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2656186.6666666665, ans=0.125
2023-11-24 03:13:28,027 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.243e+01 8.406e+01 9.091e+01 9.636e+01 1.326e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 03:13:30,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2656186.6666666665, ans=0.0
2023-11-24 03:13:33,939 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1650, loss[loss=0.08404, simple_loss=0.1122, pruned_loss=0.02099, audio_tagging_loss=0.006955, over 13969.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09056, pruned_loss=0.01341, audio_tagging_loss=0.009279, over 3040403.97 frames. ], batch size: 52, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:13:45,980 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398450
2023-11-24 03:13:54,724 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2656320.0, ans=0.95
2023-11-24 03:14:04,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2656386.6666666665, ans=0.2
2023-11-24 03:14:17,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.19 vs. limit=22.5
2023-11-24 03:14:19,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2656453.3333333335, ans=0.125
2023-11-24 03:14:24,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2656520.0, ans=0.09899494936611666
2023-11-24 03:14:36,425 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1700, loss[loss=0.06103, simple_loss=0.08013, pruned_loss=0.01299, audio_tagging_loss=0.007977, over 15618.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09053, pruned_loss=0.01327, audio_tagging_loss=0.009354, over 3051292.49 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:14:39,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2656586.6666666665, ans=0.1
2023-11-24 03:14:49,571 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398500
2023-11-24 03:14:55,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2656653.3333333335, ans=10.0
2023-11-24 03:14:59,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2656653.3333333335, ans=0.2
2023-11-24 03:15:07,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2656720.0, ans=0.125
2023-11-24 03:15:07,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.19 vs. limit=15.0
2023-11-24 03:15:32,712 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.306e+01 8.348e+01 8.790e+01 9.631e+01 1.141e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-24 03:15:33,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.02 vs. limit=12.0
2023-11-24 03:15:35,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2656853.3333333335, ans=0.125
2023-11-24 03:15:36,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2656853.3333333335, ans=0.125
2023-11-24 03:15:39,525 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1750, loss[loss=0.07562, simple_loss=0.09956, pruned_loss=0.01639, audio_tagging_loss=0.009454, over 14821.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.08959, pruned_loss=0.01316, audio_tagging_loss=0.009261, over 3053313.10 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:15:48,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.66 vs. limit=15.0
2023-11-24 03:15:51,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398550
2023-11-24 03:16:13,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2657053.3333333335, ans=0.1
2023-11-24 03:16:13,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2657053.3333333335, ans=0.0
2023-11-24 03:16:22,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2657120.0, ans=0.1
2023-11-24 03:16:41,188 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1800, loss[loss=0.07049, simple_loss=0.09373, pruned_loss=0.01486, audio_tagging_loss=0.008764, over 15592.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.08979, pruned_loss=0.01306, audio_tagging_loss=0.009212, over 3058074.88 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:16:43,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2657253.3333333335, ans=0.125
2023-11-24 03:16:45,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2657253.3333333335, ans=0.0
2023-11-24 03:16:46,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2657253.3333333335, ans=0.125
2023-11-24 03:16:53,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398600
2023-11-24 03:17:11,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2657386.6666666665, ans=0.1
2023-11-24 03:17:13,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2657386.6666666665, ans=0.1
2023-11-24 03:17:20,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2657453.3333333335, ans=0.125
2023-11-24 03:17:37,903 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.560e+01 9.161e+01 9.912e+01 1.351e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 03:17:43,923 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1850, loss[loss=0.08884, simple_loss=0.1105, pruned_loss=0.02321, audio_tagging_loss=0.01041, over 14918.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.0896, pruned_loss=0.01311, audio_tagging_loss=0.009141, over 3052697.68 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:17:45,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.84 vs. limit=15.0
2023-11-24 03:17:47,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2657586.6666666665, ans=0.125
2023-11-24 03:17:47,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2657586.6666666665, ans=0.0
2023-11-24 03:17:56,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398650
2023-11-24 03:18:23,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2657786.6666666665, ans=0.0
2023-11-24 03:18:23,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2657786.6666666665, ans=0.0
2023-11-24 03:18:35,745 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.45 vs. limit=15.0
2023-11-24 03:18:46,485 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.11 vs. limit=15.0
2023-11-24 03:18:47,045 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1900, loss[loss=0.07794, simple_loss=0.1097, pruned_loss=0.01716, audio_tagging_loss=0.005904, over 15439.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08868, pruned_loss=0.01302, audio_tagging_loss=0.009082, over 3044649.85 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:18:59,030 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398700
2023-11-24 03:19:02,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2657986.6666666665, ans=0.2
2023-11-24 03:19:05,573 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.86 vs. limit=6.0
2023-11-24 03:19:37,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2658186.6666666665, ans=0.125
2023-11-24 03:19:43,007 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.306e+01 8.939e+01 9.602e+01 1.298e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 03:19:47,772 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 1950, loss[loss=0.05245, simple_loss=0.06543, pruned_loss=0.01059, audio_tagging_loss=0.009144, over 13555.00 frames. ], tot_loss[loss=0.06637, simple_loss=0.08893, pruned_loss=0.01292, audio_tagging_loss=0.008986, over 3043476.32 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:20:00,307 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398750
2023-11-24 03:20:17,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff2.min_abs, batch_count=2658386.6666666665, ans=0.1
2023-11-24 03:20:49,781 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2000, loss[loss=0.06125, simple_loss=0.08147, pruned_loss=0.01158, audio_tagging_loss=0.008938, over 15239.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.08935, pruned_loss=0.01314, audio_tagging_loss=0.00903, over 3045026.95 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:21:02,277 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398800
2023-11-24 03:21:06,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2658653.3333333335, ans=0.2
2023-11-24 03:21:07,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2658653.3333333335, ans=0.0
2023-11-24 03:21:22,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2658720.0, ans=0.125
2023-11-24 03:21:23,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2658720.0, ans=0.0
2023-11-24 03:21:46,678 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.321e+01 9.002e+01 9.770e+01 1.545e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 03:21:49,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2658853.3333333335, ans=0.0
2023-11-24 03:21:51,412 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2050, loss[loss=0.08483, simple_loss=0.1142, pruned_loss=0.01813, audio_tagging_loss=0.009598, over 15478.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09001, pruned_loss=0.01328, audio_tagging_loss=0.009049, over 3047130.86 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:22:04,411 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398850
2023-11-24 03:22:05,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2658986.6666666665, ans=0.2
2023-11-24 03:22:06,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2658986.6666666665, ans=0.125
2023-11-24 03:22:19,169 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.20 vs. limit=22.5
2023-11-24 03:22:21,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2659053.3333333335, ans=0.2
2023-11-24 03:22:23,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2659053.3333333335, ans=0.125
2023-11-24 03:22:41,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.31 vs. limit=10.0
2023-11-24 03:22:43,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2659186.6666666665, ans=0.125
2023-11-24 03:22:54,381 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2100, loss[loss=0.06878, simple_loss=0.087, pruned_loss=0.01517, audio_tagging_loss=0.01012, over 14632.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09094, pruned_loss=0.01343, audio_tagging_loss=0.008875, over 3040748.20 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:23:06,747 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398900
2023-11-24 03:23:10,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2659320.0, ans=0.0
2023-11-24 03:23:22,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2659386.6666666665, ans=0.0
2023-11-24 03:23:35,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2659453.3333333335, ans=0.125
2023-11-24 03:23:52,485 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.464e+01 8.942e+01 9.567e+01 1.124e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 03:23:56,151 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2150, loss[loss=0.07109, simple_loss=0.09069, pruned_loss=0.01481, audio_tagging_loss=0.01092, over 15721.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09001, pruned_loss=0.01328, audio_tagging_loss=0.008907, over 3039460.66 frames. ], batch size: 64, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:24:01,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.89 vs. limit=10.0
2023-11-24 03:24:09,333 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 398950
2023-11-24 03:24:23,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2659720.0, ans=0.125
2023-11-24 03:24:24,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2659720.0, ans=0.125
2023-11-24 03:24:27,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2659720.0, ans=0.0
2023-11-24 03:24:34,090 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:24:37,008 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.59 vs. limit=15.0
2023-11-24 03:24:47,923 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.91 vs. limit=10.0
2023-11-24 03:24:52,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2659853.3333333335, ans=0.0
2023-11-24 03:24:59,250 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2200, loss[loss=0.06772, simple_loss=0.08061, pruned_loss=0.01619, audio_tagging_loss=0.01122, over 14817.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09114, pruned_loss=0.01341, audio_tagging_loss=0.00885, over 3045868.20 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:25:03,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2659920.0, ans=0.125
2023-11-24 03:25:11,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2659986.6666666665, ans=0.0
2023-11-24 03:25:12,592 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399000
2023-11-24 03:25:34,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2660053.3333333335, ans=0.125
2023-11-24 03:25:35,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2660120.0, ans=0.125
2023-11-24 03:25:44,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2660120.0, ans=0.125
2023-11-24 03:25:55,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2660186.6666666665, ans=0.125
2023-11-24 03:25:58,367 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.513e+01 9.169e+01 1.008e+02 1.249e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 03:26:01,965 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2250, loss[loss=0.08162, simple_loss=0.1058, pruned_loss=0.01866, audio_tagging_loss=0.01007, over 15671.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09109, pruned_loss=0.01333, audio_tagging_loss=0.008873, over 3042390.89 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:26:02,468 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.92 vs. limit=6.0
2023-11-24 03:26:13,827 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399050
2023-11-24 03:26:22,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2660320.0, ans=0.1
2023-11-24 03:26:40,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2660453.3333333335, ans=0.2
2023-11-24 03:27:02,887 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2300, loss[loss=0.0806, simple_loss=0.1047, pruned_loss=0.01888, audio_tagging_loss=0.009393, over 14475.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.0914, pruned_loss=0.01332, audio_tagging_loss=0.008997, over 3044128.53 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:27:05,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2660586.6666666665, ans=0.125
2023-11-24 03:27:06,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2660586.6666666665, ans=10.0
2023-11-24 03:27:11,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2660586.6666666665, ans=0.2
2023-11-24 03:27:14,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399100
2023-11-24 03:27:17,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2660653.3333333335, ans=0.0
2023-11-24 03:27:19,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2660653.3333333335, ans=0.09899494936611666
2023-11-24 03:27:35,032 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2660720.0, ans=0.125
2023-11-24 03:27:57,447 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:28:00,932 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.716e+01 8.488e+01 8.965e+01 9.835e+01 1.215e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 03:28:05,075 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2350, loss[loss=0.06948, simple_loss=0.09672, pruned_loss=0.01169, audio_tagging_loss=0.009437, over 14608.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09131, pruned_loss=0.01343, audio_tagging_loss=0.009089, over 3045440.71 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:28:18,717 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399150
2023-11-24 03:28:31,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2661053.3333333335, ans=0.125
2023-11-24 03:28:42,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2661120.0, ans=0.125
2023-11-24 03:29:08,998 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2400, loss[loss=0.08581, simple_loss=0.1255, pruned_loss=0.01732, audio_tagging_loss=0.005757, over 15364.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09182, pruned_loss=0.01348, audio_tagging_loss=0.009246, over 3058333.66 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:29:09,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2661253.3333333335, ans=0.0
2023-11-24 03:29:09,766 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.83 vs. limit=15.0
2023-11-24 03:29:13,382 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.31 vs. limit=6.0
2023-11-24 03:29:21,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399200
2023-11-24 03:29:31,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2661320.0, ans=0.0
2023-11-24 03:29:42,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2661386.6666666665, ans=0.125
2023-11-24 03:29:45,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.42 vs. limit=22.5
2023-11-24 03:30:00,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.99 vs. limit=10.0
2023-11-24 03:30:01,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2661520.0, ans=0.125
2023-11-24 03:30:08,340 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.386e+01 9.132e+01 1.001e+02 1.473e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 03:30:10,821 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2450, loss[loss=0.05864, simple_loss=0.07362, pruned_loss=0.01197, audio_tagging_loss=0.009867, over 14530.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09072, pruned_loss=0.01315, audio_tagging_loss=0.009266, over 3049519.00 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:30:15,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2661586.6666666665, ans=0.1
2023-11-24 03:30:22,806 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399250
2023-11-24 03:31:11,903 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2500, loss[loss=0.07527, simple_loss=0.1093, pruned_loss=0.01282, audio_tagging_loss=0.007788, over 15979.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09163, pruned_loss=0.01319, audio_tagging_loss=0.009245, over 3056239.23 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:31:14,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2661920.0, ans=0.1
2023-11-24 03:31:26,299 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399300
2023-11-24 03:31:28,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2661986.6666666665, ans=0.125
2023-11-24 03:31:37,408 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.99 vs. limit=10.0
2023-11-24 03:31:40,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2662053.3333333335, ans=0.0
2023-11-24 03:31:58,169 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.24 vs. limit=15.0
2023-11-24 03:32:13,409 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.419e+01 8.936e+01 9.796e+01 1.161e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-24 03:32:16,384 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2550, loss[loss=0.07442, simple_loss=0.1035, pruned_loss=0.0148, audio_tagging_loss=0.007847, over 15259.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.0914, pruned_loss=0.01333, audio_tagging_loss=0.009099, over 3048858.51 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:32:16,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2662253.3333333335, ans=10.0
2023-11-24 03:32:20,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2662253.3333333335, ans=0.0
2023-11-24 03:32:23,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2662253.3333333335, ans=0.125
2023-11-24 03:32:28,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399350
2023-11-24 03:32:34,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2662320.0, ans=0.0
2023-11-24 03:32:37,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2662320.0, ans=0.0
2023-11-24 03:32:41,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2662386.6666666665, ans=0.04949747468305833
2023-11-24 03:32:44,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2662386.6666666665, ans=0.125
2023-11-24 03:32:47,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2662386.6666666665, ans=0.125
2023-11-24 03:32:54,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2662453.3333333335, ans=0.125
2023-11-24 03:33:18,425 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2600, loss[loss=0.07794, simple_loss=0.1088, pruned_loss=0.0151, audio_tagging_loss=0.008431, over 15663.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09239, pruned_loss=0.01344, audio_tagging_loss=0.00896, over 3051908.94 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:33:27,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2662586.6666666665, ans=0.125
2023-11-24 03:33:30,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399400
2023-11-24 03:34:17,522 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.667e+01 9.247e+01 9.854e+01 1.236e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 03:34:19,849 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2650, loss[loss=0.07157, simple_loss=0.09625, pruned_loss=0.01597, audio_tagging_loss=0.007473, over 16075.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09276, pruned_loss=0.01358, audio_tagging_loss=0.008908, over 3047965.58 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:34:22,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2662920.0, ans=0.0
2023-11-24 03:34:32,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399450
2023-11-24 03:34:43,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2662986.6666666665, ans=0.0
2023-11-24 03:34:48,908 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.46 vs. limit=22.5
2023-11-24 03:35:08,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2663186.6666666665, ans=0.125
2023-11-24 03:35:23,162 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2700, loss[loss=0.06016, simple_loss=0.0815, pruned_loss=0.01217, audio_tagging_loss=0.007239, over 15229.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09286, pruned_loss=0.01371, audio_tagging_loss=0.008843, over 3048135.03 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:35:35,649 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399500
2023-11-24 03:35:40,624 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:35:41,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2663320.0, ans=0.0
2023-11-24 03:36:14,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2663520.0, ans=0.125
2023-11-24 03:36:23,397 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.382e+01 8.295e+01 8.758e+01 9.490e+01 1.330e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-24 03:36:25,799 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2750, loss[loss=0.0771, simple_loss=0.1122, pruned_loss=0.01377, audio_tagging_loss=0.007224, over 15420.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09193, pruned_loss=0.0135, audio_tagging_loss=0.008824, over 3042393.01 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:36:28,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2663586.6666666665, ans=0.1
2023-11-24 03:36:38,028 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399550
2023-11-24 03:36:38,272 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:36:39,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2663653.3333333335, ans=0.125
2023-11-24 03:36:41,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.69 vs. limit=22.5
2023-11-24 03:36:57,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2663720.0, ans=0.125
2023-11-24 03:37:03,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2663786.6666666665, ans=0.0
2023-11-24 03:37:03,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2663786.6666666665, ans=0.125
2023-11-24 03:37:18,084 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:37:20,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2663853.3333333335, ans=0.125
2023-11-24 03:37:27,423 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2800, loss[loss=0.08354, simple_loss=0.1181, pruned_loss=0.01828, audio_tagging_loss=0.006222, over 15049.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.0911, pruned_loss=0.01343, audio_tagging_loss=0.008893, over 3038750.67 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:37:35,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2663920.0, ans=0.1
2023-11-24 03:37:35,756 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.49 vs. limit=15.0
2023-11-24 03:37:40,123 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399600
2023-11-24 03:37:52,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.26 vs. limit=15.0
2023-11-24 03:38:02,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2664053.3333333335, ans=0.015
2023-11-24 03:38:08,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2664120.0, ans=0.125
2023-11-24 03:38:12,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2664120.0, ans=0.0
2023-11-24 03:38:27,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.351e+01 8.911e+01 9.699e+01 1.155e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-24 03:38:30,610 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2850, loss[loss=0.08402, simple_loss=0.1147, pruned_loss=0.02064, audio_tagging_loss=0.006051, over 14733.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09067, pruned_loss=0.01338, audio_tagging_loss=0.008848, over 3035349.57 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:38:42,907 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399650
2023-11-24 03:38:58,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2664386.6666666665, ans=0.125
2023-11-24 03:39:08,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.56 vs. limit=15.0
2023-11-24 03:39:30,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2664520.0, ans=0.2
2023-11-24 03:39:32,504 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2900, loss[loss=0.06781, simple_loss=0.09641, pruned_loss=0.01066, audio_tagging_loss=0.008948, over 16156.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09077, pruned_loss=0.0135, audio_tagging_loss=0.008844, over 3040481.59 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:39:35,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2664586.6666666665, ans=0.125
2023-11-24 03:39:45,026 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399700
2023-11-24 03:39:54,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2664653.3333333335, ans=0.125
2023-11-24 03:40:01,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2664720.0, ans=0.125
2023-11-24 03:40:03,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2664720.0, ans=0.07
2023-11-24 03:40:11,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2664786.6666666665, ans=0.125
2023-11-24 03:40:11,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2664786.6666666665, ans=0.09899494936611666
2023-11-24 03:40:11,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2664786.6666666665, ans=0.2
2023-11-24 03:40:24,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2664853.3333333335, ans=0.125
2023-11-24 03:40:31,988 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.311e+01 9.104e+01 9.838e+01 1.191e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 03:40:34,372 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 2950, loss[loss=0.05927, simple_loss=0.07821, pruned_loss=0.008162, audio_tagging_loss=0.012, over 15623.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09185, pruned_loss=0.01367, audio_tagging_loss=0.008816, over 3042876.49 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:40:41,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2664920.0, ans=0.1
2023-11-24 03:40:45,045 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.14 vs. limit=15.0
2023-11-24 03:40:47,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399750
2023-11-24 03:40:49,995 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-24 03:41:00,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2665053.3333333335, ans=0.0
2023-11-24 03:41:08,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2665053.3333333335, ans=0.1
2023-11-24 03:41:15,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2665120.0, ans=0.125
2023-11-24 03:41:29,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2665186.6666666665, ans=0.125
2023-11-24 03:41:36,937 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3000, loss[loss=0.07042, simple_loss=0.1039, pruned_loss=0.01231, audio_tagging_loss=0.006152, over 15764.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09172, pruned_loss=0.01365, audio_tagging_loss=0.008927, over 3039233.54 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:41:36,937 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 03:42:00,640 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.5048, 2.2378, 3.0988, 2.5380], device='cuda:2')
2023-11-24 03:42:18,352 INFO [train_asr.py:1253] (2/4) Epoch 34, validation: loss=0.05766, simple_loss=0.05087, pruned_loss=0.005081, audio_tagging_loss=0.02714, over 4681554.00 frames. 
2023-11-24 03:42:18,352 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 03:42:29,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2665253.3333333335, ans=0.1
2023-11-24 03:42:31,052 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399800
2023-11-24 03:43:09,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2665520.0, ans=0.09899494936611666
2023-11-24 03:43:18,712 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.546e+01 8.558e+01 9.257e+01 9.837e+01 1.292e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 03:43:21,063 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3050, loss[loss=0.08055, simple_loss=0.1088, pruned_loss=0.01987, audio_tagging_loss=0.006283, over 15674.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09198, pruned_loss=0.01373, audio_tagging_loss=0.008936, over 3039533.17 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:43:21,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2665586.6666666665, ans=0.0
2023-11-24 03:43:34,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399850
2023-11-24 03:43:36,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2665653.3333333335, ans=0.0
2023-11-24 03:43:57,319 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:44:02,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2665786.6666666665, ans=0.125
2023-11-24 03:44:12,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2665853.3333333335, ans=0.125
2023-11-24 03:44:12,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2665853.3333333335, ans=0.125
2023-11-24 03:44:12,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2665853.3333333335, ans=0.2
2023-11-24 03:44:18,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2665853.3333333335, ans=0.0
2023-11-24 03:44:23,095 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2665920.0, ans=0.1
2023-11-24 03:44:23,995 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3100, loss[loss=0.05644, simple_loss=0.0643, pruned_loss=0.0107, audio_tagging_loss=0.01359, over 15749.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09224, pruned_loss=0.01384, audio_tagging_loss=0.009043, over 3048049.33 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:44:29,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2665920.0, ans=0.125
2023-11-24 03:44:36,341 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399900
2023-11-24 03:44:38,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2665986.6666666665, ans=0.125
2023-11-24 03:44:38,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2665986.6666666665, ans=0.05
2023-11-24 03:44:42,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2665986.6666666665, ans=0.1
2023-11-24 03:45:00,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2666120.0, ans=0.125
2023-11-24 03:45:06,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2666120.0, ans=0.0
2023-11-24 03:45:11,787 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:45:23,800 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.765e+01 9.339e+01 1.018e+02 1.384e+02, threshold=1.868e+02, percent-clipped=0.0
2023-11-24 03:45:26,226 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3150, loss[loss=0.04952, simple_loss=0.05454, pruned_loss=0.008713, audio_tagging_loss=0.01354, over 14675.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09291, pruned_loss=0.01404, audio_tagging_loss=0.009082, over 3048096.24 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:45:38,328 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 399950
2023-11-24 03:45:50,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2666386.6666666665, ans=0.0
2023-11-24 03:46:05,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2666453.3333333335, ans=0.125
2023-11-24 03:46:08,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2666453.3333333335, ans=0.95
2023-11-24 03:46:10,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2666453.3333333335, ans=0.125
2023-11-24 03:46:24,077 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.91 vs. limit=6.0
2023-11-24 03:46:26,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2666520.0, ans=0.125
2023-11-24 03:46:27,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.40 vs. limit=15.0
2023-11-24 03:46:28,852 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3200, loss[loss=0.08043, simple_loss=0.09936, pruned_loss=0.0209, audio_tagging_loss=0.009845, over 14322.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09306, pruned_loss=0.01401, audio_tagging_loss=0.009156, over 3050943.31 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:46:30,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.61 vs. limit=15.0
2023-11-24 03:46:31,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2666586.6666666665, ans=0.0
2023-11-24 03:46:41,325 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400000
2023-11-24 03:46:53,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2666653.3333333335, ans=0.0
2023-11-24 03:46:56,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2666720.0, ans=0.125
2023-11-24 03:47:31,741 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.504e+01 9.216e+01 9.938e+01 1.136e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 03:47:34,775 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3250, loss[loss=0.06894, simple_loss=0.09488, pruned_loss=0.01455, audio_tagging_loss=0.006946, over 14341.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09236, pruned_loss=0.01392, audio_tagging_loss=0.009274, over 3047917.50 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:47:42,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2666920.0, ans=0.125
2023-11-24 03:47:47,461 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400050
2023-11-24 03:47:47,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2666986.6666666665, ans=0.0
2023-11-24 03:47:51,225 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:48:18,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2667120.0, ans=0.1
2023-11-24 03:48:23,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2667186.6666666665, ans=0.125
2023-11-24 03:48:37,195 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3300, loss[loss=0.06861, simple_loss=0.08533, pruned_loss=0.01759, audio_tagging_loss=0.008358, over 15694.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09231, pruned_loss=0.01382, audio_tagging_loss=0.009195, over 3048036.24 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:48:49,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400100
2023-11-24 03:48:55,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2667320.0, ans=0.125
2023-11-24 03:49:02,306 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.06 vs. limit=12.0
2023-11-24 03:49:04,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.36 vs. limit=6.0
2023-11-24 03:49:06,376 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.54 vs. limit=15.0
2023-11-24 03:49:35,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2667520.0, ans=0.0
2023-11-24 03:49:37,335 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.541e+01 9.179e+01 9.840e+01 1.429e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 03:49:38,555 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3350, loss[loss=0.0768, simple_loss=0.1108, pruned_loss=0.0121, audio_tagging_loss=0.009281, over 15453.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09201, pruned_loss=0.01381, audio_tagging_loss=0.009208, over 3041130.91 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:49:41,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2667586.6666666665, ans=0.95
2023-11-24 03:49:42,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2667586.6666666665, ans=0.125
2023-11-24 03:49:51,816 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400150
2023-11-24 03:49:58,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2667653.3333333335, ans=0.07
2023-11-24 03:50:06,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2667720.0, ans=0.1
2023-11-24 03:50:12,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2667720.0, ans=0.2
2023-11-24 03:50:15,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2667786.6666666665, ans=0.125
2023-11-24 03:50:19,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2667786.6666666665, ans=0.1
2023-11-24 03:50:21,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2667786.6666666665, ans=0.0
2023-11-24 03:50:22,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2667786.6666666665, ans=0.2
2023-11-24 03:50:25,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2667786.6666666665, ans=0.05
2023-11-24 03:50:36,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2667853.3333333335, ans=0.125
2023-11-24 03:50:41,063 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.25 vs. limit=15.0
2023-11-24 03:50:41,764 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3400, loss[loss=0.08357, simple_loss=0.1189, pruned_loss=0.01789, audio_tagging_loss=0.006248, over 15520.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09178, pruned_loss=0.01365, audio_tagging_loss=0.009098, over 3034077.51 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:50:51,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2667920.0, ans=0.125
2023-11-24 03:50:54,710 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400200
2023-11-24 03:51:04,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2667986.6666666665, ans=0.125
2023-11-24 03:51:04,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.15 vs. limit=15.0
2023-11-24 03:51:12,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2668053.3333333335, ans=0.0
2023-11-24 03:51:12,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2668053.3333333335, ans=0.0
2023-11-24 03:51:23,738 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.30 vs. limit=15.0
2023-11-24 03:51:43,872 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.468e+01 9.078e+01 9.580e+01 1.134e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 03:51:45,128 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3450, loss[loss=0.09716, simple_loss=0.1302, pruned_loss=0.02326, audio_tagging_loss=0.008792, over 16014.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09214, pruned_loss=0.01358, audio_tagging_loss=0.009002, over 3035943.88 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:51:52,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2668253.3333333335, ans=0.125
2023-11-24 03:51:57,033 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400250
2023-11-24 03:51:57,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2668320.0, ans=0.0
2023-11-24 03:51:59,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2668320.0, ans=0.04949747468305833
2023-11-24 03:52:03,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2668320.0, ans=0.125
2023-11-24 03:52:08,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.30 vs. limit=22.5
2023-11-24 03:52:25,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.15 vs. limit=22.5
2023-11-24 03:52:37,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2668520.0, ans=0.125
2023-11-24 03:52:41,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2668520.0, ans=0.125
2023-11-24 03:52:47,003 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3500, loss[loss=0.05854, simple_loss=0.06429, pruned_loss=0.01313, audio_tagging_loss=0.01327, over 14834.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.0923, pruned_loss=0.01367, audio_tagging_loss=0.00894, over 3037971.13 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:52:59,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400300
2023-11-24 03:53:04,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2668653.3333333335, ans=0.0
2023-11-24 03:53:18,506 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:53:31,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2668786.6666666665, ans=0.1
2023-11-24 03:53:41,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2668853.3333333335, ans=0.0
2023-11-24 03:53:47,627 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.784e+01 9.365e+01 1.022e+02 1.307e+02, threshold=1.873e+02, percent-clipped=0.0
2023-11-24 03:53:48,880 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3550, loss[loss=0.06007, simple_loss=0.08336, pruned_loss=0.009732, audio_tagging_loss=0.008659, over 15809.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09296, pruned_loss=0.01385, audio_tagging_loss=0.008845, over 3043819.92 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:54:00,411 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:54:02,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400350
2023-11-24 03:54:25,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2669120.0, ans=0.125
2023-11-24 03:54:42,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2669186.6666666665, ans=0.125
2023-11-24 03:54:52,807 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3600, loss[loss=0.07709, simple_loss=0.1107, pruned_loss=0.01612, audio_tagging_loss=0.005639, over 15364.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09184, pruned_loss=0.0135, audio_tagging_loss=0.008805, over 3044425.07 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:55:04,825 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400400
2023-11-24 03:55:20,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.65 vs. limit=6.0
2023-11-24 03:55:26,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2669386.6666666665, ans=0.125
2023-11-24 03:55:27,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2669386.6666666665, ans=0.0
2023-11-24 03:55:38,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2669453.3333333335, ans=0.125
2023-11-24 03:55:53,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2669586.6666666665, ans=0.07
2023-11-24 03:55:54,320 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.332e+01 8.867e+01 9.623e+01 1.551e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-24 03:55:54,363 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3650, loss[loss=0.08618, simple_loss=0.112, pruned_loss=0.02113, audio_tagging_loss=0.009042, over 15519.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09183, pruned_loss=0.01348, audio_tagging_loss=0.008819, over 3038340.61 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:56:06,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400450
2023-11-24 03:56:19,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2669720.0, ans=0.1
2023-11-24 03:56:36,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2669786.6666666665, ans=0.125
2023-11-24 03:56:56,224 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3700, loss[loss=0.05659, simple_loss=0.07988, pruned_loss=0.00824, audio_tagging_loss=0.008403, over 15538.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09242, pruned_loss=0.01355, audio_tagging_loss=0.008679, over 3047028.54 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:57:07,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2669920.0, ans=0.125
2023-11-24 03:57:10,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400500
2023-11-24 03:57:22,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2670053.3333333335, ans=0.1
2023-11-24 03:57:30,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2670053.3333333335, ans=0.0
2023-11-24 03:57:38,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2670120.0, ans=0.125
2023-11-24 03:57:57,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2670186.6666666665, ans=0.125
2023-11-24 03:58:00,089 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.459e+01 9.233e+01 9.964e+01 1.289e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 03:58:00,131 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3750, loss[loss=0.06789, simple_loss=0.09382, pruned_loss=0.01377, audio_tagging_loss=0.007209, over 15561.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09239, pruned_loss=0.01361, audio_tagging_loss=0.008713, over 3053836.57 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:58:02,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2670253.3333333335, ans=0.125
2023-11-24 03:58:03,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2670253.3333333335, ans=0.0
2023-11-24 03:58:07,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2670253.3333333335, ans=0.0
2023-11-24 03:58:11,984 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400550
2023-11-24 03:58:15,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2670320.0, ans=0.1
2023-11-24 03:58:41,607 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:58:44,918 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:59:01,180 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3800, loss[loss=0.07157, simple_loss=0.1011, pruned_loss=0.01374, audio_tagging_loss=0.007298, over 14049.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09155, pruned_loss=0.01351, audio_tagging_loss=0.008837, over 3045581.38 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:59:01,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2670586.6666666665, ans=0.015
2023-11-24 03:59:02,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2670586.6666666665, ans=10.0
2023-11-24 03:59:08,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2670586.6666666665, ans=0.125
2023-11-24 03:59:11,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2670586.6666666665, ans=0.125
2023-11-24 03:59:13,368 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400600
2023-11-24 03:59:22,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2670653.3333333335, ans=0.5
2023-11-24 03:59:37,376 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:59:38,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.66 vs. limit=6.0
2023-11-24 03:59:40,817 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:59:57,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2670853.3333333335, ans=0.125
2023-11-24 04:00:03,047 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.597e+01 8.605e+01 9.126e+01 9.740e+01 1.269e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 04:00:03,090 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3850, loss[loss=0.08069, simple_loss=0.1092, pruned_loss=0.01694, audio_tagging_loss=0.009135, over 14821.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.0919, pruned_loss=0.01352, audio_tagging_loss=0.008877, over 3045542.59 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:00:12,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2670920.0, ans=0.0
2023-11-24 04:00:16,292 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400650
2023-11-24 04:00:16,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2670986.6666666665, ans=0.0
2023-11-24 04:00:26,160 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.45 vs. limit=15.0
2023-11-24 04:00:27,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2670986.6666666665, ans=0.2
2023-11-24 04:00:34,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2671053.3333333335, ans=0.2
2023-11-24 04:00:46,182 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0
2023-11-24 04:00:55,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2671186.6666666665, ans=0.0
2023-11-24 04:01:06,126 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3900, loss[loss=0.05098, simple_loss=0.0635, pruned_loss=0.007495, audio_tagging_loss=0.01174, over 15036.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09139, pruned_loss=0.0133, audio_tagging_loss=0.008983, over 3043623.60 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:01:18,914 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400700
2023-11-24 04:01:19,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2671320.0, ans=0.1
2023-11-24 04:01:20,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.51 vs. limit=15.0
2023-11-24 04:01:28,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.73 vs. limit=15.0
2023-11-24 04:02:08,914 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.467e+01 9.035e+01 9.770e+01 1.225e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 04:02:08,958 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 3950, loss[loss=0.06375, simple_loss=0.08296, pruned_loss=0.01276, audio_tagging_loss=0.009509, over 14564.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09129, pruned_loss=0.01336, audio_tagging_loss=0.009106, over 3047042.77 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:02:20,939 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400750
2023-11-24 04:02:23,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2671653.3333333335, ans=0.125
2023-11-24 04:02:37,577 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:03:00,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2671853.3333333335, ans=0.0
2023-11-24 04:03:10,949 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4000, loss[loss=0.06106, simple_loss=0.07933, pruned_loss=0.01263, audio_tagging_loss=0.008757, over 15395.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09141, pruned_loss=0.01338, audio_tagging_loss=0.009176, over 3055895.82 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:03:24,121 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400800
2023-11-24 04:03:55,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2672120.0, ans=0.2
2023-11-24 04:03:59,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.85 vs. limit=15.0
2023-11-24 04:04:11,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2672186.6666666665, ans=0.1
2023-11-24 04:04:13,886 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.813e+01 8.478e+01 9.170e+01 1.003e+02 1.153e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 04:04:13,931 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4050, loss[loss=0.06979, simple_loss=0.09893, pruned_loss=0.013, audio_tagging_loss=0.007319, over 15203.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09013, pruned_loss=0.01316, audio_tagging_loss=0.009203, over 3050984.70 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:04:16,810 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:04:17,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2672253.3333333335, ans=0.0
2023-11-24 04:04:26,451 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400850
2023-11-24 04:04:28,073 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.68 vs. limit=22.5
2023-11-24 04:04:28,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.73 vs. limit=15.0
2023-11-24 04:04:40,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2672386.6666666665, ans=0.125
2023-11-24 04:04:43,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2672386.6666666665, ans=0.125
2023-11-24 04:05:15,963 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4100, loss[loss=0.04193, simple_loss=0.05716, pruned_loss=0.006316, audio_tagging_loss=0.007034, over 14705.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09107, pruned_loss=0.01323, audio_tagging_loss=0.009159, over 3052378.80 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:05:22,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2672586.6666666665, ans=0.125
2023-11-24 04:05:28,555 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400900
2023-11-24 04:05:31,411 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.87 vs. limit=15.0
2023-11-24 04:05:50,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.20 vs. limit=15.0
2023-11-24 04:06:05,071 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5
2023-11-24 04:06:17,402 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.38 vs. limit=15.0
2023-11-24 04:06:18,022 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4150, loss[loss=0.05866, simple_loss=0.08301, pruned_loss=0.009015, audio_tagging_loss=0.008141, over 15669.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09084, pruned_loss=0.01331, audio_tagging_loss=0.009056, over 3049212.30 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:06:19,137 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.718e+01 9.320e+01 1.030e+02 1.270e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 04:06:31,268 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 400950
2023-11-24 04:06:32,670 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:06:38,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2672986.6666666665, ans=0.125
2023-11-24 04:06:46,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2673053.3333333335, ans=0.0
2023-11-24 04:06:56,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2673120.0, ans=0.125
2023-11-24 04:07:02,441 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:07:06,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2673120.0, ans=0.125
2023-11-24 04:07:11,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2673186.6666666665, ans=0.125
2023-11-24 04:07:17,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2673186.6666666665, ans=0.2
2023-11-24 04:07:21,681 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4200, loss[loss=0.07233, simple_loss=0.09176, pruned_loss=0.01719, audio_tagging_loss=0.009251, over 14371.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.0913, pruned_loss=0.01345, audio_tagging_loss=0.008882, over 3042734.60 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:07:21,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2673253.3333333335, ans=0.125
2023-11-24 04:07:34,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401000
2023-11-24 04:07:40,572 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:07:40,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2673320.0, ans=0.125
2023-11-24 04:08:11,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2673520.0, ans=0.125
2023-11-24 04:08:11,235 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.27 vs. limit=6.0
2023-11-24 04:08:24,420 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4250, loss[loss=0.05887, simple_loss=0.07313, pruned_loss=0.01143, audio_tagging_loss=0.01088, over 16585.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09024, pruned_loss=0.01318, audio_tagging_loss=0.00887, over 3050247.97 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:08:25,551 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.323e+01 9.029e+01 9.665e+01 1.222e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 04:08:35,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2673653.3333333335, ans=0.0
2023-11-24 04:08:36,840 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401050
2023-11-24 04:08:41,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2673653.3333333335, ans=0.125
2023-11-24 04:08:57,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.01 vs. limit=10.0
2023-11-24 04:09:14,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2673853.3333333335, ans=0.2
2023-11-24 04:09:24,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2673853.3333333335, ans=0.125
2023-11-24 04:09:26,475 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4300, loss[loss=0.06962, simple_loss=0.09627, pruned_loss=0.01174, audio_tagging_loss=0.009744, over 15107.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09074, pruned_loss=0.01331, audio_tagging_loss=0.008865, over 3041993.62 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:09:29,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2673920.0, ans=0.125
2023-11-24 04:09:38,411 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401100
2023-11-24 04:10:00,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2674053.3333333335, ans=0.125
2023-11-24 04:10:27,631 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.00 vs. limit=6.0
2023-11-24 04:10:28,549 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4350, loss[loss=0.05045, simple_loss=0.06394, pruned_loss=0.008969, audio_tagging_loss=0.00951, over 16162.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09072, pruned_loss=0.01328, audio_tagging_loss=0.008908, over 3036547.21 frames. ], batch size: 62, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:10:28,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2674253.3333333335, ans=0.0
2023-11-24 04:10:30,158 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.375e+01 8.839e+01 9.580e+01 1.285e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-24 04:10:41,771 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401150
2023-11-24 04:10:45,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2674320.0, ans=0.0
2023-11-24 04:10:46,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2674320.0, ans=0.125
2023-11-24 04:10:57,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2674386.6666666665, ans=0.125
2023-11-24 04:11:04,795 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:11:27,880 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:11:30,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2674586.6666666665, ans=15.0
2023-11-24 04:11:31,216 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4400, loss[loss=0.0775, simple_loss=0.1124, pruned_loss=0.01393, audio_tagging_loss=0.007367, over 15857.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09146, pruned_loss=0.01337, audio_tagging_loss=0.008962, over 3046431.91 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:11:43,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401200
2023-11-24 04:11:52,714 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.88 vs. limit=15.0
2023-11-24 04:12:02,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2674720.0, ans=0.2
2023-11-24 04:12:02,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2674720.0, ans=0.125
2023-11-24 04:12:18,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2674786.6666666665, ans=0.125
2023-11-24 04:12:29,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2674853.3333333335, ans=0.0
2023-11-24 04:12:32,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2674920.0, ans=0.0
2023-11-24 04:12:33,402 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4450, loss[loss=0.07051, simple_loss=0.09052, pruned_loss=0.01447, audio_tagging_loss=0.01078, over 15188.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09206, pruned_loss=0.01355, audio_tagging_loss=0.00886, over 3042622.34 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:12:34,496 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.511e+01 8.725e+01 9.329e+01 1.003e+02 1.264e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-24 04:12:34,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2674920.0, ans=0.125
2023-11-24 04:12:36,315 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.39 vs. limit=15.0
2023-11-24 04:12:45,905 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401250
2023-11-24 04:13:11,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2675120.0, ans=0.0
2023-11-24 04:13:17,982 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0
2023-11-24 04:13:31,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2675186.6666666665, ans=0.125
2023-11-24 04:13:32,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2675186.6666666665, ans=0.125
2023-11-24 04:13:35,764 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4500, loss[loss=0.07501, simple_loss=0.1014, pruned_loss=0.01474, audio_tagging_loss=0.009573, over 15966.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09236, pruned_loss=0.01355, audio_tagging_loss=0.008836, over 3040876.54 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:13:49,247 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401300
2023-11-24 04:14:00,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.44 vs. limit=12.0
2023-11-24 04:14:03,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2675386.6666666665, ans=0.125
2023-11-24 04:14:22,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.71 vs. limit=22.5
2023-11-24 04:14:23,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2675453.3333333335, ans=0.125
2023-11-24 04:14:38,764 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4550, loss[loss=0.06728, simple_loss=0.08325, pruned_loss=0.01737, audio_tagging_loss=0.008286, over 14905.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09205, pruned_loss=0.01336, audio_tagging_loss=0.008845, over 3048799.45 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:14:39,937 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.652e+01 8.786e+01 9.235e+01 9.923e+01 1.429e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 04:14:45,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.05 vs. limit=10.0
2023-11-24 04:14:50,687 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401350
2023-11-24 04:14:57,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2675653.3333333335, ans=0.125
2023-11-24 04:15:24,795 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:15:40,263 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4600, loss[loss=0.05914, simple_loss=0.07817, pruned_loss=0.01095, audio_tagging_loss=0.009104, over 15003.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.0913, pruned_loss=0.01315, audio_tagging_loss=0.008931, over 3042982.76 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:15:50,352 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.67 vs. limit=15.0
2023-11-24 04:15:52,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401400
2023-11-24 04:16:06,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2676053.3333333335, ans=0.2
2023-11-24 04:16:09,061 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:16:28,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2676120.0, ans=0.125
2023-11-24 04:16:31,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2676186.6666666665, ans=0.1
2023-11-24 04:16:31,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2676186.6666666665, ans=0.125
2023-11-24 04:16:42,187 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4650, loss[loss=0.03824, simple_loss=0.04728, pruned_loss=0.003839, audio_tagging_loss=0.01076, over 15129.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09066, pruned_loss=0.01314, audio_tagging_loss=0.009005, over 3043169.72 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:16:43,858 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.311e+01 8.909e+01 9.483e+01 1.432e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-24 04:16:56,284 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401450
2023-11-24 04:16:59,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2676320.0, ans=0.0
2023-11-24 04:17:03,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2676320.0, ans=0.5
2023-11-24 04:17:04,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2676320.0, ans=0.09899494936611666
2023-11-24 04:17:04,954 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.11 vs. limit=10.0
2023-11-24 04:17:10,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2676386.6666666665, ans=0.1
2023-11-24 04:17:35,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2676520.0, ans=0.07
2023-11-24 04:17:45,188 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4700, loss[loss=0.0563, simple_loss=0.0781, pruned_loss=0.008253, audio_tagging_loss=0.008992, over 14524.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.0903, pruned_loss=0.01313, audio_tagging_loss=0.009192, over 3048292.27 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:17:47,084 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.74 vs. limit=15.0
2023-11-24 04:17:57,232 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401500
2023-11-24 04:18:03,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2676653.3333333335, ans=0.0
2023-11-24 04:18:07,614 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.20 vs. limit=15.0
2023-11-24 04:18:14,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2676720.0, ans=0.0
2023-11-24 04:18:16,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2676720.0, ans=0.125
2023-11-24 04:18:35,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2676853.3333333335, ans=0.125
2023-11-24 04:18:38,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2676853.3333333335, ans=0.1
2023-11-24 04:18:44,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2676853.3333333335, ans=0.0
2023-11-24 04:18:45,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2676920.0, ans=0.2
2023-11-24 04:18:46,880 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4750, loss[loss=0.05985, simple_loss=0.07866, pruned_loss=0.01352, audio_tagging_loss=0.007006, over 14737.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.0894, pruned_loss=0.01301, audio_tagging_loss=0.009265, over 3042745.53 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:18:47,986 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.351e+01 9.004e+01 9.816e+01 1.181e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 04:18:58,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401550
2023-11-24 04:19:05,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2676986.6666666665, ans=0.2
2023-11-24 04:19:13,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2677053.3333333335, ans=0.1
2023-11-24 04:19:19,811 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:19:23,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.19 vs. limit=15.0
2023-11-24 04:19:24,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2677120.0, ans=10.0
2023-11-24 04:19:25,612 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:19:33,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2677120.0, ans=0.125
2023-11-24 04:19:35,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.52 vs. limit=15.0
2023-11-24 04:19:47,771 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4800, loss[loss=0.06416, simple_loss=0.08732, pruned_loss=0.01354, audio_tagging_loss=0.006962, over 14866.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09092, pruned_loss=0.0132, audio_tagging_loss=0.009215, over 3044668.13 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:19:49,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2677253.3333333335, ans=0.125
2023-11-24 04:19:53,120 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.56 vs. limit=22.5
2023-11-24 04:20:01,486 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401600
2023-11-24 04:20:49,784 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.41 vs. limit=12.0
2023-11-24 04:20:52,264 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4850, loss[loss=0.06925, simple_loss=0.08336, pruned_loss=0.01567, audio_tagging_loss=0.0119, over 15161.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09077, pruned_loss=0.01321, audio_tagging_loss=0.009273, over 3046997.73 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:20:53,332 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.256e+01 8.375e+01 9.054e+01 9.821e+01 1.202e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 04:20:54,177 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.47 vs. limit=22.5
2023-11-24 04:21:04,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401650
2023-11-24 04:21:04,180 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:21:19,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2677720.0, ans=0.95
2023-11-24 04:21:20,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2677720.0, ans=0.1
2023-11-24 04:21:20,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2677720.0, ans=0.1
2023-11-24 04:21:23,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2677720.0, ans=0.0
2023-11-24 04:21:29,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2677786.6666666665, ans=0.0
2023-11-24 04:21:53,711 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4900, loss[loss=0.07633, simple_loss=0.1022, pruned_loss=0.01749, audio_tagging_loss=0.007736, over 14613.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09145, pruned_loss=0.01334, audio_tagging_loss=0.009124, over 3052909.75 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:22:05,638 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401700
2023-11-24 04:22:20,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.22 vs. limit=12.0
2023-11-24 04:22:54,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2678253.3333333335, ans=0.125
2023-11-24 04:22:55,414 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 4950, loss[loss=0.05456, simple_loss=0.06661, pruned_loss=0.009981, audio_tagging_loss=0.01128, over 14407.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09061, pruned_loss=0.01326, audio_tagging_loss=0.009066, over 3046738.58 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:22:56,550 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.172e+01 8.560e+01 8.976e+01 9.753e+01 1.389e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-24 04:22:59,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2678253.3333333335, ans=0.125
2023-11-24 04:23:08,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401750
2023-11-24 04:23:35,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2678453.3333333335, ans=0.125
2023-11-24 04:23:57,845 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5000, loss[loss=0.05689, simple_loss=0.07924, pruned_loss=0.009132, audio_tagging_loss=0.008135, over 15217.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09059, pruned_loss=0.01314, audio_tagging_loss=0.008902, over 3045074.42 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:24:02,394 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:24:10,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.97 vs. limit=6.0
2023-11-24 04:24:10,919 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401800
2023-11-24 04:24:22,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2678720.0, ans=0.09899494936611666
2023-11-24 04:24:47,360 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.92 vs. limit=15.0
2023-11-24 04:25:00,940 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5050, loss[loss=0.06996, simple_loss=0.1048, pruned_loss=0.01198, audio_tagging_loss=0.005575, over 14780.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09067, pruned_loss=0.01308, audio_tagging_loss=0.008824, over 3047572.73 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:25:02,046 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.608e+01 9.238e+01 9.938e+01 4.414e+02, threshold=1.848e+02, percent-clipped=1.0
2023-11-24 04:25:04,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2678920.0, ans=0.1
2023-11-24 04:25:05,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2678920.0, ans=0.125
2023-11-24 04:25:12,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401850
2023-11-24 04:25:18,305 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.19 vs. limit=15.0
2023-11-24 04:25:23,097 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0
2023-11-24 04:25:33,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2679053.3333333335, ans=0.125
2023-11-24 04:25:56,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2679186.6666666665, ans=0.2
2023-11-24 04:26:02,694 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5100, loss[loss=0.06018, simple_loss=0.07997, pruned_loss=0.008405, audio_tagging_loss=0.01179, over 15130.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09133, pruned_loss=0.0131, audio_tagging_loss=0.008756, over 3049537.86 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:26:03,229 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.14 vs. limit=15.0
2023-11-24 04:26:04,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2679253.3333333335, ans=0.125
2023-11-24 04:26:16,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401900
2023-11-24 04:26:28,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2679386.6666666665, ans=0.04949747468305833
2023-11-24 04:26:48,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2679453.3333333335, ans=0.0
2023-11-24 04:26:50,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2679453.3333333335, ans=0.125
2023-11-24 04:27:05,522 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5150, loss[loss=0.0882, simple_loss=0.1279, pruned_loss=0.0183, audio_tagging_loss=0.005959, over 15390.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09123, pruned_loss=0.01309, audio_tagging_loss=0.008754, over 3053438.44 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:27:07,943 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.413e+01 8.884e+01 9.698e+01 1.235e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 04:27:18,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 401950
2023-11-24 04:28:08,606 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5200, loss[loss=0.06357, simple_loss=0.08338, pruned_loss=0.01291, audio_tagging_loss=0.008977, over 15818.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09117, pruned_loss=0.01319, audio_tagging_loss=0.008756, over 3046493.85 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:28:14,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2679920.0, ans=0.2
2023-11-24 04:28:20,572 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402000
2023-11-24 04:28:35,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2680053.3333333335, ans=0.0
2023-11-24 04:28:35,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2680053.3333333335, ans=0.125
2023-11-24 04:28:55,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2680120.0, ans=0.125
2023-11-24 04:28:58,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2680186.6666666665, ans=0.0
2023-11-24 04:29:04,272 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:29:09,996 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5250, loss[loss=0.06342, simple_loss=0.09088, pruned_loss=0.008001, audio_tagging_loss=0.009979, over 15480.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09165, pruned_loss=0.01324, audio_tagging_loss=0.008649, over 3048827.76 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:29:12,358 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.228e+01 8.916e+01 9.618e+01 1.278e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-24 04:29:12,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2680253.3333333335, ans=0.125
2023-11-24 04:29:14,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2680253.3333333335, ans=10.0
2023-11-24 04:29:21,907 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402050
2023-11-24 04:29:28,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2680320.0, ans=0.125
2023-11-24 04:29:30,764 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.41 vs. limit=15.0
2023-11-24 04:29:32,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2680320.0, ans=0.2
2023-11-24 04:29:35,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2680386.6666666665, ans=0.2
2023-11-24 04:29:36,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2680386.6666666665, ans=0.125
2023-11-24 04:29:41,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2680386.6666666665, ans=0.125
2023-11-24 04:29:57,031 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.51 vs. limit=15.0
2023-11-24 04:30:09,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.32 vs. limit=15.0
2023-11-24 04:30:12,382 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5300, loss[loss=0.06503, simple_loss=0.08746, pruned_loss=0.01228, audio_tagging_loss=0.009024, over 16165.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09161, pruned_loss=0.01318, audio_tagging_loss=0.008716, over 3050264.86 frames. ], batch size: 62, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:30:16,951 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-24 04:30:18,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2680586.6666666665, ans=0.125
2023-11-24 04:30:23,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0
2023-11-24 04:30:24,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2680653.3333333335, ans=0.2
2023-11-24 04:30:25,270 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402100
2023-11-24 04:30:26,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2.whitening_limit, batch_count=2680653.3333333335, ans=15.0
2023-11-24 04:30:30,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2680653.3333333335, ans=0.125
2023-11-24 04:30:40,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2680720.0, ans=0.1
2023-11-24 04:30:57,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2680786.6666666665, ans=0.1
2023-11-24 04:30:58,585 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0
2023-11-24 04:31:15,249 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5350, loss[loss=0.05704, simple_loss=0.0684, pruned_loss=0.01369, audio_tagging_loss=0.009151, over 15974.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.0925, pruned_loss=0.01333, audio_tagging_loss=0.008676, over 3055512.92 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:31:18,679 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.481e+01 8.724e+01 9.285e+01 9.851e+01 1.330e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 04:31:27,810 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402150
2023-11-24 04:31:38,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2681053.3333333335, ans=0.125
2023-11-24 04:31:44,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2681053.3333333335, ans=0.125
2023-11-24 04:31:52,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2681120.0, ans=0.125
2023-11-24 04:32:16,964 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5400, loss[loss=0.05592, simple_loss=0.07618, pruned_loss=0.009057, audio_tagging_loss=0.008769, over 14778.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.0916, pruned_loss=0.01321, audio_tagging_loss=0.008796, over 3051867.57 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:32:26,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2681253.3333333335, ans=0.5
2023-11-24 04:32:28,756 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402200
2023-11-24 04:32:36,612 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.77 vs. limit=15.0
2023-11-24 04:32:58,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2681453.3333333335, ans=22.5
2023-11-24 04:33:16,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2681520.0, ans=0.0
2023-11-24 04:33:18,589 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5450, loss[loss=0.06492, simple_loss=0.092, pruned_loss=0.008794, audio_tagging_loss=0.01013, over 14931.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09114, pruned_loss=0.01307, audio_tagging_loss=0.00881, over 3049875.02 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 8.0
2023-11-24 04:33:24,395 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.363e+01 9.005e+01 9.739e+01 1.241e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 04:33:28,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2681586.6666666665, ans=0.0
2023-11-24 04:33:31,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402250
2023-11-24 04:33:35,935 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:33:40,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2681653.3333333335, ans=0.125
2023-11-24 04:33:45,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2681720.0, ans=0.125
2023-11-24 04:34:00,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2681786.6666666665, ans=0.125
2023-11-24 04:34:21,297 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5500, loss[loss=0.07405, simple_loss=0.09854, pruned_loss=0.01743, audio_tagging_loss=0.007349, over 15258.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09212, pruned_loss=0.01341, audio_tagging_loss=0.00889, over 3055420.43 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 8.0
2023-11-24 04:34:33,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402300
2023-11-24 04:34:54,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2682053.3333333335, ans=0.05
2023-11-24 04:34:57,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2682120.0, ans=0.0
2023-11-24 04:35:05,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2682120.0, ans=0.125
2023-11-24 04:35:22,495 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5550, loss[loss=0.05158, simple_loss=0.06725, pruned_loss=0.006553, audio_tagging_loss=0.0114, over 15928.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09148, pruned_loss=0.01333, audio_tagging_loss=0.008934, over 3054392.63 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 8.0
2023-11-24 04:35:27,156 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.398e+01 8.518e+01 9.178e+01 1.002e+02 1.571e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 04:35:34,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402350
2023-11-24 04:35:39,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2682320.0, ans=0.125
2023-11-24 04:36:00,278 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:36:24,093 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5600, loss[loss=0.06015, simple_loss=0.08595, pruned_loss=0.009101, audio_tagging_loss=0.008071, over 15037.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09117, pruned_loss=0.01318, audio_tagging_loss=0.009104, over 3052386.22 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:36:28,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2682586.6666666665, ans=0.1
2023-11-24 04:36:28,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2682586.6666666665, ans=0.0
2023-11-24 04:36:36,746 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.98 vs. limit=22.5
2023-11-24 04:36:37,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402400
2023-11-24 04:36:43,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2682653.3333333335, ans=0.0
2023-11-24 04:36:56,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2682720.0, ans=0.07
2023-11-24 04:37:08,385 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:37:08,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2682786.6666666665, ans=0.0
2023-11-24 04:37:18,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2682853.3333333335, ans=0.125
2023-11-24 04:37:27,404 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5650, loss[loss=0.07034, simple_loss=0.1009, pruned_loss=0.01085, audio_tagging_loss=0.009049, over 15391.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.08941, pruned_loss=0.01298, audio_tagging_loss=0.009218, over 3047320.34 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:37:33,011 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.272e+01 8.812e+01 9.562e+01 1.491e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-24 04:37:34,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2682920.0, ans=0.0
2023-11-24 04:37:35,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2682920.0, ans=0.2
2023-11-24 04:37:40,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402450
2023-11-24 04:37:45,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten.whitening_limit, batch_count=2682986.6666666665, ans=22.5
2023-11-24 04:37:49,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.whiten.whitening_limit, batch_count=2682986.6666666665, ans=12.0
2023-11-24 04:37:51,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2683053.3333333335, ans=0.0
2023-11-24 04:38:01,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2683053.3333333335, ans=0.2
2023-11-24 04:38:09,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.16 vs. limit=15.0
2023-11-24 04:38:10,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2683120.0, ans=0.0
2023-11-24 04:38:20,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2683186.6666666665, ans=0.0
2023-11-24 04:38:20,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2683186.6666666665, ans=0.125
2023-11-24 04:38:22,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2683186.6666666665, ans=0.125
2023-11-24 04:38:24,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2683186.6666666665, ans=0.125
2023-11-24 04:38:31,390 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5700, loss[loss=0.06291, simple_loss=0.08589, pruned_loss=0.01044, audio_tagging_loss=0.009533, over 15059.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.0896, pruned_loss=0.013, audio_tagging_loss=0.009222, over 3047551.21 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:38:41,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2683253.3333333335, ans=0.125
2023-11-24 04:38:43,615 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402500
2023-11-24 04:38:46,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2683320.0, ans=0.07
2023-11-24 04:39:02,235 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2683386.6666666665, ans=0.125
2023-11-24 04:39:03,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2683386.6666666665, ans=0.1
2023-11-24 04:39:13,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2683453.3333333335, ans=0.125
2023-11-24 04:39:23,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2683520.0, ans=0.2
2023-11-24 04:39:33,796 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5750, loss[loss=0.08981, simple_loss=0.1211, pruned_loss=0.02124, audio_tagging_loss=0.008023, over 15218.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.08959, pruned_loss=0.01302, audio_tagging_loss=0.009146, over 3038621.80 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:39:39,189 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.554e+01 9.219e+01 1.000e+02 1.500e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 04:39:47,576 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402550
2023-11-24 04:39:50,495 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.76 vs. limit=15.0
2023-11-24 04:40:07,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2683720.0, ans=0.2
2023-11-24 04:40:08,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2683720.0, ans=0.2
2023-11-24 04:40:09,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2683720.0, ans=0.125
2023-11-24 04:40:10,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2683786.6666666665, ans=0.0
2023-11-24 04:40:13,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2683786.6666666665, ans=0.0
2023-11-24 04:40:37,107 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5800, loss[loss=0.05737, simple_loss=0.06609, pruned_loss=0.009525, audio_tagging_loss=0.0148, over 16426.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08903, pruned_loss=0.01292, audio_tagging_loss=0.009006, over 3036411.41 frames. ], batch size: 64, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:40:40,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2683920.0, ans=0.1
2023-11-24 04:40:40,606 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0
2023-11-24 04:40:41,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2683920.0, ans=0.125
2023-11-24 04:40:44,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.37 vs. limit=22.5
2023-11-24 04:40:46,544 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.04 vs. limit=22.5
2023-11-24 04:40:47,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2683920.0, ans=10.0
2023-11-24 04:40:49,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402600
2023-11-24 04:40:54,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2683986.6666666665, ans=0.0
2023-11-24 04:40:56,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.79 vs. limit=6.0
2023-11-24 04:40:58,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2683986.6666666665, ans=0.0
2023-11-24 04:41:11,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2684053.3333333335, ans=0.07
2023-11-24 04:41:11,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.27 vs. limit=6.0
2023-11-24 04:41:17,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2684120.0, ans=0.125
2023-11-24 04:41:26,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2684186.6666666665, ans=0.07
2023-11-24 04:41:39,373 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5850, loss[loss=0.07522, simple_loss=0.09952, pruned_loss=0.01635, audio_tagging_loss=0.009114, over 14813.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09024, pruned_loss=0.0131, audio_tagging_loss=0.008832, over 3035115.89 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:41:44,139 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 8.321e+01 9.009e+01 9.790e+01 1.176e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 04:41:47,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2684253.3333333335, ans=0.1
2023-11-24 04:41:49,314 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=15.0
2023-11-24 04:41:51,396 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402650
2023-11-24 04:42:16,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2684453.3333333335, ans=0.1
2023-11-24 04:42:18,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2684453.3333333335, ans=0.125
2023-11-24 04:42:39,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2684520.0, ans=0.025
2023-11-24 04:42:40,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2684586.6666666665, ans=0.2
2023-11-24 04:42:41,223 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5900, loss[loss=0.09569, simple_loss=0.1246, pruned_loss=0.02516, audio_tagging_loss=0.008215, over 14931.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09002, pruned_loss=0.01306, audio_tagging_loss=0.008869, over 3036069.51 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:42:42,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2684586.6666666665, ans=0.125
2023-11-24 04:42:54,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402700
2023-11-24 04:42:54,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2684653.3333333335, ans=0.125
2023-11-24 04:43:42,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2684853.3333333335, ans=0.2
2023-11-24 04:43:44,370 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 5950, loss[loss=0.07618, simple_loss=0.1005, pruned_loss=0.01961, audio_tagging_loss=0.006309, over 14932.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09005, pruned_loss=0.01314, audio_tagging_loss=0.008865, over 3043949.31 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:43:47,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2684920.0, ans=0.125
2023-11-24 04:43:49,681 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.371e+01 8.328e+01 9.136e+01 9.988e+01 1.334e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 04:43:50,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2684920.0, ans=0.1
2023-11-24 04:43:52,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2684920.0, ans=0.0
2023-11-24 04:43:56,994 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402750
2023-11-24 04:43:57,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2684986.6666666665, ans=0.125
2023-11-24 04:44:07,136 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.66 vs. limit=15.0
2023-11-24 04:44:15,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=2685053.3333333335, ans=0.2
2023-11-24 04:44:45,806 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6000, loss[loss=0.05262, simple_loss=0.07596, pruned_loss=0.007783, audio_tagging_loss=0.006859, over 15403.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.08997, pruned_loss=0.01306, audio_tagging_loss=0.008896, over 3036757.38 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:44:45,806 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 04:45:26,691 INFO [train_asr.py:1253] (2/4) Epoch 34, validation: loss=0.05772, simple_loss=0.05082, pruned_loss=0.005023, audio_tagging_loss=0.02728, over 4681554.00 frames. 
2023-11-24 04:45:26,692 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 04:45:30,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2685253.3333333335, ans=0.0
2023-11-24 04:45:40,293 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402800
2023-11-24 04:45:41,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2685320.0, ans=0.125
2023-11-24 04:45:53,339 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.19 vs. limit=15.0
2023-11-24 04:46:07,445 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:46:11,972 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:46:29,988 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6050, loss[loss=0.05552, simple_loss=0.06685, pruned_loss=0.008552, audio_tagging_loss=0.01355, over 15528.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09105, pruned_loss=0.01323, audio_tagging_loss=0.00887, over 3040284.15 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:46:34,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2685586.6666666665, ans=0.2
2023-11-24 04:46:34,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2685586.6666666665, ans=0.125
2023-11-24 04:46:35,237 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.324e+01 9.101e+01 9.713e+01 1.173e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 04:46:42,476 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402850
2023-11-24 04:46:48,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2685653.3333333335, ans=0.0
2023-11-24 04:46:48,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.50 vs. limit=22.5
2023-11-24 04:46:55,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2685720.0, ans=0.0
2023-11-24 04:47:07,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2685786.6666666665, ans=0.125
2023-11-24 04:47:15,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2685786.6666666665, ans=0.0
2023-11-24 04:47:31,916 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6100, loss[loss=0.08715, simple_loss=0.1245, pruned_loss=0.0188, audio_tagging_loss=0.006113, over 16312.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09185, pruned_loss=0.01344, audio_tagging_loss=0.008907, over 3050695.42 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:47:43,830 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402900
2023-11-24 04:48:01,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2686053.3333333335, ans=0.0
2023-11-24 04:48:33,316 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6150, loss[loss=0.08969, simple_loss=0.1296, pruned_loss=0.01865, audio_tagging_loss=0.006235, over 14384.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09178, pruned_loss=0.01351, audio_tagging_loss=0.009009, over 3045758.83 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:48:38,028 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.476e+01 9.051e+01 9.662e+01 1.133e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 04:48:41,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2686253.3333333335, ans=0.125
2023-11-24 04:48:46,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 402950
2023-11-24 04:48:52,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2686320.0, ans=0.0
2023-11-24 04:48:57,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2686320.0, ans=0.1
2023-11-24 04:49:15,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2686453.3333333335, ans=0.125
2023-11-24 04:49:28,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2686520.0, ans=0.125
2023-11-24 04:49:29,070 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=15.0
2023-11-24 04:49:36,247 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6200, loss[loss=0.04746, simple_loss=0.05943, pruned_loss=0.009912, audio_tagging_loss=0.007833, over 14902.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09132, pruned_loss=0.01333, audio_tagging_loss=0.009116, over 3049154.32 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:49:49,204 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403000
2023-11-24 04:50:13,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2686786.6666666665, ans=0.125
2023-11-24 04:50:16,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2686786.6666666665, ans=0.125
2023-11-24 04:50:17,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2686786.6666666665, ans=0.125
2023-11-24 04:50:24,648 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.44 vs. limit=15.0
2023-11-24 04:50:39,665 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6250, loss[loss=0.05615, simple_loss=0.07352, pruned_loss=0.009267, audio_tagging_loss=0.01013, over 15837.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09076, pruned_loss=0.01306, audio_tagging_loss=0.009141, over 3047854.15 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:50:39,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2686920.0, ans=0.0
2023-11-24 04:50:45,709 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.512e+01 9.019e+01 9.887e+01 1.410e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 04:50:49,563 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2686920.0, ans=0.0
2023-11-24 04:50:50,113 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.39 vs. limit=15.0
2023-11-24 04:50:50,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2686986.6666666665, ans=0.125
2023-11-24 04:50:51,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403050
2023-11-24 04:51:04,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2687053.3333333335, ans=0.125
2023-11-24 04:51:09,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2687053.3333333335, ans=0.2
2023-11-24 04:51:09,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2687053.3333333335, ans=0.0
2023-11-24 04:51:32,297 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0
2023-11-24 04:51:33,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2687186.6666666665, ans=0.125
2023-11-24 04:51:36,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2687186.6666666665, ans=0.0
2023-11-24 04:51:38,149 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.88 vs. limit=22.5
2023-11-24 04:51:41,176 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6300, loss[loss=0.07943, simple_loss=0.1092, pruned_loss=0.01778, audio_tagging_loss=0.007036, over 14941.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09001, pruned_loss=0.01292, audio_tagging_loss=0.009161, over 3045579.01 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:51:41,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2687253.3333333335, ans=0.0
2023-11-24 04:51:43,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2687253.3333333335, ans=0.0
2023-11-24 04:51:44,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2687253.3333333335, ans=0.1
2023-11-24 04:51:45,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2687253.3333333335, ans=0.1
2023-11-24 04:51:46,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2687253.3333333335, ans=0.0
2023-11-24 04:51:53,696 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403100
2023-11-24 04:52:21,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2687453.3333333335, ans=0.125
2023-11-24 04:52:28,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2687453.3333333335, ans=0.125
2023-11-24 04:52:38,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2687520.0, ans=0.125
2023-11-24 04:52:40,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2687520.0, ans=0.125
2023-11-24 04:52:43,757 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6350, loss[loss=0.06731, simple_loss=0.09252, pruned_loss=0.01201, audio_tagging_loss=0.009031, over 15693.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.0897, pruned_loss=0.01285, audio_tagging_loss=0.009159, over 3042057.18 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:52:44,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2687586.6666666665, ans=0.1
2023-11-24 04:52:46,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.12 vs. limit=15.0
2023-11-24 04:52:50,171 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.641e+01 8.312e+01 8.895e+01 9.882e+01 1.344e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-24 04:52:56,170 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403150
2023-11-24 04:53:31,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2687853.3333333335, ans=0.1
2023-11-24 04:53:46,085 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6400, loss[loss=0.07144, simple_loss=0.09003, pruned_loss=0.01759, audio_tagging_loss=0.008838, over 15549.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.08976, pruned_loss=0.01291, audio_tagging_loss=0.009281, over 3050949.51 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:53:53,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2687920.0, ans=0.1
2023-11-24 04:53:57,938 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403200
2023-11-24 04:54:16,632 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:54:45,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2688186.6666666665, ans=0.2
2023-11-24 04:54:47,656 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6450, loss[loss=0.09242, simple_loss=0.1319, pruned_loss=0.01781, audio_tagging_loss=0.008687, over 15163.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09003, pruned_loss=0.01302, audio_tagging_loss=0.009351, over 3050694.23 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:54:53,476 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.209e+01 8.936e+01 9.468e+01 2.072e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-24 04:55:00,080 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403250
2023-11-24 04:55:05,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2688320.0, ans=0.025
2023-11-24 04:55:05,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2688320.0, ans=0.2
2023-11-24 04:55:10,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2688320.0, ans=0.125
2023-11-24 04:55:13,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2688386.6666666665, ans=0.0
2023-11-24 04:55:21,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2688386.6666666665, ans=0.5
2023-11-24 04:55:43,798 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0
2023-11-24 04:55:49,654 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6500, loss[loss=0.05702, simple_loss=0.07783, pruned_loss=0.008248, audio_tagging_loss=0.009857, over 15689.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.0909, pruned_loss=0.01307, audio_tagging_loss=0.009292, over 3054232.07 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:55:55,403 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.34 vs. limit=12.0
2023-11-24 04:56:02,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403300
2023-11-24 04:56:10,488 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.64 vs. limit=6.0
2023-11-24 04:56:13,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.22 vs. limit=15.0
2023-11-24 04:56:52,702 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6550, loss[loss=0.07025, simple_loss=0.09859, pruned_loss=0.01288, audio_tagging_loss=0.008075, over 14916.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09133, pruned_loss=0.01316, audio_tagging_loss=0.009102, over 3062104.27 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:56:59,700 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.455e+01 8.633e+01 9.315e+01 9.907e+01 1.273e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 04:57:05,007 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403350
2023-11-24 04:57:34,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2689120.0, ans=0.125
2023-11-24 04:57:38,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2689120.0, ans=0.125
2023-11-24 04:57:54,963 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6600, loss[loss=0.06469, simple_loss=0.08629, pruned_loss=0.01072, audio_tagging_loss=0.01083, over 15361.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09137, pruned_loss=0.01325, audio_tagging_loss=0.008965, over 3057507.99 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:57:56,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2689253.3333333335, ans=0.0
2023-11-24 04:58:06,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403400
2023-11-24 04:58:33,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2689453.3333333335, ans=0.5
2023-11-24 04:58:36,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2689453.3333333335, ans=0.1
2023-11-24 04:58:40,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2689453.3333333335, ans=0.1
2023-11-24 04:58:53,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2689520.0, ans=0.0
2023-11-24 04:58:57,527 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6650, loss[loss=0.09149, simple_loss=0.1158, pruned_loss=0.02555, audio_tagging_loss=0.008023, over 15755.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09111, pruned_loss=0.01315, audio_tagging_loss=0.009001, over 3058226.87 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:59:05,521 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.526e+01 8.351e+01 9.016e+01 9.664e+01 1.302e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 04:59:05,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2689586.6666666665, ans=0.125
2023-11-24 04:59:11,162 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403450
2023-11-24 04:59:17,151 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:59:29,464 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.10 vs. limit=10.0
2023-11-24 04:59:53,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.66 vs. limit=15.0
2023-11-24 05:00:00,592 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6700, loss[loss=0.05496, simple_loss=0.07058, pruned_loss=0.008729, audio_tagging_loss=0.01094, over 15251.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.0906, pruned_loss=0.01314, audio_tagging_loss=0.00898, over 3048402.68 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:00:04,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2689920.0, ans=0.125
2023-11-24 05:00:12,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403500
2023-11-24 05:00:24,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2690053.3333333335, ans=0.1
2023-11-24 05:00:40,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2690120.0, ans=22.5
2023-11-24 05:00:58,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2690186.6666666665, ans=0.0
2023-11-24 05:01:01,980 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6750, loss[loss=0.06798, simple_loss=0.09715, pruned_loss=0.01207, audio_tagging_loss=0.007332, over 14519.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09021, pruned_loss=0.01304, audio_tagging_loss=0.008994, over 3041750.07 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:01:09,698 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.241e+01 8.716e+01 9.523e+01 2.058e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-24 05:01:14,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403550
2023-11-24 05:01:50,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2690520.0, ans=0.125
2023-11-24 05:02:04,018 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6800, loss[loss=0.07367, simple_loss=0.09526, pruned_loss=0.01513, audio_tagging_loss=0.01091, over 15923.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.0898, pruned_loss=0.01298, audio_tagging_loss=0.008949, over 3042488.44 frames. ], batch size: 62, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:02:07,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2690586.6666666665, ans=0.0
2023-11-24 05:02:17,544 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403600
2023-11-24 05:02:24,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2690653.3333333335, ans=0.0
2023-11-24 05:03:05,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2690853.3333333335, ans=0.2
2023-11-24 05:03:07,283 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6850, loss[loss=0.0691, simple_loss=0.0971, pruned_loss=0.0123, audio_tagging_loss=0.008249, over 14194.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.0907, pruned_loss=0.01309, audio_tagging_loss=0.008887, over 3042179.02 frames. ], batch size: 52, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:03:15,552 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.436e+01 8.418e+01 8.943e+01 9.846e+01 1.358e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-24 05:03:19,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403650
2023-11-24 05:03:19,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2690986.6666666665, ans=0.125
2023-11-24 05:03:28,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2690986.6666666665, ans=0.125
2023-11-24 05:03:53,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2691120.0, ans=0.0
2023-11-24 05:03:58,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2691186.6666666665, ans=0.2
2023-11-24 05:04:07,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2691253.3333333335, ans=0.125
2023-11-24 05:04:08,464 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6900, loss[loss=0.05154, simple_loss=0.07388, pruned_loss=0.006091, audio_tagging_loss=0.00851, over 16193.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.08972, pruned_loss=0.01288, audio_tagging_loss=0.008854, over 3037435.67 frames. ], batch size: 63, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:04:14,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2691253.3333333335, ans=0.125
2023-11-24 05:04:17,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2691253.3333333335, ans=0.1
2023-11-24 05:04:20,338 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403700
2023-11-24 05:04:23,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2691320.0, ans=0.2
2023-11-24 05:04:55,801 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:05:10,069 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 6950, loss[loss=0.05668, simple_loss=0.07722, pruned_loss=0.009009, audio_tagging_loss=0.009064, over 14089.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09077, pruned_loss=0.01309, audio_tagging_loss=0.008776, over 3038439.19 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:05:17,207 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:05:18,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2691586.6666666665, ans=0.125
2023-11-24 05:05:19,310 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.551e+01 9.218e+01 9.903e+01 1.445e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 05:05:23,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403750
2023-11-24 05:05:29,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2691653.3333333335, ans=0.5
2023-11-24 05:05:40,468 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.61 vs. limit=22.5
2023-11-24 05:06:13,667 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7000, loss[loss=0.05309, simple_loss=0.06193, pruned_loss=0.01184, audio_tagging_loss=0.01028, over 15447.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09126, pruned_loss=0.01326, audio_tagging_loss=0.008824, over 3035314.13 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:06:25,479 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403800
2023-11-24 05:06:30,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2691986.6666666665, ans=0.125
2023-11-24 05:06:33,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2691986.6666666665, ans=0.2
2023-11-24 05:06:57,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2692120.0, ans=0.1
2023-11-24 05:07:10,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2692186.6666666665, ans=0.1
2023-11-24 05:07:15,357 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7050, loss[loss=0.07571, simple_loss=0.1133, pruned_loss=0.01349, audio_tagging_loss=0.005582, over 14695.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09002, pruned_loss=0.01292, audio_tagging_loss=0.00901, over 3033592.73 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:07:17,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.79 vs. limit=22.5
2023-11-24 05:07:22,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2692253.3333333335, ans=0.2
2023-11-24 05:07:23,606 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.305e+01 8.871e+01 9.954e+01 1.371e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-24 05:07:25,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2692253.3333333335, ans=0.125
2023-11-24 05:07:27,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403850
2023-11-24 05:07:43,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2692386.6666666665, ans=0.0
2023-11-24 05:08:09,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2692520.0, ans=0.2
2023-11-24 05:08:16,594 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7100, loss[loss=0.07892, simple_loss=0.101, pruned_loss=0.017, audio_tagging_loss=0.01142, over 15972.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09005, pruned_loss=0.01299, audio_tagging_loss=0.009021, over 3040609.66 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:08:30,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403900
2023-11-24 05:08:31,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2692653.3333333335, ans=0.2
2023-11-24 05:08:37,241 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:08:51,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2692720.0, ans=0.2
2023-11-24 05:09:07,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2692853.3333333335, ans=0.1
2023-11-24 05:09:20,742 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7150, loss[loss=0.08307, simple_loss=0.1175, pruned_loss=0.01617, audio_tagging_loss=0.008146, over 15362.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09135, pruned_loss=0.01337, audio_tagging_loss=0.009077, over 3046426.21 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:09:28,995 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-24 05:09:29,601 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 8.812e+01 9.268e+01 1.001e+02 1.464e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 05:09:33,368 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 403950
2023-11-24 05:09:36,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.45 vs. limit=15.0
2023-11-24 05:09:52,988 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.92 vs. limit=10.0
2023-11-24 05:10:01,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2693120.0, ans=0.07
2023-11-24 05:10:22,845 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7200, loss[loss=0.07905, simple_loss=0.1085, pruned_loss=0.01925, audio_tagging_loss=0.005549, over 14938.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09097, pruned_loss=0.01321, audio_tagging_loss=0.009134, over 3043982.20 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:10:26,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2693253.3333333335, ans=0.1
2023-11-24 05:10:33,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2693320.0, ans=0.125
2023-11-24 05:10:34,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404000
2023-11-24 05:10:57,301 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.10 vs. limit=15.0
2023-11-24 05:11:01,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2693386.6666666665, ans=0.07
2023-11-24 05:11:18,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2693520.0, ans=0.1
2023-11-24 05:11:20,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2693520.0, ans=0.2
2023-11-24 05:11:22,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2693520.0, ans=0.09899494936611666
2023-11-24 05:11:28,066 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7250, loss[loss=0.06102, simple_loss=0.08299, pruned_loss=0.01154, audio_tagging_loss=0.007977, over 15537.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09082, pruned_loss=0.01326, audio_tagging_loss=0.009307, over 3042690.20 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:11:34,545 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.37 vs. limit=6.0
2023-11-24 05:11:36,162 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.320e+01 8.579e+01 9.139e+01 1.008e+02 1.154e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 05:11:38,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2693586.6666666665, ans=0.125
2023-11-24 05:11:40,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404050
2023-11-24 05:11:58,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2693720.0, ans=0.125
2023-11-24 05:12:23,276 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.26 vs. limit=15.0
2023-11-24 05:12:29,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2693920.0, ans=0.125
2023-11-24 05:12:30,880 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7300, loss[loss=0.04719, simple_loss=0.06021, pruned_loss=0.007975, audio_tagging_loss=0.009113, over 14708.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09215, pruned_loss=0.01354, audio_tagging_loss=0.009122, over 3041781.57 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:12:41,629 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2693920.0, ans=0.125
2023-11-24 05:12:43,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404100
2023-11-24 05:12:51,502 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.86 vs. limit=6.0
2023-11-24 05:13:07,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2694120.0, ans=0.0
2023-11-24 05:13:27,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0
2023-11-24 05:13:30,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2694186.6666666665, ans=0.125
2023-11-24 05:13:32,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=15.0
2023-11-24 05:13:32,966 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7350, loss[loss=0.06848, simple_loss=0.0879, pruned_loss=0.0143, audio_tagging_loss=0.01023, over 15868.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09202, pruned_loss=0.01347, audio_tagging_loss=0.009024, over 3047728.38 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:13:33,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2694253.3333333335, ans=0.125
2023-11-24 05:13:35,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2694253.3333333335, ans=0.2
2023-11-24 05:13:36,928 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:13:43,684 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.435e+01 9.060e+01 9.696e+01 1.244e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 05:13:45,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404150
2023-11-24 05:14:00,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2694386.6666666665, ans=0.0
2023-11-24 05:14:03,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2694386.6666666665, ans=0.0
2023-11-24 05:14:13,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2694453.3333333335, ans=0.0
2023-11-24 05:14:13,380 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.03 vs. limit=15.0
2023-11-24 05:14:16,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2694453.3333333335, ans=0.125
2023-11-24 05:14:25,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2694520.0, ans=0.1
2023-11-24 05:14:34,638 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7400, loss[loss=0.08983, simple_loss=0.1312, pruned_loss=0.01764, audio_tagging_loss=0.006595, over 15289.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.0917, pruned_loss=0.01326, audio_tagging_loss=0.008898, over 3048587.02 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:14:40,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2694586.6666666665, ans=0.125
2023-11-24 05:14:47,289 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404200
2023-11-24 05:15:00,039 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.68 vs. limit=6.0
2023-11-24 05:15:09,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2694720.0, ans=0.0
2023-11-24 05:15:33,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0
2023-11-24 05:15:35,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2694853.3333333335, ans=0.125
2023-11-24 05:15:37,377 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7450, loss[loss=0.07876, simple_loss=0.1082, pruned_loss=0.01622, audio_tagging_loss=0.008453, over 14701.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09172, pruned_loss=0.0134, audio_tagging_loss=0.008841, over 3044833.28 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:15:48,986 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.292e+01 8.446e+01 9.170e+01 9.700e+01 1.341e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 05:15:50,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404250
2023-11-24 05:16:01,019 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.62 vs. limit=6.0
2023-11-24 05:16:11,553 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0
2023-11-24 05:16:16,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2695120.0, ans=0.1
2023-11-24 05:16:28,104 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.14 vs. limit=10.0
2023-11-24 05:16:30,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2695186.6666666665, ans=0.2
2023-11-24 05:16:40,529 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7500, loss[loss=0.06944, simple_loss=0.09366, pruned_loss=0.01457, audio_tagging_loss=0.00804, over 14119.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.0919, pruned_loss=0.01347, audio_tagging_loss=0.008878, over 3047495.60 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:16:41,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2695253.3333333335, ans=0.125
2023-11-24 05:16:41,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2695253.3333333335, ans=0.0
2023-11-24 05:16:44,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.14 vs. limit=6.0
2023-11-24 05:16:52,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404300
2023-11-24 05:16:56,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2695320.0, ans=0.0
2023-11-24 05:17:00,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2695320.0, ans=0.125
2023-11-24 05:17:04,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2695386.6666666665, ans=0.1
2023-11-24 05:17:06,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.40 vs. limit=15.0
2023-11-24 05:17:07,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2695386.6666666665, ans=0.125
2023-11-24 05:17:24,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2695453.3333333335, ans=0.0
2023-11-24 05:17:41,543 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7550, loss[loss=0.08745, simple_loss=0.1206, pruned_loss=0.01779, audio_tagging_loss=0.009376, over 16069.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09138, pruned_loss=0.01334, audio_tagging_loss=0.008917, over 3043718.61 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:17:46,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2695586.6666666665, ans=0.2
2023-11-24 05:17:51,951 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.237e+01 8.670e+01 9.167e+01 9.885e+01 1.310e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 05:17:53,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404350
2023-11-24 05:18:03,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.12 vs. limit=10.0
2023-11-24 05:18:26,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2695786.6666666665, ans=0.0
2023-11-24 05:18:26,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2695786.6666666665, ans=0.0
2023-11-24 05:18:43,189 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7600, loss[loss=0.06547, simple_loss=0.08281, pruned_loss=0.01647, audio_tagging_loss=0.0076, over 15169.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09069, pruned_loss=0.01306, audio_tagging_loss=0.008906, over 3041967.92 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:18:43,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2695920.0, ans=0.125
2023-11-24 05:18:56,225 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404400
2023-11-24 05:19:14,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2696053.3333333335, ans=0.1
2023-11-24 05:19:22,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2696120.0, ans=0.0
2023-11-24 05:19:42,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2696186.6666666665, ans=0.125
2023-11-24 05:19:45,981 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7650, loss[loss=0.07356, simple_loss=0.09813, pruned_loss=0.01611, audio_tagging_loss=0.008387, over 15492.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09037, pruned_loss=0.0131, audio_tagging_loss=0.008851, over 3041262.62 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:19:57,211 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.265e+01 8.997e+01 9.695e+01 1.208e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 05:19:57,827 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.45 vs. limit=22.5
2023-11-24 05:19:58,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404450
2023-11-24 05:20:10,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2696386.6666666665, ans=0.125
2023-11-24 05:20:13,011 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:20:26,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2696453.3333333335, ans=0.1
2023-11-24 05:20:26,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2696453.3333333335, ans=0.125
2023-11-24 05:20:35,368 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.49 vs. limit=6.0
2023-11-24 05:20:48,144 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7700, loss[loss=0.06701, simple_loss=0.0916, pruned_loss=0.009918, audio_tagging_loss=0.01129, over 14816.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09082, pruned_loss=0.01318, audio_tagging_loss=0.008855, over 3043425.00 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:20:50,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2696586.6666666665, ans=0.2
2023-11-24 05:21:00,127 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404500
2023-11-24 05:21:01,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2696653.3333333335, ans=0.125
2023-11-24 05:21:17,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2696720.0, ans=0.1
2023-11-24 05:21:22,870 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-24 05:21:25,057 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.39 vs. limit=15.0
2023-11-24 05:21:50,147 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7750, loss[loss=0.06465, simple_loss=0.08715, pruned_loss=0.01179, audio_tagging_loss=0.009288, over 14443.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09138, pruned_loss=0.01332, audio_tagging_loss=0.008935, over 3036892.81 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:21:58,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2696920.0, ans=0.1
2023-11-24 05:21:59,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2696920.0, ans=0.125
2023-11-24 05:22:01,430 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.10 vs. limit=15.0
2023-11-24 05:22:01,916 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.444e+01 9.119e+01 9.673e+01 1.144e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 05:22:03,948 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404550
2023-11-24 05:22:11,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2696986.6666666665, ans=0.0
2023-11-24 05:22:14,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2697053.3333333335, ans=0.125
2023-11-24 05:22:39,784 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0
2023-11-24 05:22:53,804 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7800, loss[loss=0.064, simple_loss=0.0839, pruned_loss=0.01434, audio_tagging_loss=0.007709, over 14228.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.0921, pruned_loss=0.01343, audio_tagging_loss=0.00898, over 3040417.23 frames. ], batch size: 52, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:22:55,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2697253.3333333335, ans=0.125
2023-11-24 05:23:05,773 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404600
2023-11-24 05:23:44,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.25 vs. limit=15.0
2023-11-24 05:23:48,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2697520.0, ans=0.2
2023-11-24 05:23:56,279 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7850, loss[loss=0.04335, simple_loss=0.05261, pruned_loss=0.004485, audio_tagging_loss=0.01256, over 14848.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09129, pruned_loss=0.01326, audio_tagging_loss=0.009075, over 3043197.06 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:23:58,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=17.76 vs. limit=22.5
2023-11-24 05:24:03,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2697586.6666666665, ans=0.125
2023-11-24 05:24:07,513 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.405e+01 8.451e+01 9.106e+01 9.579e+01 1.135e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 05:24:08,842 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404650
2023-11-24 05:24:22,368 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.20 vs. limit=15.0
2023-11-24 05:24:36,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2697786.6666666665, ans=0.0
2023-11-24 05:24:40,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2697786.6666666665, ans=0.125
2023-11-24 05:24:50,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.98 vs. limit=15.0
2023-11-24 05:24:52,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2697853.3333333335, ans=0.125
2023-11-24 05:24:57,987 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7900, loss[loss=0.07377, simple_loss=0.1007, pruned_loss=0.01758, audio_tagging_loss=0.00586, over 15531.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.0921, pruned_loss=0.01341, audio_tagging_loss=0.009066, over 3047430.01 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:25:11,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404700
2023-11-24 05:25:26,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.12 vs. limit=15.0
2023-11-24 05:25:27,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.29 vs. limit=15.0
2023-11-24 05:25:38,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2698120.0, ans=0.125
2023-11-24 05:25:41,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2698120.0, ans=0.025
2023-11-24 05:25:50,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2698186.6666666665, ans=0.125
2023-11-24 05:25:52,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2698186.6666666665, ans=0.125
2023-11-24 05:26:01,540 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 7950, loss[loss=0.07935, simple_loss=0.0978, pruned_loss=0.01985, audio_tagging_loss=0.0106, over 13758.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09141, pruned_loss=0.01339, audio_tagging_loss=0.009243, over 3045397.95 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:26:06,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2698253.3333333335, ans=0.2
2023-11-24 05:26:12,223 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.346e+01 8.927e+01 9.659e+01 1.276e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-24 05:26:13,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404750
2023-11-24 05:26:13,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2698320.0, ans=0.1
2023-11-24 05:26:15,940 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:26:28,903 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-24 05:27:03,768 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8000, loss[loss=0.05894, simple_loss=0.06834, pruned_loss=0.01403, audio_tagging_loss=0.01074, over 14154.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09034, pruned_loss=0.01326, audio_tagging_loss=0.00937, over 3045003.87 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:27:16,452 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404800
2023-11-24 05:27:39,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2698720.0, ans=0.0
2023-11-24 05:28:03,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2698853.3333333335, ans=0.0
2023-11-24 05:28:06,014 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8050, loss[loss=0.08691, simple_loss=0.1187, pruned_loss=0.01647, audio_tagging_loss=0.01108, over 15188.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09026, pruned_loss=0.0132, audio_tagging_loss=0.009379, over 3044805.28 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:28:17,288 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.661e+01 8.621e+01 9.093e+01 9.694e+01 1.272e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 05:28:19,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404850
2023-11-24 05:28:24,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2698986.6666666665, ans=0.125
2023-11-24 05:28:35,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2699053.3333333335, ans=0.0
2023-11-24 05:28:57,364 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.16 vs. limit=22.5
2023-11-24 05:29:08,674 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8100, loss[loss=0.0757, simple_loss=0.1099, pruned_loss=0.0152, audio_tagging_loss=0.005546, over 14517.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09088, pruned_loss=0.0133, audio_tagging_loss=0.009299, over 3039885.19 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:29:15,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2699253.3333333335, ans=0.2
2023-11-24 05:29:16,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2699253.3333333335, ans=0.0
2023-11-24 05:29:20,329 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:29:21,371 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404900
2023-11-24 05:29:23,336 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.32 vs. limit=15.0
2023-11-24 05:29:23,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2699320.0, ans=0.125
2023-11-24 05:29:31,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2699320.0, ans=0.125
2023-11-24 05:29:43,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2699386.6666666665, ans=0.0
2023-11-24 05:29:57,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2699520.0, ans=0.1
2023-11-24 05:30:10,891 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8150, loss[loss=0.05279, simple_loss=0.06622, pruned_loss=0.008961, audio_tagging_loss=0.01072, over 15095.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09085, pruned_loss=0.0133, audio_tagging_loss=0.009098, over 3045661.38 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:30:18,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2699586.6666666665, ans=0.125
2023-11-24 05:30:22,676 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.549e+01 9.218e+01 9.858e+01 1.332e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 05:30:22,807 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 404950
2023-11-24 05:30:33,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.06 vs. limit=15.0
2023-11-24 05:30:39,818 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.75 vs. limit=6.0
2023-11-24 05:30:41,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2699720.0, ans=0.125
2023-11-24 05:30:53,849 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.04 vs. limit=15.0
2023-11-24 05:31:00,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2699853.3333333335, ans=0.125
2023-11-24 05:31:01,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2699853.3333333335, ans=0.0
2023-11-24 05:31:12,016 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8200, loss[loss=0.05401, simple_loss=0.07745, pruned_loss=0.008019, audio_tagging_loss=0.007266, over 15619.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09198, pruned_loss=0.01366, audio_tagging_loss=0.009057, over 3054047.34 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:31:12,065 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:31:24,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405000
2023-11-24 05:31:26,794 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.37 vs. limit=15.0
2023-11-24 05:31:40,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2700053.3333333335, ans=0.2
2023-11-24 05:31:41,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2700053.3333333335, ans=0.2
2023-11-24 05:31:42,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2700053.3333333335, ans=0.125
2023-11-24 05:31:47,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2700053.3333333335, ans=0.2
2023-11-24 05:31:51,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=2700120.0, ans=0.1
2023-11-24 05:32:00,521 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.62 vs. limit=15.0
2023-11-24 05:32:03,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2700186.6666666665, ans=0.125
2023-11-24 05:32:14,713 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8250, loss[loss=0.05652, simple_loss=0.07008, pruned_loss=0.0101, audio_tagging_loss=0.01137, over 14126.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09221, pruned_loss=0.01378, audio_tagging_loss=0.008955, over 3054043.88 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:32:27,206 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405050
2023-11-24 05:32:28,207 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.524e+01 9.156e+01 9.753e+01 1.627e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 05:32:46,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2700386.6666666665, ans=0.125
2023-11-24 05:32:51,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2700453.3333333335, ans=0.2
2023-11-24 05:33:08,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2700520.0, ans=0.1
2023-11-24 05:33:13,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2700520.0, ans=0.1
2023-11-24 05:33:16,930 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8300, loss[loss=0.06953, simple_loss=0.08805, pruned_loss=0.016, audio_tagging_loss=0.009506, over 14993.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09132, pruned_loss=0.01357, audio_tagging_loss=0.008932, over 3055944.00 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:33:28,875 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405100
2023-11-24 05:33:38,908 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0
2023-11-24 05:33:39,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2700720.0, ans=0.2
2023-11-24 05:33:42,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2700720.0, ans=0.125
2023-11-24 05:33:58,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2700786.6666666665, ans=0.0
2023-11-24 05:34:06,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2700853.3333333335, ans=0.125
2023-11-24 05:34:07,210 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.98 vs. limit=22.5
2023-11-24 05:34:18,253 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8350, loss[loss=0.0741, simple_loss=0.09649, pruned_loss=0.01785, audio_tagging_loss=0.007997, over 14892.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09181, pruned_loss=0.01369, audio_tagging_loss=0.008875, over 3052422.76 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:34:30,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405150
2023-11-24 05:34:31,206 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.407e+01 9.166e+01 9.742e+01 1.258e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 05:34:37,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2700986.6666666665, ans=0.0
2023-11-24 05:34:42,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2701053.3333333335, ans=0.07
2023-11-24 05:34:42,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2701053.3333333335, ans=0.125
2023-11-24 05:34:45,387 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.13 vs. limit=22.5
2023-11-24 05:34:47,584 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.96 vs. limit=15.0
2023-11-24 05:34:50,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2701053.3333333335, ans=0.0
2023-11-24 05:35:15,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2701186.6666666665, ans=0.125
2023-11-24 05:35:19,460 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8400, loss[loss=0.06558, simple_loss=0.0857, pruned_loss=0.012, audio_tagging_loss=0.01073, over 14553.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09208, pruned_loss=0.0135, audio_tagging_loss=0.008798, over 3044633.84 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:35:20,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2701253.3333333335, ans=0.125
2023-11-24 05:35:21,283 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.42 vs. limit=12.0
2023-11-24 05:35:32,217 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405200
2023-11-24 05:35:42,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2701320.0, ans=10.0
2023-11-24 05:35:45,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2701386.6666666665, ans=0.0
2023-11-24 05:35:52,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2701386.6666666665, ans=0.025
2023-11-24 05:35:57,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.38 vs. limit=15.0
2023-11-24 05:36:09,576 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.29 vs. limit=10.0
2023-11-24 05:36:16,313 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.87 vs. limit=15.0
2023-11-24 05:36:21,625 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8450, loss[loss=0.07452, simple_loss=0.09479, pruned_loss=0.01875, audio_tagging_loss=0.008376, over 14558.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09154, pruned_loss=0.01352, audio_tagging_loss=0.008781, over 3046099.89 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:36:32,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2701653.3333333335, ans=0.125
2023-11-24 05:36:33,782 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405250
2023-11-24 05:36:34,767 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.098e+01 8.659e+01 9.489e+01 1.219e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-24 05:36:39,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2701653.3333333335, ans=0.125
2023-11-24 05:36:50,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2701720.0, ans=0.0
2023-11-24 05:37:09,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2701786.6666666665, ans=0.125
2023-11-24 05:37:13,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2701853.3333333335, ans=0.0
2023-11-24 05:37:22,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2701920.0, ans=0.125
2023-11-24 05:37:23,298 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8500, loss[loss=0.09103, simple_loss=0.1262, pruned_loss=0.02092, audio_tagging_loss=0.007026, over 14665.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09121, pruned_loss=0.01354, audio_tagging_loss=0.008879, over 3044445.08 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:37:35,346 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405300
2023-11-24 05:37:36,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2701986.6666666665, ans=0.05
2023-11-24 05:37:37,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2701986.6666666665, ans=0.125
2023-11-24 05:37:43,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2701986.6666666665, ans=0.2
2023-11-24 05:37:52,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2702053.3333333335, ans=0.125
2023-11-24 05:38:24,806 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8550, loss[loss=0.07957, simple_loss=0.1106, pruned_loss=0.01565, audio_tagging_loss=0.008626, over 17406.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09146, pruned_loss=0.01334, audio_tagging_loss=0.008878, over 3054477.43 frames. ], batch size: 63, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:38:37,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2702253.3333333335, ans=0.125
2023-11-24 05:38:39,279 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405350
2023-11-24 05:38:40,288 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.730e+01 8.556e+01 9.186e+01 9.838e+01 1.237e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 05:38:53,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2702386.6666666665, ans=0.125
2023-11-24 05:39:21,255 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2702520.0, ans=0.5
2023-11-24 05:39:29,200 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8600, loss[loss=0.07033, simple_loss=0.09613, pruned_loss=0.0121, audio_tagging_loss=0.01016, over 15388.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09162, pruned_loss=0.0134, audio_tagging_loss=0.008987, over 3055587.88 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:39:36,554 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:39:41,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405400
2023-11-24 05:40:01,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2702720.0, ans=0.125
2023-11-24 05:40:03,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2702720.0, ans=0.05
2023-11-24 05:40:06,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2702786.6666666665, ans=0.1
2023-11-24 05:40:31,241 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8650, loss[loss=0.07368, simple_loss=0.09948, pruned_loss=0.01488, audio_tagging_loss=0.009062, over 15712.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09256, pruned_loss=0.01348, audio_tagging_loss=0.008976, over 3059080.90 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:40:32,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2702920.0, ans=0.125
2023-11-24 05:40:42,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2702986.6666666665, ans=0.0
2023-11-24 05:40:43,252 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405450
2023-11-24 05:40:44,409 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.340e+01 8.964e+01 9.438e+01 1.251e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 05:40:51,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2702986.6666666665, ans=0.1
2023-11-24 05:41:00,029 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.48 vs. limit=6.0
2023-11-24 05:41:16,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2703120.0, ans=0.125
2023-11-24 05:41:24,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2703186.6666666665, ans=0.125
2023-11-24 05:41:33,409 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8700, loss[loss=0.05755, simple_loss=0.07231, pruned_loss=0.01028, audio_tagging_loss=0.01112, over 15408.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09245, pruned_loss=0.01348, audio_tagging_loss=0.00906, over 3054259.11 frames. ], batch size: 62, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:41:39,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2703253.3333333335, ans=0.0
2023-11-24 05:41:46,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405500
2023-11-24 05:42:00,159 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:42:10,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2703453.3333333335, ans=0.125
2023-11-24 05:42:36,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2703586.6666666665, ans=0.0
2023-11-24 05:42:37,529 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8750, loss[loss=0.05209, simple_loss=0.06089, pruned_loss=0.01282, audio_tagging_loss=0.008827, over 15523.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09274, pruned_loss=0.01358, audio_tagging_loss=0.00909, over 3053465.57 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:42:40,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2703586.6666666665, ans=0.125
2023-11-24 05:42:50,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405550
2023-11-24 05:42:51,164 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.699e+01 9.632e+01 1.055e+02 1.285e+02, threshold=1.926e+02, percent-clipped=0.0
2023-11-24 05:42:59,860 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:43:00,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2703720.0, ans=0.125
2023-11-24 05:43:07,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2703720.0, ans=0.0
2023-11-24 05:43:39,329 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8800, loss[loss=0.08318, simple_loss=0.1106, pruned_loss=0.01653, audio_tagging_loss=0.01135, over 13966.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09311, pruned_loss=0.01366, audio_tagging_loss=0.009188, over 3052386.03 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:43:43,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2703920.0, ans=0.0
2023-11-24 05:43:51,382 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405600
2023-11-24 05:44:09,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2704053.3333333335, ans=0.05
2023-11-24 05:44:12,034 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.56 vs. limit=12.0
2023-11-24 05:44:14,242 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.08 vs. limit=12.0
2023-11-24 05:44:41,163 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8850, loss[loss=0.05878, simple_loss=0.06991, pruned_loss=0.01377, audio_tagging_loss=0.01005, over 15733.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09251, pruned_loss=0.01357, audio_tagging_loss=0.009252, over 3051211.17 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:44:43,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2704253.3333333335, ans=0.125
2023-11-24 05:44:53,067 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:44:54,396 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405650
2023-11-24 05:44:55,475 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.409e+01 8.541e+01 8.982e+01 9.482e+01 1.517e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 05:44:57,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2704320.0, ans=0.125
2023-11-24 05:45:07,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2704386.6666666665, ans=6.0
2023-11-24 05:45:20,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2704453.3333333335, ans=0.015
2023-11-24 05:45:26,858 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.32 vs. limit=12.0
2023-11-24 05:45:44,002 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8900, loss[loss=0.0796, simple_loss=0.1066, pruned_loss=0.01793, audio_tagging_loss=0.008366, over 15399.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09301, pruned_loss=0.01373, audio_tagging_loss=0.009049, over 3051435.83 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:45:56,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405700
2023-11-24 05:46:11,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2704720.0, ans=0.0
2023-11-24 05:46:38,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2704853.3333333335, ans=0.125
2023-11-24 05:46:45,643 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 8950, loss[loss=0.07621, simple_loss=0.1061, pruned_loss=0.01589, audio_tagging_loss=0.007281, over 15170.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09297, pruned_loss=0.01375, audio_tagging_loss=0.008836, over 3051235.97 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:46:57,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2704986.6666666665, ans=0.125
2023-11-24 05:46:58,045 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405750
2023-11-24 05:46:59,185 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.112e+01 8.729e+01 9.252e+01 9.908e+01 1.235e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 05:47:23,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2705120.0, ans=0.2
2023-11-24 05:47:26,270 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.23 vs. limit=15.0
2023-11-24 05:47:47,498 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9000, loss[loss=0.06157, simple_loss=0.07782, pruned_loss=0.01242, audio_tagging_loss=0.01024, over 14981.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09314, pruned_loss=0.01384, audio_tagging_loss=0.008781, over 3044706.90 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:47:47,499 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 05:48:30,297 INFO [train_asr.py:1253] (2/4) Epoch 34, validation: loss=0.05898, simple_loss=0.05084, pruned_loss=0.005097, audio_tagging_loss=0.02846, over 4681554.00 frames. 
2023-11-24 05:48:30,298 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 05:48:33,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2705253.3333333335, ans=0.04949747468305833
2023-11-24 05:48:36,669 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.02 vs. limit=10.0
2023-11-24 05:48:39,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2705253.3333333335, ans=0.0
2023-11-24 05:48:42,176 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405800
2023-11-24 05:48:56,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_na.min_abs, batch_count=2705386.6666666665, ans=0.02
2023-11-24 05:48:56,892 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.67 vs. limit=15.0
2023-11-24 05:49:03,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2705386.6666666665, ans=0.07
2023-11-24 05:49:23,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2705520.0, ans=0.2
2023-11-24 05:49:32,037 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9050, loss[loss=0.0881, simple_loss=0.1234, pruned_loss=0.01906, audio_tagging_loss=0.007352, over 15619.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09167, pruned_loss=0.01359, audio_tagging_loss=0.008821, over 3051411.79 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:49:32,741 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0
2023-11-24 05:49:44,525 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405850
2023-11-24 05:49:46,750 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.178e+01 8.606e+01 9.162e+01 9.789e+01 1.610e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 05:50:08,379 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.14 vs. limit=22.5
2023-11-24 05:50:11,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2705786.6666666665, ans=0.05
2023-11-24 05:50:11,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2705786.6666666665, ans=0.1
2023-11-24 05:50:34,564 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9100, loss[loss=0.06678, simple_loss=0.08676, pruned_loss=0.01607, audio_tagging_loss=0.007337, over 15428.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09189, pruned_loss=0.01378, audio_tagging_loss=0.008821, over 3053643.67 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:50:47,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405900
2023-11-24 05:50:58,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2706053.3333333335, ans=0.1
2023-11-24 05:51:29,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2706186.6666666665, ans=0.125
2023-11-24 05:51:33,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2706186.6666666665, ans=0.1
2023-11-24 05:51:36,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.51 vs. limit=22.5
2023-11-24 05:51:37,130 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9150, loss[loss=0.07497, simple_loss=0.1039, pruned_loss=0.01542, audio_tagging_loss=0.007622, over 14427.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09165, pruned_loss=0.01377, audio_tagging_loss=0.008805, over 3051536.03 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:51:49,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 405950
2023-11-24 05:51:51,510 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.335e+01 8.866e+01 9.361e+01 1.025e+02 1.380e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 05:52:10,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2706386.6666666665, ans=0.0
2023-11-24 05:52:10,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2706386.6666666665, ans=0.125
2023-11-24 05:52:20,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2706453.3333333335, ans=0.2
2023-11-24 05:52:38,798 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.19 vs. limit=15.0
2023-11-24 05:52:39,403 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9200, loss[loss=0.0664, simple_loss=0.08968, pruned_loss=0.01337, audio_tagging_loss=0.008191, over 15258.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09159, pruned_loss=0.01366, audio_tagging_loss=0.008747, over 3051107.45 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:52:41,526 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.43 vs. limit=15.0
2023-11-24 05:52:51,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406000
2023-11-24 05:52:59,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2706653.3333333335, ans=0.0
2023-11-24 05:53:03,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2706720.0, ans=0.125
2023-11-24 05:53:37,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2706853.3333333335, ans=0.0
2023-11-24 05:53:39,850 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.34 vs. limit=22.5
2023-11-24 05:53:41,420 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9250, loss[loss=0.1028, simple_loss=0.1328, pruned_loss=0.03091, audio_tagging_loss=0.005489, over 15726.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09117, pruned_loss=0.0136, audio_tagging_loss=0.008768, over 3044980.95 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:53:54,531 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406050
2023-11-24 05:53:57,358 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 8.286e+01 8.929e+01 9.652e+01 1.621e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 05:54:03,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2706986.6666666665, ans=0.1
2023-11-24 05:54:04,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2706986.6666666665, ans=0.125
2023-11-24 05:54:05,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2706986.6666666665, ans=0.0
2023-11-24 05:54:22,429 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.10 vs. limit=15.0
2023-11-24 05:54:34,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2707186.6666666665, ans=0.05
2023-11-24 05:54:45,408 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9300, loss[loss=0.0708, simple_loss=0.09568, pruned_loss=0.01474, audio_tagging_loss=0.008223, over 15458.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09209, pruned_loss=0.01348, audio_tagging_loss=0.008837, over 3050194.94 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:54:47,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.17 vs. limit=15.0
2023-11-24 05:54:48,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2707253.3333333335, ans=0.2
2023-11-24 05:54:52,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2707253.3333333335, ans=0.125
2023-11-24 05:54:57,985 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406100
2023-11-24 05:55:03,214 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.09 vs. limit=22.5
2023-11-24 05:55:17,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2707386.6666666665, ans=0.0
2023-11-24 05:55:28,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2707453.3333333335, ans=0.125
2023-11-24 05:55:29,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2707453.3333333335, ans=0.125
2023-11-24 05:55:30,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.whiten.whitening_limit, batch_count=2707453.3333333335, ans=12.0
2023-11-24 05:55:31,178 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:55:45,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2707520.0, ans=0.125
2023-11-24 05:55:47,270 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9350, loss[loss=0.07254, simple_loss=0.1066, pruned_loss=0.01216, audio_tagging_loss=0.007055, over 16127.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09185, pruned_loss=0.01345, audio_tagging_loss=0.008814, over 3050954.92 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:55:57,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.77 vs. limit=12.0
2023-11-24 05:55:58,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2707653.3333333335, ans=0.125
2023-11-24 05:55:59,137 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406150
2023-11-24 05:56:01,326 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.542e+01 9.315e+01 1.015e+02 1.329e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 05:56:22,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2707720.0, ans=0.1
2023-11-24 05:56:23,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2707720.0, ans=0.125
2023-11-24 05:56:31,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2707786.6666666665, ans=0.125
2023-11-24 05:56:49,158 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9400, loss[loss=0.08089, simple_loss=0.1067, pruned_loss=0.01706, audio_tagging_loss=0.0105, over 15553.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09149, pruned_loss=0.01329, audio_tagging_loss=0.008968, over 3039947.33 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:56:51,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2707920.0, ans=0.1
2023-11-24 05:56:53,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2707920.0, ans=15.0
2023-11-24 05:57:02,283 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406200
2023-11-24 05:57:16,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2708053.3333333335, ans=0.125
2023-11-24 05:57:20,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2708053.3333333335, ans=0.125
2023-11-24 05:57:30,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2708120.0, ans=0.5
2023-11-24 05:57:44,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2708186.6666666665, ans=0.2
2023-11-24 05:57:46,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2708186.6666666665, ans=0.125
2023-11-24 05:57:50,058 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:57:52,354 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9450, loss[loss=0.08132, simple_loss=0.1178, pruned_loss=0.01517, audio_tagging_loss=0.007273, over 16539.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09115, pruned_loss=0.01304, audio_tagging_loss=0.008906, over 3040076.18 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:58:05,405 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406250
2023-11-24 05:58:06,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2708320.0, ans=0.04949747468305833
2023-11-24 05:58:07,663 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.368e+01 8.774e+01 9.342e+01 1.287e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-24 05:58:23,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2708386.6666666665, ans=0.0
2023-11-24 05:58:55,170 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9500, loss[loss=0.05281, simple_loss=0.07451, pruned_loss=0.006776, audio_tagging_loss=0.008778, over 14177.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09096, pruned_loss=0.01309, audio_tagging_loss=0.008988, over 3042451.09 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:59:06,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2708653.3333333335, ans=0.1
2023-11-24 05:59:07,308 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406300
2023-11-24 05:59:15,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2708653.3333333335, ans=0.0
2023-11-24 05:59:35,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2708786.6666666665, ans=0.125
2023-11-24 05:59:44,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.68 vs. limit=22.5
2023-11-24 05:59:47,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2708853.3333333335, ans=0.1
2023-11-24 05:59:56,980 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9550, loss[loss=0.06929, simple_loss=0.09041, pruned_loss=0.0135, audio_tagging_loss=0.01058, over 15231.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09126, pruned_loss=0.01316, audio_tagging_loss=0.008997, over 3047801.36 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 06:00:09,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406350
2023-11-24 06:00:13,193 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.403e+01 9.161e+01 9.740e+01 1.180e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 06:00:34,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2709120.0, ans=0.125
2023-11-24 06:00:40,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2709120.0, ans=0.1
2023-11-24 06:00:47,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2709186.6666666665, ans=0.125
2023-11-24 06:00:59,383 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9600, loss[loss=0.06344, simple_loss=0.09024, pruned_loss=0.009278, audio_tagging_loss=0.009042, over 14633.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09097, pruned_loss=0.01302, audio_tagging_loss=0.009127, over 3050822.01 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 06:01:02,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2709253.3333333335, ans=0.0
2023-11-24 06:01:05,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2709253.3333333335, ans=0.125
2023-11-24 06:01:08,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2709253.3333333335, ans=0.125
2023-11-24 06:01:13,014 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406400
2023-11-24 06:01:22,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2709320.0, ans=0.1
2023-11-24 06:01:26,710 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:01:30,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2709386.6666666665, ans=0.125
2023-11-24 06:01:31,789 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.19 vs. limit=15.0
2023-11-24 06:01:37,910 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.38 vs. limit=15.0
2023-11-24 06:01:44,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2709453.3333333335, ans=0.1
2023-11-24 06:01:47,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2709453.3333333335, ans=0.0
2023-11-24 06:01:55,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2709520.0, ans=0.0
2023-11-24 06:02:03,407 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9650, loss[loss=0.08306, simple_loss=0.1108, pruned_loss=0.01965, audio_tagging_loss=0.008002, over 15112.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09114, pruned_loss=0.01321, audio_tagging_loss=0.009098, over 3051405.93 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 06:02:15,196 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406450
2023-11-24 06:02:17,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2709653.3333333335, ans=0.125
2023-11-24 06:02:18,589 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.885e+01 8.449e+01 8.956e+01 9.585e+01 1.411e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 06:02:21,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.62 vs. limit=15.0
2023-11-24 06:02:35,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2709720.0, ans=0.2
2023-11-24 06:02:55,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2709853.3333333335, ans=10.0
2023-11-24 06:03:04,824 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9700, loss[loss=0.08064, simple_loss=0.1, pruned_loss=0.02148, audio_tagging_loss=0.009142, over 15019.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09076, pruned_loss=0.01308, audio_tagging_loss=0.008955, over 3053576.40 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:03:16,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406500
2023-11-24 06:03:19,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2709986.6666666665, ans=0.125
2023-11-24 06:03:24,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2709986.6666666665, ans=0.125
2023-11-24 06:03:31,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2710053.3333333335, ans=0.0
2023-11-24 06:03:31,515 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.36 vs. limit=22.5
2023-11-24 06:03:35,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2710053.3333333335, ans=0.0
2023-11-24 06:03:36,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2710053.3333333335, ans=0.0
2023-11-24 06:03:44,235 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.92 vs. limit=22.5
2023-11-24 06:03:49,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2710120.0, ans=0.125
2023-11-24 06:03:53,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2710186.6666666665, ans=0.125
2023-11-24 06:03:58,455 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.17 vs. limit=22.5
2023-11-24 06:04:06,191 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9750, loss[loss=0.0895, simple_loss=0.119, pruned_loss=0.02187, audio_tagging_loss=0.008128, over 15432.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09133, pruned_loss=0.01322, audio_tagging_loss=0.008913, over 3051981.43 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:04:19,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406550
2023-11-24 06:04:24,968 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.565e+01 9.211e+01 9.883e+01 1.176e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 06:04:26,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2710320.0, ans=0.125
2023-11-24 06:04:27,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2710320.0, ans=0.1
2023-11-24 06:04:46,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.05 vs. limit=22.5
2023-11-24 06:04:57,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2710520.0, ans=0.125
2023-11-24 06:04:58,217 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0
2023-11-24 06:05:06,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2710520.0, ans=0.0
2023-11-24 06:05:09,513 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9800, loss[loss=0.05813, simple_loss=0.08019, pruned_loss=0.01037, audio_tagging_loss=0.007671, over 14453.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09125, pruned_loss=0.0132, audio_tagging_loss=0.008957, over 3054160.44 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:05:22,215 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406600
2023-11-24 06:05:23,678 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:05:28,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2710653.3333333335, ans=0.125
2023-11-24 06:05:39,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2710720.0, ans=0.125
2023-11-24 06:05:41,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2710720.0, ans=0.1
2023-11-24 06:05:46,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2710786.6666666665, ans=0.0
2023-11-24 06:06:01,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2710853.3333333335, ans=0.125
2023-11-24 06:06:04,968 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:06:12,006 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9850, loss[loss=0.08031, simple_loss=0.1143, pruned_loss=0.01621, audio_tagging_loss=0.006921, over 15977.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09179, pruned_loss=0.01334, audio_tagging_loss=0.008826, over 3052490.85 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:06:23,918 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406650
2023-11-24 06:06:28,338 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.400e+01 8.541e+01 9.131e+01 9.880e+01 1.241e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 06:06:33,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2710986.6666666665, ans=0.0
2023-11-24 06:06:35,966 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.08 vs. limit=15.0
2023-11-24 06:06:49,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2711120.0, ans=0.125
2023-11-24 06:06:54,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2711120.0, ans=0.1
2023-11-24 06:06:58,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2711120.0, ans=0.1
2023-11-24 06:07:08,184 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.97 vs. limit=22.5
2023-11-24 06:07:13,488 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9900, loss[loss=0.06146, simple_loss=0.07409, pruned_loss=0.01053, audio_tagging_loss=0.01389, over 16220.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09215, pruned_loss=0.01343, audio_tagging_loss=0.008783, over 3050848.47 frames. ], batch size: 62, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:07:19,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2711253.3333333335, ans=0.0
2023-11-24 06:07:27,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406700
2023-11-24 06:07:47,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2711386.6666666665, ans=0.125
2023-11-24 06:07:53,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2711453.3333333335, ans=0.125
2023-11-24 06:08:07,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2711520.0, ans=0.125
2023-11-24 06:08:07,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2711520.0, ans=0.125
2023-11-24 06:08:10,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2711520.0, ans=0.5
2023-11-24 06:08:16,817 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 9950, loss[loss=0.05368, simple_loss=0.07095, pruned_loss=0.008566, audio_tagging_loss=0.009636, over 15824.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09209, pruned_loss=0.01341, audio_tagging_loss=0.008772, over 3046890.83 frames. ], batch size: 61, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:08:29,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406750
2023-11-24 06:08:30,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2711653.3333333335, ans=0.2
2023-11-24 06:08:34,881 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.047e+01 8.279e+01 9.180e+01 9.821e+01 1.550e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 06:08:57,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2711786.6666666665, ans=0.2
2023-11-24 06:09:06,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2711853.3333333335, ans=0.2
2023-11-24 06:09:18,675 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10000, loss[loss=0.08047, simple_loss=0.1089, pruned_loss=0.01458, audio_tagging_loss=0.01143, over 15557.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09199, pruned_loss=0.01346, audio_tagging_loss=0.008751, over 3048304.78 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:09:18,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2711920.0, ans=0.0
2023-11-24 06:09:30,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406800
2023-11-24 06:10:12,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2712186.6666666665, ans=0.125
2023-11-24 06:10:20,486 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10050, loss[loss=0.06967, simple_loss=0.09292, pruned_loss=0.01392, audio_tagging_loss=0.009291, over 14670.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09143, pruned_loss=0.01336, audio_tagging_loss=0.008775, over 3049232.35 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:10:33,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406850
2023-11-24 06:10:39,683 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.233e+01 8.347e+01 9.098e+01 9.653e+01 1.255e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 06:10:57,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2712453.3333333335, ans=0.125
2023-11-24 06:10:59,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2712453.3333333335, ans=0.0
2023-11-24 06:11:22,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2712586.6666666665, ans=0.1
2023-11-24 06:11:23,351 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10100, loss[loss=0.07761, simple_loss=0.1022, pruned_loss=0.01572, audio_tagging_loss=0.01078, over 14166.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09189, pruned_loss=0.01349, audio_tagging_loss=0.008861, over 3046393.78 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:11:31,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2712586.6666666665, ans=0.125
2023-11-24 06:11:35,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406900
2023-11-24 06:11:48,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2712720.0, ans=0.125
2023-11-24 06:12:12,372 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:12:24,232 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10150, loss[loss=0.08282, simple_loss=0.1121, pruned_loss=0.01883, audio_tagging_loss=0.007953, over 16045.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09189, pruned_loss=0.0135, audio_tagging_loss=0.00887, over 3054970.14 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:12:33,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.55 vs. limit=15.0
2023-11-24 06:12:36,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 406950
2023-11-24 06:12:42,365 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.240e+01 8.502e+01 9.089e+01 9.719e+01 1.256e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 06:12:51,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2713053.3333333335, ans=10.0
2023-11-24 06:12:52,477 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:12:52,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2713053.3333333335, ans=0.125
2023-11-24 06:12:56,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2713053.3333333335, ans=0.0
2023-11-24 06:13:26,449 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10200, loss[loss=0.06151, simple_loss=0.07679, pruned_loss=0.01541, audio_tagging_loss=0.007708, over 15071.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09187, pruned_loss=0.01352, audio_tagging_loss=0.008909, over 3053659.96 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:13:35,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2713253.3333333335, ans=0.125
2023-11-24 06:13:39,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407000
2023-11-24 06:13:46,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2713320.0, ans=0.0
2023-11-24 06:13:51,001 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:14:29,225 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10250, loss[loss=0.06683, simple_loss=0.09604, pruned_loss=0.01054, audio_tagging_loss=0.008266, over 16660.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09253, pruned_loss=0.01352, audio_tagging_loss=0.008995, over 3061835.40 frames. ], batch size: 62, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:14:41,630 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407050
2023-11-24 06:14:46,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2713653.3333333335, ans=0.0
2023-11-24 06:14:47,352 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.429e+01 8.707e+01 9.386e+01 1.043e+02 1.340e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-24 06:14:58,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2713720.0, ans=0.0
2023-11-24 06:14:59,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2713720.0, ans=0.1
2023-11-24 06:15:17,413 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.80 vs. limit=22.5
2023-11-24 06:15:20,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2713853.3333333335, ans=0.125
2023-11-24 06:15:27,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2713853.3333333335, ans=0.07
2023-11-24 06:15:30,883 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10300, loss[loss=0.05615, simple_loss=0.07402, pruned_loss=0.01043, audio_tagging_loss=0.008713, over 15766.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.0925, pruned_loss=0.01369, audio_tagging_loss=0.008943, over 3057212.68 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:15:38,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2713920.0, ans=0.1
2023-11-24 06:15:38,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.07 vs. limit=15.0
2023-11-24 06:15:42,732 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407100
2023-11-24 06:15:59,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2714053.3333333335, ans=0.1
2023-11-24 06:16:08,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2714120.0, ans=0.0
2023-11-24 06:16:18,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2714120.0, ans=0.125
2023-11-24 06:16:32,181 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10350, loss[loss=0.05199, simple_loss=0.06548, pruned_loss=0.008457, audio_tagging_loss=0.01079, over 16066.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09218, pruned_loss=0.01359, audio_tagging_loss=0.009067, over 3057368.51 frames. ], batch size: 61, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:16:45,325 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407150
2023-11-24 06:16:52,214 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.280e+01 8.459e+01 9.063e+01 9.492e+01 1.258e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 06:16:57,637 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.87 vs. limit=15.0
2023-11-24 06:17:23,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2714520.0, ans=0.1
2023-11-24 06:17:28,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2714520.0, ans=0.125
2023-11-24 06:17:31,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2714520.0, ans=0.1
2023-11-24 06:17:35,099 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10400, loss[loss=0.06536, simple_loss=0.09071, pruned_loss=0.01208, audio_tagging_loss=0.007918, over 14340.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09086, pruned_loss=0.01325, audio_tagging_loss=0.009291, over 3054456.42 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:17:43,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2714586.6666666665, ans=0.1
2023-11-24 06:17:45,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.59 vs. limit=22.5
2023-11-24 06:17:47,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407200
2023-11-24 06:17:57,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2714653.3333333335, ans=0.125
2023-11-24 06:18:13,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2714786.6666666665, ans=0.125
2023-11-24 06:18:31,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2714853.3333333335, ans=0.05
2023-11-24 06:18:31,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.14 vs. limit=15.0
2023-11-24 06:18:38,039 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10450, loss[loss=0.08174, simple_loss=0.1095, pruned_loss=0.01818, audio_tagging_loss=0.008827, over 15474.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09144, pruned_loss=0.01322, audio_tagging_loss=0.009166, over 3056032.96 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:18:49,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407250
2023-11-24 06:18:55,559 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.473e+01 9.221e+01 1.007e+02 1.315e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 06:19:04,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2715053.3333333335, ans=0.0
2023-11-24 06:19:20,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2715120.0, ans=0.0
2023-11-24 06:19:34,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.58 vs. limit=10.0
2023-11-24 06:19:35,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.99 vs. limit=15.0
2023-11-24 06:19:38,974 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10500, loss[loss=0.0639, simple_loss=0.09376, pruned_loss=0.01215, audio_tagging_loss=0.004873, over 15276.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09104, pruned_loss=0.01314, audio_tagging_loss=0.009058, over 3050120.22 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:19:45,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2715253.3333333335, ans=0.0
2023-11-24 06:19:51,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407300
2023-11-24 06:20:41,118 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10550, loss[loss=0.06192, simple_loss=0.08509, pruned_loss=0.008945, audio_tagging_loss=0.01043, over 15112.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09043, pruned_loss=0.01317, audio_tagging_loss=0.008986, over 3039531.82 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:20:45,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2715586.6666666665, ans=0.0
2023-11-24 06:20:54,300 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407350
2023-11-24 06:20:56,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2715653.3333333335, ans=0.125
2023-11-24 06:21:01,302 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.501e+01 9.231e+01 9.962e+01 1.175e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 06:21:05,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2715720.0, ans=0.125
2023-11-24 06:21:43,197 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10600, loss[loss=0.07432, simple_loss=0.1008, pruned_loss=0.0175, audio_tagging_loss=0.006436, over 15384.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09102, pruned_loss=0.01341, audio_tagging_loss=0.008934, over 3041380.37 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:21:44,066 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.02 vs. limit=12.0
2023-11-24 06:21:45,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2715920.0, ans=0.1
2023-11-24 06:21:50,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2715920.0, ans=0.125
2023-11-24 06:21:55,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407400
2023-11-24 06:21:58,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2715986.6666666665, ans=0.125
2023-11-24 06:22:05,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2715986.6666666665, ans=0.125
2023-11-24 06:22:22,238 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0
2023-11-24 06:22:45,109 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10650, loss[loss=0.04083, simple_loss=0.04871, pruned_loss=0.005004, audio_tagging_loss=0.01147, over 14222.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09095, pruned_loss=0.01335, audio_tagging_loss=0.008883, over 3043429.55 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:22:46,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2716253.3333333335, ans=0.1
2023-11-24 06:22:56,877 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407450
2023-11-24 06:22:59,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2716320.0, ans=0.125
2023-11-24 06:23:04,944 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.300e+01 8.906e+01 9.684e+01 1.178e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-24 06:23:08,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.whiten.whitening_limit, batch_count=2716320.0, ans=12.0
2023-11-24 06:23:10,406 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.34 vs. limit=15.0
2023-11-24 06:23:26,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2716453.3333333335, ans=0.1
2023-11-24 06:23:46,562 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10700, loss[loss=0.06011, simple_loss=0.0853, pruned_loss=0.009516, audio_tagging_loss=0.007941, over 13940.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09124, pruned_loss=0.01338, audio_tagging_loss=0.008793, over 3044691.23 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:23:53,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2716586.6666666665, ans=0.0
2023-11-24 06:23:58,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2716586.6666666665, ans=0.125
2023-11-24 06:24:00,600 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407500
2023-11-24 06:24:06,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2716653.3333333335, ans=0.0
2023-11-24 06:24:07,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2716653.3333333335, ans=0.125
2023-11-24 06:24:09,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2716653.3333333335, ans=0.5
2023-11-24 06:24:11,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2716720.0, ans=0.125
2023-11-24 06:24:17,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2716720.0, ans=0.2
2023-11-24 06:24:39,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2716853.3333333335, ans=0.05
2023-11-24 06:24:45,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2716853.3333333335, ans=0.125
2023-11-24 06:24:50,035 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10750, loss[loss=0.07254, simple_loss=0.09065, pruned_loss=0.01558, audio_tagging_loss=0.01163, over 14584.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.091, pruned_loss=0.01325, audio_tagging_loss=0.008738, over 3046111.61 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:25:02,097 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407550
2023-11-24 06:25:04,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2716986.6666666665, ans=0.125
2023-11-24 06:25:09,202 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.785e+01 8.501e+01 8.981e+01 9.807e+01 1.307e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 06:25:21,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2717053.3333333335, ans=0.0
2023-11-24 06:25:25,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2717120.0, ans=0.125
2023-11-24 06:25:27,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2717120.0, ans=0.0
2023-11-24 06:25:36,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2717120.0, ans=0.2
2023-11-24 06:25:51,867 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10800, loss[loss=0.07483, simple_loss=0.1078, pruned_loss=0.01536, audio_tagging_loss=0.005593, over 16664.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09151, pruned_loss=0.01336, audio_tagging_loss=0.008714, over 3045815.28 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:26:04,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407600
2023-11-24 06:26:06,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2717320.0, ans=0.125
2023-11-24 06:26:41,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2717520.0, ans=0.125
2023-11-24 06:26:41,573 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.08 vs. limit=12.0
2023-11-24 06:26:42,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2717520.0, ans=0.2
2023-11-24 06:26:54,163 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10850, loss[loss=0.0773, simple_loss=0.1021, pruned_loss=0.01748, audio_tagging_loss=0.008754, over 15975.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09178, pruned_loss=0.01332, audio_tagging_loss=0.008816, over 3044743.77 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:27:00,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2717586.6666666665, ans=0.0
2023-11-24 06:27:07,776 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407650
2023-11-24 06:27:15,318 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.695e+01 9.228e+01 1.009e+02 1.373e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 06:27:17,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2717653.3333333335, ans=0.125
2023-11-24 06:27:24,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2717720.0, ans=0.2
2023-11-24 06:27:24,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2717720.0, ans=0.0
2023-11-24 06:27:37,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2717786.6666666665, ans=0.1
2023-11-24 06:27:39,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2717786.6666666665, ans=0.0
2023-11-24 06:27:43,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2717853.3333333335, ans=0.1
2023-11-24 06:27:54,161 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:27:57,728 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10900, loss[loss=0.08366, simple_loss=0.1094, pruned_loss=0.01873, audio_tagging_loss=0.01025, over 15103.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09091, pruned_loss=0.01315, audio_tagging_loss=0.00886, over 3038612.27 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:28:00,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.52 vs. limit=15.0
2023-11-24 06:28:10,233 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407700
2023-11-24 06:28:12,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2717986.6666666665, ans=0.0
2023-11-24 06:28:19,038 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.57 vs. limit=22.5
2023-11-24 06:28:24,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2718053.3333333335, ans=0.0
2023-11-24 06:28:35,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2718120.0, ans=0.0
2023-11-24 06:28:59,465 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 10950, loss[loss=0.0784, simple_loss=0.1128, pruned_loss=0.01534, audio_tagging_loss=0.00666, over 16063.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09163, pruned_loss=0.01324, audio_tagging_loss=0.008834, over 3045935.11 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:29:11,395 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407750
2023-11-24 06:29:12,991 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.96 vs. limit=15.0
2023-11-24 06:29:20,637 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.403e+01 8.864e+01 9.655e+01 1.476e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-24 06:29:33,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2718386.6666666665, ans=0.2
2023-11-24 06:29:47,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2718453.3333333335, ans=0.125
2023-11-24 06:29:49,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2718520.0, ans=0.125
2023-11-24 06:29:50,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2718520.0, ans=0.0
2023-11-24 06:30:00,919 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11000, loss[loss=0.08129, simple_loss=0.1055, pruned_loss=0.01959, audio_tagging_loss=0.008941, over 15491.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.0906, pruned_loss=0.01314, audio_tagging_loss=0.008936, over 3048430.81 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:30:02,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=12.0
2023-11-24 06:30:11,027 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:30:13,997 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407800
2023-11-24 06:30:33,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2718720.0, ans=0.125
2023-11-24 06:30:40,658 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2718786.6666666665, ans=0.0
2023-11-24 06:30:41,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2718786.6666666665, ans=0.125
2023-11-24 06:30:59,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2718853.3333333335, ans=0.125
2023-11-24 06:31:04,164 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.82 vs. limit=15.0
2023-11-24 06:31:04,752 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11050, loss[loss=0.07881, simple_loss=0.111, pruned_loss=0.01259, audio_tagging_loss=0.01073, over 16415.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09067, pruned_loss=0.01321, audio_tagging_loss=0.009112, over 3048830.71 frames. ], batch size: 61, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:31:15,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2718986.6666666665, ans=0.0
2023-11-24 06:31:16,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.48 vs. limit=15.0
2023-11-24 06:31:17,357 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407850
2023-11-24 06:31:17,826 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0
2023-11-24 06:31:26,786 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.514e+01 8.990e+01 9.675e+01 1.640e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 06:31:32,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=15.0
2023-11-24 06:31:35,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2719053.3333333335, ans=0.125
2023-11-24 06:31:38,109 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.43 vs. limit=22.5
2023-11-24 06:31:51,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.16 vs. limit=10.0
2023-11-24 06:32:01,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2719186.6666666665, ans=0.125
2023-11-24 06:32:06,868 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11100, loss[loss=0.07003, simple_loss=0.09689, pruned_loss=0.01367, audio_tagging_loss=0.007916, over 15039.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09198, pruned_loss=0.01332, audio_tagging_loss=0.009093, over 3045212.03 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:32:08,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2719253.3333333335, ans=0.125
2023-11-24 06:32:12,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2719253.3333333335, ans=0.5
2023-11-24 06:32:19,007 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407900
2023-11-24 06:32:38,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2719386.6666666665, ans=0.04949747468305833
2023-11-24 06:32:39,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2719386.6666666665, ans=0.125
2023-11-24 06:32:45,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2719453.3333333335, ans=0.2
2023-11-24 06:32:48,154 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:32:49,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=2719453.3333333335, ans=0.02
2023-11-24 06:32:56,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.46 vs. limit=15.0
2023-11-24 06:33:00,680 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.12 vs. limit=6.0
2023-11-24 06:33:08,625 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11150, loss[loss=0.07014, simple_loss=0.09569, pruned_loss=0.009025, audio_tagging_loss=0.01327, over 15229.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09135, pruned_loss=0.01312, audio_tagging_loss=0.009159, over 3050077.23 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:33:21,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 407950
2023-11-24 06:33:31,717 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.657e+01 9.225e+01 9.905e+01 1.226e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 06:33:35,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2719720.0, ans=0.1
2023-11-24 06:33:36,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2719720.0, ans=0.0
2023-11-24 06:33:44,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2719720.0, ans=0.035
2023-11-24 06:33:50,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.whiten.whitening_limit, batch_count=2719786.6666666665, ans=12.0
2023-11-24 06:33:57,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.05 vs. limit=10.0
2023-11-24 06:34:00,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2719853.3333333335, ans=0.2
2023-11-24 06:34:10,875 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11200, loss[loss=0.09, simple_loss=0.1231, pruned_loss=0.02094, audio_tagging_loss=0.007521, over 14576.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09098, pruned_loss=0.01298, audio_tagging_loss=0.009304, over 3049810.62 frames. ], batch size: 53, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:34:23,151 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408000
2023-11-24 06:34:23,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2719986.6666666665, ans=0.125
2023-11-24 06:34:23,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2719986.6666666665, ans=0.125
2023-11-24 06:34:36,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2719986.6666666665, ans=0.125
2023-11-24 06:34:59,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2720120.0, ans=0.125
2023-11-24 06:35:08,804 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:35:16,696 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11250, loss[loss=0.06364, simple_loss=0.0893, pruned_loss=0.01078, audio_tagging_loss=0.008211, over 15069.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09116, pruned_loss=0.01325, audio_tagging_loss=0.009228, over 3054199.01 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:35:29,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408050
2023-11-24 06:35:32,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2720320.0, ans=0.2
2023-11-24 06:35:39,635 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.153e+01 8.539e+01 9.181e+01 9.755e+01 1.709e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 06:35:46,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2720386.6666666665, ans=0.09899494936611666
2023-11-24 06:35:46,751 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.41 vs. limit=22.5
2023-11-24 06:35:58,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2720453.3333333335, ans=0.125
2023-11-24 06:36:07,645 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.18 vs. limit=15.0
2023-11-24 06:36:08,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2720520.0, ans=0.0
2023-11-24 06:36:14,315 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.11 vs. limit=15.0
2023-11-24 06:36:18,125 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11300, loss[loss=0.04174, simple_loss=0.0507, pruned_loss=0.005512, audio_tagging_loss=0.01087, over 16479.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09135, pruned_loss=0.01319, audio_tagging_loss=0.009188, over 3059305.65 frames. ], batch size: 63, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:36:20,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2720586.6666666665, ans=0.0
2023-11-24 06:36:30,469 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408100
2023-11-24 06:36:40,498 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.46 vs. limit=15.0
2023-11-24 06:36:45,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2720720.0, ans=0.0
2023-11-24 06:36:48,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2720720.0, ans=0.0
2023-11-24 06:36:48,385 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.79 vs. limit=15.0
2023-11-24 06:37:04,623 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.09 vs. limit=15.0
2023-11-24 06:37:07,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2720853.3333333335, ans=0.1
2023-11-24 06:37:19,929 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11350, loss[loss=0.0536, simple_loss=0.07604, pruned_loss=0.008385, audio_tagging_loss=0.007198, over 14901.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09127, pruned_loss=0.01321, audio_tagging_loss=0.009015, over 3053010.35 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:37:26,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2720920.0, ans=0.0
2023-11-24 06:37:33,081 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408150
2023-11-24 06:37:33,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2720986.6666666665, ans=0.0
2023-11-24 06:37:41,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2720986.6666666665, ans=0.0
2023-11-24 06:37:43,672 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.373e+01 9.046e+01 9.724e+01 1.113e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 06:37:45,559 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.05 vs. limit=10.0
2023-11-24 06:37:46,344 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2721053.3333333335, ans=0.1
2023-11-24 06:37:54,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2721053.3333333335, ans=0.125
2023-11-24 06:38:12,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2721186.6666666665, ans=0.0
2023-11-24 06:38:16,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2721186.6666666665, ans=0.125
2023-11-24 06:38:21,918 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:38:22,785 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11400, loss[loss=0.08426, simple_loss=0.1293, pruned_loss=0.01486, audio_tagging_loss=0.004766, over 15297.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09145, pruned_loss=0.01339, audio_tagging_loss=0.008859, over 3050225.95 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:38:34,683 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408200
2023-11-24 06:38:39,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2721320.0, ans=0.125
2023-11-24 06:38:42,034 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:38:57,747 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.20 vs. limit=15.0
2023-11-24 06:39:01,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=22.5
2023-11-24 06:39:02,027 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:39:24,330 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11450, loss[loss=0.05074, simple_loss=0.06447, pruned_loss=0.007877, audio_tagging_loss=0.01063, over 14561.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09122, pruned_loss=0.01318, audio_tagging_loss=0.008789, over 3051866.04 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:39:36,972 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408250
2023-11-24 06:39:48,647 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.611e+01 9.148e+01 9.766e+01 1.313e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 06:39:48,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2721720.0, ans=0.0
2023-11-24 06:40:00,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2721786.6666666665, ans=0.125
2023-11-24 06:40:01,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2721786.6666666665, ans=0.125
2023-11-24 06:40:14,698 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:40:14,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2721853.3333333335, ans=0.125
2023-11-24 06:40:24,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2721853.3333333335, ans=0.5
2023-11-24 06:40:26,822 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11500, loss[loss=0.06343, simple_loss=0.09004, pruned_loss=0.01058, audio_tagging_loss=0.00783, over 15846.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09151, pruned_loss=0.01321, audio_tagging_loss=0.008757, over 3055938.51 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:40:31,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2721920.0, ans=0.125
2023-11-24 06:40:39,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408300
2023-11-24 06:41:04,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2722120.0, ans=0.1
2023-11-24 06:41:17,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2722186.6666666665, ans=0.0
2023-11-24 06:41:17,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2722186.6666666665, ans=0.125
2023-11-24 06:41:20,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2722186.6666666665, ans=0.125
2023-11-24 06:41:28,735 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11550, loss[loss=0.06211, simple_loss=0.07125, pruned_loss=0.01467, audio_tagging_loss=0.01182, over 15494.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09141, pruned_loss=0.01327, audio_tagging_loss=0.008803, over 3051419.18 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:41:33,658 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:41:36,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2722253.3333333335, ans=0.125
2023-11-24 06:41:40,672 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408350
2023-11-24 06:41:51,959 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.495e+01 8.414e+01 9.031e+01 9.585e+01 1.239e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 06:41:55,067 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.91 vs. limit=22.5
2023-11-24 06:41:58,209 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:42:07,463 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:42:08,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2722453.3333333335, ans=0.0
2023-11-24 06:42:12,862 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-24 06:42:18,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2722520.0, ans=0.0
2023-11-24 06:42:29,711 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11600, loss[loss=0.03969, simple_loss=0.04894, pruned_loss=0.005393, audio_tagging_loss=0.009826, over 14834.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09092, pruned_loss=0.01325, audio_tagging_loss=0.008897, over 3055389.52 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:42:42,243 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408400
2023-11-24 06:43:04,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2722720.0, ans=0.2
2023-11-24 06:43:18,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2722853.3333333335, ans=0.0
2023-11-24 06:43:23,779 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.10 vs. limit=15.0
2023-11-24 06:43:31,822 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11650, loss[loss=0.05618, simple_loss=0.07285, pruned_loss=0.009801, audio_tagging_loss=0.009949, over 14846.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09085, pruned_loss=0.01327, audio_tagging_loss=0.008973, over 3058918.56 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:43:41,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2722920.0, ans=0.0
2023-11-24 06:43:43,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2722920.0, ans=0.2
2023-11-24 06:43:45,497 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408450
2023-11-24 06:43:52,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2722986.6666666665, ans=0.125
2023-11-24 06:43:55,841 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.490e+01 8.921e+01 9.548e+01 1.142e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 06:44:00,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.11 vs. limit=6.0
2023-11-24 06:44:00,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2723053.3333333335, ans=0.0
2023-11-24 06:44:04,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2723053.3333333335, ans=0.125
2023-11-24 06:44:08,205 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.14 vs. limit=15.0
2023-11-24 06:44:34,534 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11700, loss[loss=0.05737, simple_loss=0.07877, pruned_loss=0.008423, audio_tagging_loss=0.009558, over 14719.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09033, pruned_loss=0.01312, audio_tagging_loss=0.009019, over 3059204.07 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:44:37,543 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.61 vs. limit=22.5
2023-11-24 06:44:43,543 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.43 vs. limit=12.0
2023-11-24 06:44:46,454 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408500
2023-11-24 06:44:49,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2723320.0, ans=0.1
2023-11-24 06:44:58,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2723386.6666666665, ans=0.04949747468305833
2023-11-24 06:45:03,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2723386.6666666665, ans=0.05
2023-11-24 06:45:03,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2723386.6666666665, ans=0.0
2023-11-24 06:45:09,381 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.25 vs. limit=15.0
2023-11-24 06:45:23,215 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:45:35,014 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:45:35,966 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11750, loss[loss=0.07272, simple_loss=0.09517, pruned_loss=0.01654, audio_tagging_loss=0.008595, over 16207.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.0908, pruned_loss=0.0133, audio_tagging_loss=0.009035, over 3061230.99 frames. ], batch size: 61, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:45:36,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2723586.6666666665, ans=0.0
2023-11-24 06:45:48,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408550
2023-11-24 06:45:51,898 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.13 vs. limit=10.0
2023-11-24 06:45:54,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2723653.3333333335, ans=0.05
2023-11-24 06:45:59,492 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.522e+01 8.698e+01 9.400e+01 1.049e+02 1.274e+02, threshold=1.880e+02, percent-clipped=0.0
2023-11-24 06:46:27,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2723853.3333333335, ans=0.125
2023-11-24 06:46:36,838 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11800, loss[loss=0.07736, simple_loss=0.101, pruned_loss=0.01703, audio_tagging_loss=0.009843, over 14365.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09123, pruned_loss=0.01328, audio_tagging_loss=0.009061, over 3064747.99 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:46:45,895 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:46:51,026 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408600
2023-11-24 06:47:40,764 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11850, loss[loss=0.06031, simple_loss=0.07772, pruned_loss=0.008934, audio_tagging_loss=0.01251, over 14896.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.0913, pruned_loss=0.01337, audio_tagging_loss=0.009151, over 3049377.14 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:47:41,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.80 vs. limit=22.5
2023-11-24 06:47:51,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2724320.0, ans=0.0
2023-11-24 06:47:52,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408650
2023-11-24 06:48:03,385 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.550e+01 9.032e+01 9.774e+01 1.343e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 06:48:24,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2724453.3333333335, ans=0.2
2023-11-24 06:48:29,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2724520.0, ans=0.1
2023-11-24 06:48:42,012 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11900, loss[loss=0.07474, simple_loss=0.09503, pruned_loss=0.01591, audio_tagging_loss=0.01131, over 15028.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09142, pruned_loss=0.0133, audio_tagging_loss=0.009155, over 3055389.29 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:48:46,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.79 vs. limit=10.0
2023-11-24 06:48:54,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408700
2023-11-24 06:49:08,871 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.46 vs. limit=12.0
2023-11-24 06:49:25,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2724786.6666666665, ans=0.1
2023-11-24 06:49:27,884 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:49:28,479 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.92 vs. limit=15.0
2023-11-24 06:49:30,853 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.95 vs. limit=22.5
2023-11-24 06:49:37,416 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:49:42,884 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 11950, loss[loss=0.05799, simple_loss=0.07332, pruned_loss=0.01035, audio_tagging_loss=0.01098, over 14394.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09159, pruned_loss=0.01345, audio_tagging_loss=0.009198, over 3055244.85 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:49:44,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2724920.0, ans=0.025
2023-11-24 06:49:56,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408750
2023-11-24 06:50:07,401 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.035e+01 8.293e+01 8.993e+01 9.649e+01 1.275e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 06:50:08,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.46 vs. limit=15.0
2023-11-24 06:50:13,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2725053.3333333335, ans=0.0
2023-11-24 06:50:23,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2725120.0, ans=0.125
2023-11-24 06:50:24,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2725120.0, ans=0.125
2023-11-24 06:50:25,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2725120.0, ans=0.125
2023-11-24 06:50:29,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2725120.0, ans=0.125
2023-11-24 06:50:29,533 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2725120.0, ans=0.125
2023-11-24 06:50:36,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2725186.6666666665, ans=0.0
2023-11-24 06:50:42,949 INFO [train_asr.py:1221] (2/4) Epoch 34, batch 12000, loss[loss=0.07203, simple_loss=0.08879, pruned_loss=0.01855, audio_tagging_loss=0.009083, over 15496.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09125, pruned_loss=0.01334, audio_tagging_loss=0.009252, over 3058647.50 frames. ], batch size: 63, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:50:42,949 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 06:51:14,064 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.8001, 5.8075, 5.8664, 5.8143], device='cuda:2')
2023-11-24 06:51:17,813 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4576, 3.7070, 2.8695, 3.7224], device='cuda:2')
2023-11-24 06:51:25,754 INFO [train_asr.py:1253] (2/4) Epoch 34, validation: loss=0.05837, simple_loss=0.05087, pruned_loss=0.005158, audio_tagging_loss=0.02778, over 4681554.00 frames. 
2023-11-24 06:51:25,755 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 06:51:28,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2725253.3333333335, ans=0.1
2023-11-24 06:51:36,886 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408800
2023-11-24 06:52:25,021 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 0, loss[loss=0.05915, simple_loss=0.05545, pruned_loss=0.006585, audio_tagging_loss=0.02484, over 15597.00 frames. ], tot_loss[loss=0.05915, simple_loss=0.05545, pruned_loss=0.006585, audio_tagging_loss=0.02484, over 15597.00 frames. ], batch size: 62, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:52:25,022 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 06:52:55,741 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.2376, 3.0565, 3.3467, 2.9590, 3.6572, 3.7526, 3.3312, 3.1907],
       device='cuda:2')
2023-11-24 06:53:00,549 INFO [train_asr.py:1253] (2/4) Epoch 35, validation: loss=0.05805, simple_loss=0.05089, pruned_loss=0.005144, audio_tagging_loss=0.02746, over 4681554.00 frames. 
2023-11-24 06:53:00,550 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 06:53:11,953 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.87 vs. limit=12.0
2023-11-24 06:53:15,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2725473.3333333335, ans=0.2
2023-11-24 06:53:24,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2725473.3333333335, ans=0.125
2023-11-24 06:53:47,117 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408850
2023-11-24 06:53:57,723 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.642e+01 8.972e+01 1.010e+02 1.109e+02 1.515e+02, threshold=2.019e+02, percent-clipped=0.0
2023-11-24 06:54:03,119 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 50, loss[loss=0.06541, simple_loss=0.0739, pruned_loss=0.01046, audio_tagging_loss=0.01801, over 16728.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09029, pruned_loss=0.01277, audio_tagging_loss=0.0175, over 686485.27 frames. ], batch size: 63, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:54:09,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2725740.0, ans=0.0
2023-11-24 06:54:24,959 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.03 vs. limit=15.0
2023-11-24 06:54:26,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2725806.6666666665, ans=0.125
2023-11-24 06:54:37,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2725873.3333333335, ans=0.0
2023-11-24 06:54:50,009 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408900
2023-11-24 06:54:56,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0
2023-11-24 06:55:03,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2726006.6666666665, ans=0.07
2023-11-24 06:55:06,614 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 100, loss[loss=0.08513, simple_loss=0.1075, pruned_loss=0.01771, audio_tagging_loss=0.01367, over 15372.00 frames. ], tot_loss[loss=0.07576, simple_loss=0.09155, pruned_loss=0.01353, audio_tagging_loss=0.01646, over 1209576.35 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:55:25,675 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.24 vs. limit=22.5
2023-11-24 06:55:29,534 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.46 vs. limit=22.5
2023-11-24 06:55:53,041 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 408950
2023-11-24 06:55:58,317 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.77 vs. limit=12.0
2023-11-24 06:56:04,089 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.733e+01 9.169e+01 9.656e+01 1.031e+02 1.256e+02, threshold=1.931e+02, percent-clipped=0.0
2023-11-24 06:56:08,793 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 150, loss[loss=0.07608, simple_loss=0.09949, pruned_loss=0.01659, audio_tagging_loss=0.009751, over 15136.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09253, pruned_loss=0.01349, audio_tagging_loss=0.0149, over 1621603.00 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:56:09,420 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.90 vs. limit=12.0
2023-11-24 06:56:11,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2726406.6666666665, ans=0.125
2023-11-24 06:56:21,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2726473.3333333335, ans=0.0
2023-11-24 06:56:21,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2726473.3333333335, ans=0.125
2023-11-24 06:56:24,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2726473.3333333335, ans=0.125
2023-11-24 06:56:34,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2726540.0, ans=0.1
2023-11-24 06:56:34,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2726540.0, ans=0.1
2023-11-24 06:56:54,763 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409000
2023-11-24 06:56:57,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2726673.3333333335, ans=0.0
2023-11-24 06:57:03,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2726673.3333333335, ans=0.0
2023-11-24 06:57:11,055 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 200, loss[loss=0.1019, simple_loss=0.1341, pruned_loss=0.0269, audio_tagging_loss=0.007961, over 16148.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09196, pruned_loss=0.01317, audio_tagging_loss=0.01302, over 1941347.04 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:57:11,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2726740.0, ans=0.1
2023-11-24 06:57:12,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2726740.0, ans=0.0
2023-11-24 06:57:25,784 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.80 vs. limit=10.0
2023-11-24 06:57:49,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2726940.0, ans=0.125
2023-11-24 06:57:51,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2726940.0, ans=0.125
2023-11-24 06:57:56,173 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409050
2023-11-24 06:58:05,855 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2727006.6666666665, ans=0.04949747468305833
2023-11-24 06:58:09,519 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.892e+01 8.427e+01 9.100e+01 9.790e+01 1.266e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 06:58:10,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2727006.6666666665, ans=0.2
2023-11-24 06:58:13,111 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 250, loss[loss=0.07544, simple_loss=0.09901, pruned_loss=0.01695, audio_tagging_loss=0.008978, over 15048.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09242, pruned_loss=0.01329, audio_tagging_loss=0.01178, over 2188654.96 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 06:58:22,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2727073.3333333335, ans=0.125
2023-11-24 06:58:23,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2727140.0, ans=0.0
2023-11-24 06:58:30,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2727140.0, ans=0.125
2023-11-24 06:58:31,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2727140.0, ans=0.09899494936611666
2023-11-24 06:58:59,007 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409100
2023-11-24 06:59:06,765 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.34 vs. limit=15.0
2023-11-24 06:59:14,738 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 300, loss[loss=0.05993, simple_loss=0.07614, pruned_loss=0.01147, audio_tagging_loss=0.01039, over 14800.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09152, pruned_loss=0.01321, audio_tagging_loss=0.01106, over 2378739.78 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 06:59:47,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2727540.0, ans=0.2
2023-11-24 06:59:48,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2727540.0, ans=0.125
2023-11-24 06:59:59,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2727606.6666666665, ans=0.125
2023-11-24 06:59:59,252 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.53 vs. limit=22.5
2023-11-24 07:00:00,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409150
2023-11-24 07:00:13,702 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.065e+01 8.645e+01 9.268e+01 9.915e+01 1.259e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 07:00:16,047 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 350, loss[loss=0.06166, simple_loss=0.08045, pruned_loss=0.01086, audio_tagging_loss=0.01057, over 16326.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09246, pruned_loss=0.01334, audio_tagging_loss=0.01044, over 2531881.86 frames. ], batch size: 61, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:01:02,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409200
2023-11-24 07:01:14,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2728006.6666666665, ans=0.0
2023-11-24 07:01:19,549 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 400, loss[loss=0.05141, simple_loss=0.06314, pruned_loss=0.01225, audio_tagging_loss=0.007594, over 13696.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09194, pruned_loss=0.01331, audio_tagging_loss=0.01013, over 2647384.95 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:01:19,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2728073.3333333335, ans=0.0
2023-11-24 07:01:22,856 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.05 vs. limit=5.0
2023-11-24 07:01:23,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2728073.3333333335, ans=0.025
2023-11-24 07:01:34,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.90 vs. limit=22.5
2023-11-24 07:02:01,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2728273.3333333335, ans=0.0
2023-11-24 07:02:03,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2728273.3333333335, ans=0.125
2023-11-24 07:02:05,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409250
2023-11-24 07:02:13,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2728340.0, ans=0.0
2023-11-24 07:02:18,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.492e+01 8.964e+01 9.627e+01 1.454e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 07:02:21,381 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 450, loss[loss=0.05474, simple_loss=0.07297, pruned_loss=0.006907, audio_tagging_loss=0.01135, over 14285.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.0922, pruned_loss=0.01326, audio_tagging_loss=0.009878, over 2739395.19 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:02:34,703 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.73 vs. limit=10.0
2023-11-24 07:03:06,067 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:03:08,225 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409300
2023-11-24 07:03:20,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2728673.3333333335, ans=0.0
2023-11-24 07:03:24,178 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 500, loss[loss=0.05309, simple_loss=0.06268, pruned_loss=0.008821, audio_tagging_loss=0.01293, over 14623.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09229, pruned_loss=0.01327, audio_tagging_loss=0.009702, over 2805501.38 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:03:42,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2728806.6666666665, ans=0.125
2023-11-24 07:03:46,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2728806.6666666665, ans=0.2
2023-11-24 07:03:59,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.44 vs. limit=22.5
2023-11-24 07:04:10,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409350
2023-11-24 07:04:10,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2728940.0, ans=0.125
2023-11-24 07:04:10,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2728940.0, ans=0.1
2023-11-24 07:04:14,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2729006.6666666665, ans=0.1
2023-11-24 07:04:14,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.85 vs. limit=15.0
2023-11-24 07:04:15,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.48 vs. limit=15.0
2023-11-24 07:04:23,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2729006.6666666665, ans=0.125
2023-11-24 07:04:24,162 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.423e+01 9.068e+01 9.980e+01 1.380e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 07:04:27,223 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 550, loss[loss=0.06132, simple_loss=0.07682, pruned_loss=0.01413, audio_tagging_loss=0.008784, over 16080.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09154, pruned_loss=0.01308, audio_tagging_loss=0.009599, over 2851584.87 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:04:39,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2729140.0, ans=0.0
2023-11-24 07:04:49,002 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.64 vs. limit=15.0
2023-11-24 07:04:52,493 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.30 vs. limit=22.5
2023-11-24 07:04:53,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2729206.6666666665, ans=0.0
2023-11-24 07:04:53,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2729206.6666666665, ans=0.2
2023-11-24 07:04:53,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.76 vs. limit=6.0
2023-11-24 07:05:12,306 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409400
2023-11-24 07:05:26,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2729340.0, ans=0.0
2023-11-24 07:05:28,486 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 600, loss[loss=0.06173, simple_loss=0.07815, pruned_loss=0.01322, audio_tagging_loss=0.009441, over 15537.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09138, pruned_loss=0.01297, audio_tagging_loss=0.009402, over 2890199.13 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:05:57,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2729540.0, ans=0.125
2023-11-24 07:06:05,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2729606.6666666665, ans=0.0
2023-11-24 07:06:14,456 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409450
2023-11-24 07:06:20,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff2.min_abs, batch_count=2729673.3333333335, ans=0.1
2023-11-24 07:06:27,508 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.229e+01 8.791e+01 9.631e+01 1.307e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-24 07:06:27,953 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2729673.3333333335, ans=0.0
2023-11-24 07:06:29,977 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 650, loss[loss=0.06509, simple_loss=0.08398, pruned_loss=0.01132, audio_tagging_loss=0.01178, over 14443.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09096, pruned_loss=0.01305, audio_tagging_loss=0.009369, over 2919805.71 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:06:34,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2729740.0, ans=0.125
2023-11-24 07:06:35,852 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.12 vs. limit=10.0
2023-11-24 07:06:43,945 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.86 vs. limit=15.0
2023-11-24 07:06:59,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2729873.3333333335, ans=0.125
2023-11-24 07:07:10,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2729940.0, ans=0.125
2023-11-24 07:07:11,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2729940.0, ans=0.1
2023-11-24 07:07:14,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2729940.0, ans=0.125
2023-11-24 07:07:15,805 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409500
2023-11-24 07:07:32,350 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=15.0
2023-11-24 07:07:32,854 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 700, loss[loss=0.0612, simple_loss=0.07636, pruned_loss=0.01318, audio_tagging_loss=0.009841, over 14404.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09115, pruned_loss=0.01326, audio_tagging_loss=0.009262, over 2959574.51 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:07:56,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2730206.6666666665, ans=0.1
2023-11-24 07:08:00,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2730206.6666666665, ans=0.0
2023-11-24 07:08:08,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2730273.3333333335, ans=0.0
2023-11-24 07:08:10,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2730273.3333333335, ans=0.125
2023-11-24 07:08:19,024 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409550
2023-11-24 07:08:32,407 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.410e+01 9.143e+01 1.010e+02 1.194e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 07:08:34,805 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 750, loss[loss=0.07614, simple_loss=0.1041, pruned_loss=0.01777, audio_tagging_loss=0.006344, over 15581.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09279, pruned_loss=0.01356, audio_tagging_loss=0.009187, over 2988662.40 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:08:39,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2730406.6666666665, ans=0.0
2023-11-24 07:08:51,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2730473.3333333335, ans=0.1
2023-11-24 07:08:56,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2730473.3333333335, ans=0.0
2023-11-24 07:09:20,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.85 vs. limit=15.0
2023-11-24 07:09:20,977 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409600
2023-11-24 07:09:36,490 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 800, loss[loss=0.06043, simple_loss=0.07594, pruned_loss=0.01186, audio_tagging_loss=0.0106, over 15266.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09223, pruned_loss=0.01358, audio_tagging_loss=0.009179, over 3002253.80 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:09:36,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2730740.0, ans=0.125
2023-11-24 07:10:22,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409650
2023-11-24 07:10:35,978 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.567e+01 9.382e+01 1.008e+02 1.693e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 07:10:38,333 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 850, loss[loss=0.06248, simple_loss=0.08213, pruned_loss=0.01094, audio_tagging_loss=0.01048, over 15487.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09174, pruned_loss=0.01357, audio_tagging_loss=0.009192, over 3011517.71 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:11:08,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-24 07:11:13,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2731206.6666666665, ans=0.125
2023-11-24 07:11:24,096 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409700
2023-11-24 07:11:26,248 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0
2023-11-24 07:11:40,598 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 900, loss[loss=0.05195, simple_loss=0.0693, pruned_loss=0.007011, audio_tagging_loss=0.01029, over 14546.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09006, pruned_loss=0.01321, audio_tagging_loss=0.009483, over 3019061.16 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:11:44,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2731406.6666666665, ans=0.1
2023-11-24 07:11:46,853 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:11:49,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2731406.6666666665, ans=0.05
2023-11-24 07:12:25,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2731606.6666666665, ans=0.125
2023-11-24 07:12:26,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409750
2023-11-24 07:12:36,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.84 vs. limit=22.5
2023-11-24 07:12:39,637 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.786e+01 9.264e+01 9.713e+01 1.175e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 07:12:42,027 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 950, loss[loss=0.05826, simple_loss=0.077, pruned_loss=0.007221, audio_tagging_loss=0.01254, over 14311.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09128, pruned_loss=0.01334, audio_tagging_loss=0.009347, over 3024078.27 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:12:42,630 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.50 vs. limit=22.5
2023-11-24 07:12:43,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2731740.0, ans=0.125
2023-11-24 07:13:01,760 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.64 vs. limit=15.0
2023-11-24 07:13:10,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2731873.3333333335, ans=0.125
2023-11-24 07:13:18,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2731940.0, ans=0.0
2023-11-24 07:13:28,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409800
2023-11-24 07:13:31,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2732006.6666666665, ans=0.125
2023-11-24 07:13:33,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2732006.6666666665, ans=0.5
2023-11-24 07:13:44,713 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1000, loss[loss=0.03814, simple_loss=0.04301, pruned_loss=0.006704, audio_tagging_loss=0.009932, over 15204.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09107, pruned_loss=0.01339, audio_tagging_loss=0.009112, over 3023604.17 frames. ], batch size: 61, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:13:59,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2732140.0, ans=0.1
2023-11-24 07:14:11,505 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:14:31,134 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409850
2023-11-24 07:14:46,887 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.432e+01 9.037e+01 9.609e+01 1.581e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 07:14:48,078 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1050, loss[loss=0.08042, simple_loss=0.1133, pruned_loss=0.01642, audio_tagging_loss=0.007343, over 15347.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09064, pruned_loss=0.01343, audio_tagging_loss=0.009038, over 3030051.06 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:14:59,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2732473.3333333335, ans=0.125
2023-11-24 07:15:08,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2732473.3333333335, ans=0.0
2023-11-24 07:15:08,361 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:15:14,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2732540.0, ans=0.125
2023-11-24 07:15:33,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2732606.6666666665, ans=0.2
2023-11-24 07:15:34,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409900
2023-11-24 07:15:43,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=7.59 vs. limit=15.0
2023-11-24 07:15:47,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.52 vs. limit=22.5
2023-11-24 07:15:49,673 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1100, loss[loss=0.05919, simple_loss=0.07928, pruned_loss=0.0104, audio_tagging_loss=0.009142, over 15210.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09093, pruned_loss=0.01342, audio_tagging_loss=0.008901, over 3031605.12 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:15:50,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2732740.0, ans=22.5
2023-11-24 07:15:52,055 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:15:55,023 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.09 vs. limit=10.0
2023-11-24 07:15:56,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2732740.0, ans=0.125
2023-11-24 07:16:01,890 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=11.25 vs. limit=12.0
2023-11-24 07:16:02,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2732806.6666666665, ans=0.125
2023-11-24 07:16:07,397 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.00 vs. limit=15.0
2023-11-24 07:16:09,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2732806.6666666665, ans=0.1
2023-11-24 07:16:26,560 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2732940.0, ans=0.1
2023-11-24 07:16:32,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.36 vs. limit=15.0
2023-11-24 07:16:35,741 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 409950
2023-11-24 07:16:35,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2732940.0, ans=0.0
2023-11-24 07:16:49,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.554e+01 8.985e+01 9.572e+01 1.203e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 07:16:50,942 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1150, loss[loss=0.0468, simple_loss=0.05642, pruned_loss=0.007683, audio_tagging_loss=0.01091, over 13580.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09168, pruned_loss=0.01356, audio_tagging_loss=0.008787, over 3029679.99 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:17:17,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2733206.6666666665, ans=0.125
2023-11-24 07:17:37,331 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410000
2023-11-24 07:17:46,263 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.94 vs. limit=15.0
2023-11-24 07:17:54,380 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1200, loss[loss=0.06025, simple_loss=0.0717, pruned_loss=0.0132, audio_tagging_loss=0.0112, over 14921.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09118, pruned_loss=0.01343, audio_tagging_loss=0.008768, over 3031701.05 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:17:58,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.55 vs. limit=15.0
2023-11-24 07:18:04,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2733406.6666666665, ans=0.0
2023-11-24 07:18:23,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2733540.0, ans=0.1
2023-11-24 07:18:25,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.05 vs. limit=15.0
2023-11-24 07:18:40,679 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410050
2023-11-24 07:18:55,462 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.712e+01 8.610e+01 9.345e+01 9.890e+01 1.292e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 07:18:56,692 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1250, loss[loss=0.04776, simple_loss=0.06092, pruned_loss=0.006113, audio_tagging_loss=0.01118, over 15022.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09115, pruned_loss=0.01342, audio_tagging_loss=0.00884, over 3037144.46 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:19:02,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2733740.0, ans=0.0
2023-11-24 07:19:08,110 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.49 vs. limit=15.0
2023-11-24 07:19:17,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.03 vs. limit=6.0
2023-11-24 07:19:23,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2733873.3333333335, ans=0.2
2023-11-24 07:19:30,797 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2733873.3333333335, ans=0.07
2023-11-24 07:19:36,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2733940.0, ans=0.1
2023-11-24 07:19:42,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2733940.0, ans=0.09899494936611666
2023-11-24 07:19:43,153 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410100
2023-11-24 07:19:58,434 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1300, loss[loss=0.08035, simple_loss=0.1149, pruned_loss=0.01493, audio_tagging_loss=0.007961, over 16758.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09142, pruned_loss=0.01341, audio_tagging_loss=0.008823, over 3032770.00 frames. ], batch size: 61, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:20:07,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=2734073.3333333335, ans=15.0
2023-11-24 07:20:13,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2734140.0, ans=0.2
2023-11-24 07:20:14,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-24 07:20:25,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2734206.6666666665, ans=0.1
2023-11-24 07:20:45,094 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410150
2023-11-24 07:20:50,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2734340.0, ans=0.125
2023-11-24 07:20:52,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2734340.0, ans=0.125
2023-11-24 07:21:01,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff2.min_abs, batch_count=2734406.6666666665, ans=0.1
2023-11-24 07:21:01,938 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.342e+01 8.312e+01 8.919e+01 9.626e+01 1.151e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 07:21:01,982 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1350, loss[loss=0.05484, simple_loss=0.06843, pruned_loss=0.01015, audio_tagging_loss=0.01048, over 14972.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09047, pruned_loss=0.01326, audio_tagging_loss=0.008959, over 3037793.23 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:21:07,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2734406.6666666665, ans=0.125
2023-11-24 07:21:13,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2734473.3333333335, ans=0.1
2023-11-24 07:21:26,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.54 vs. limit=22.5
2023-11-24 07:21:37,972 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:21:47,020 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:21:48,343 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410200
2023-11-24 07:22:03,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2734740.0, ans=0.0
2023-11-24 07:22:04,502 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1400, loss[loss=0.08293, simple_loss=0.1161, pruned_loss=0.01603, audio_tagging_loss=0.008865, over 16451.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09128, pruned_loss=0.01323, audio_tagging_loss=0.008889, over 3044304.95 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:22:04,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2734740.0, ans=0.0
2023-11-24 07:22:15,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2734806.6666666665, ans=0.125
2023-11-24 07:22:28,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2734873.3333333335, ans=0.1
2023-11-24 07:22:50,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410250
2023-11-24 07:23:06,030 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.310e+01 8.227e+01 8.989e+01 9.587e+01 1.141e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 07:23:06,073 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1450, loss[loss=0.08009, simple_loss=0.1048, pruned_loss=0.01929, audio_tagging_loss=0.008392, over 16000.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09146, pruned_loss=0.01328, audio_tagging_loss=0.008955, over 3043755.31 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:23:12,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2735073.3333333335, ans=0.1
2023-11-24 07:23:31,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2735206.6666666665, ans=0.025
2023-11-24 07:23:35,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2735206.6666666665, ans=0.1
2023-11-24 07:23:37,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2735206.6666666665, ans=0.125
2023-11-24 07:23:47,681 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:23:52,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410300
2023-11-24 07:24:09,055 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1500, loss[loss=0.06013, simple_loss=0.06904, pruned_loss=0.01228, audio_tagging_loss=0.01333, over 14772.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09154, pruned_loss=0.01361, audio_tagging_loss=0.008971, over 3038376.62 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:24:10,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2735406.6666666665, ans=0.125
2023-11-24 07:24:13,335 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.14 vs. limit=15.0
2023-11-24 07:24:15,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2735406.6666666665, ans=0.2
2023-11-24 07:24:18,776 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:24:38,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2735540.0, ans=0.125
2023-11-24 07:24:47,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2735606.6666666665, ans=0.1
2023-11-24 07:24:49,623 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2735606.6666666665, ans=0.125
2023-11-24 07:24:54,966 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410350
2023-11-24 07:25:02,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2735673.3333333335, ans=0.0
2023-11-24 07:25:10,431 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.532e+01 9.153e+01 9.693e+01 1.540e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 07:25:10,497 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1550, loss[loss=0.07805, simple_loss=0.1051, pruned_loss=0.01708, audio_tagging_loss=0.00842, over 15654.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09187, pruned_loss=0.01372, audio_tagging_loss=0.009052, over 3038772.76 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:25:10,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2735740.0, ans=0.1
2023-11-24 07:25:18,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2735740.0, ans=0.0
2023-11-24 07:25:27,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2735806.6666666665, ans=0.0
2023-11-24 07:25:29,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2735806.6666666665, ans=0.125
2023-11-24 07:25:53,949 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2735940.0, ans=0.125
2023-11-24 07:25:57,315 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410400
2023-11-24 07:25:58,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.32 vs. limit=10.0
2023-11-24 07:26:13,377 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1600, loss[loss=0.04793, simple_loss=0.05839, pruned_loss=0.006078, audio_tagging_loss=0.01265, over 15574.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.0916, pruned_loss=0.01366, audio_tagging_loss=0.009066, over 3030462.41 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:26:53,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2736273.3333333335, ans=0.0
2023-11-24 07:26:57,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2736273.3333333335, ans=0.125
2023-11-24 07:26:58,776 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.63 vs. limit=15.0
2023-11-24 07:26:59,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410450
2023-11-24 07:27:03,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2736340.0, ans=0.1
2023-11-24 07:27:13,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2736340.0, ans=0.07
2023-11-24 07:27:15,438 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1650, loss[loss=0.05349, simple_loss=0.07156, pruned_loss=0.009422, audio_tagging_loss=0.008283, over 14030.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09113, pruned_loss=0.01347, audio_tagging_loss=0.009147, over 3025935.19 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:27:17,168 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.572e+01 9.096e+01 9.732e+01 1.252e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 07:27:23,738 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.76 vs. limit=22.5
2023-11-24 07:27:24,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2736406.6666666665, ans=0.0
2023-11-24 07:27:52,700 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.22 vs. limit=15.0
2023-11-24 07:27:54,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2736606.6666666665, ans=0.125
2023-11-24 07:28:01,131 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410500
2023-11-24 07:28:04,487 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.91 vs. limit=15.0
2023-11-24 07:28:14,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2736673.3333333335, ans=0.125
2023-11-24 07:28:15,189 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.72 vs. limit=22.5
2023-11-24 07:28:16,962 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1700, loss[loss=0.07471, simple_loss=0.1001, pruned_loss=0.01561, audio_tagging_loss=0.009052, over 15960.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09138, pruned_loss=0.01345, audio_tagging_loss=0.009144, over 3032955.69 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:28:23,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2736740.0, ans=0.125
2023-11-24 07:28:26,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2736740.0, ans=0.0
2023-11-24 07:28:30,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2736806.6666666665, ans=0.1
2023-11-24 07:28:33,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2736806.6666666665, ans=10.0
2023-11-24 07:28:53,063 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.73 vs. limit=15.0
2023-11-24 07:28:53,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2736940.0, ans=0.2
2023-11-24 07:29:03,150 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410550
2023-11-24 07:29:03,312 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:29:18,498 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1750, loss[loss=0.06419, simple_loss=0.08691, pruned_loss=0.01317, audio_tagging_loss=0.007557, over 14119.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09066, pruned_loss=0.01338, audio_tagging_loss=0.00909, over 3031593.90 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:29:19,637 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.653e+01 8.559e+01 9.195e+01 9.925e+01 1.188e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 07:29:22,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2737073.3333333335, ans=0.125
2023-11-24 07:29:34,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2737140.0, ans=0.0
2023-11-24 07:29:35,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2737140.0, ans=0.025
2023-11-24 07:29:41,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2737140.0, ans=0.1
2023-11-24 07:30:04,455 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410600
2023-11-24 07:30:21,242 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1800, loss[loss=0.05745, simple_loss=0.08118, pruned_loss=0.008616, audio_tagging_loss=0.008243, over 15299.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09096, pruned_loss=0.01335, audio_tagging_loss=0.008942, over 3039976.15 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:30:21,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2737406.6666666665, ans=0.1
2023-11-24 07:30:32,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2737473.3333333335, ans=0.125
2023-11-24 07:30:53,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2737540.0, ans=0.125
2023-11-24 07:30:53,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2737540.0, ans=0.125
2023-11-24 07:31:06,439 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410650
2023-11-24 07:31:20,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2737673.3333333335, ans=0.1
2023-11-24 07:31:21,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2737673.3333333335, ans=0.125
2023-11-24 07:31:23,078 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1850, loss[loss=0.0642, simple_loss=0.08542, pruned_loss=0.01191, audio_tagging_loss=0.009579, over 14442.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09229, pruned_loss=0.01344, audio_tagging_loss=0.00881, over 3035894.36 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:31:25,381 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.283e+01 8.932e+01 9.471e+01 1.128e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 07:31:44,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2737806.6666666665, ans=0.95
2023-11-24 07:31:55,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2737873.3333333335, ans=0.0
2023-11-24 07:32:09,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410700
2023-11-24 07:32:24,339 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1900, loss[loss=0.0673, simple_loss=0.08749, pruned_loss=0.01302, audio_tagging_loss=0.01053, over 14369.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09184, pruned_loss=0.01347, audio_tagging_loss=0.008883, over 3031587.86 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:32:28,116 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2738073.3333333335, ans=0.0
2023-11-24 07:32:32,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2738073.3333333335, ans=0.0
2023-11-24 07:32:58,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2738206.6666666665, ans=0.125
2023-11-24 07:33:02,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2738273.3333333335, ans=0.125
2023-11-24 07:33:10,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410750
2023-11-24 07:33:17,564 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:33:24,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.02 vs. limit=10.0
2023-11-24 07:33:26,570 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 1950, loss[loss=0.08122, simple_loss=0.1068, pruned_loss=0.02035, audio_tagging_loss=0.007488, over 15368.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09062, pruned_loss=0.0134, audio_tagging_loss=0.008872, over 3035793.38 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:33:28,935 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.505e+01 9.149e+01 9.609e+01 1.191e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 07:33:58,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2738540.0, ans=0.1
2023-11-24 07:34:10,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2738606.6666666665, ans=0.2
2023-11-24 07:34:12,042 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410800
2023-11-24 07:34:12,679 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.07 vs. limit=15.0
2023-11-24 07:34:15,394 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.29 vs. limit=10.0
2023-11-24 07:34:16,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2738673.3333333335, ans=0.0
2023-11-24 07:34:21,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2738673.3333333335, ans=0.125
2023-11-24 07:34:21,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2738673.3333333335, ans=0.0
2023-11-24 07:34:28,846 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2000, loss[loss=0.05604, simple_loss=0.06474, pruned_loss=0.01204, audio_tagging_loss=0.01163, over 14714.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09066, pruned_loss=0.01335, audio_tagging_loss=0.008937, over 3039618.62 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:34:56,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2738873.3333333335, ans=0.125
2023-11-24 07:35:00,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2738873.3333333335, ans=0.2
2023-11-24 07:35:15,222 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410850
2023-11-24 07:35:18,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.27 vs. limit=22.5
2023-11-24 07:35:29,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2739073.3333333335, ans=0.09899494936611666
2023-11-24 07:35:30,568 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2050, loss[loss=0.06697, simple_loss=0.09793, pruned_loss=0.01099, audio_tagging_loss=0.007018, over 15382.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09015, pruned_loss=0.01313, audio_tagging_loss=0.008853, over 3032942.74 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:35:32,815 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.370e+01 9.057e+01 9.754e+01 1.235e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 07:35:35,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2739073.3333333335, ans=0.0
2023-11-24 07:36:00,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2739206.6666666665, ans=0.125
2023-11-24 07:36:01,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2739206.6666666665, ans=0.1
2023-11-24 07:36:16,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410900
2023-11-24 07:36:31,811 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2100, loss[loss=0.06461, simple_loss=0.08426, pruned_loss=0.01338, audio_tagging_loss=0.009102, over 16051.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09033, pruned_loss=0.01321, audio_tagging_loss=0.00876, over 3040667.48 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:36:37,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2739406.6666666665, ans=0.1
2023-11-24 07:36:37,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2739406.6666666665, ans=0.0
2023-11-24 07:36:53,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2739473.3333333335, ans=10.0
2023-11-24 07:36:58,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2739540.0, ans=0.1
2023-11-24 07:37:07,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2739540.0, ans=0.09899494936611666
2023-11-24 07:37:12,823 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0
2023-11-24 07:37:18,233 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 410950
2023-11-24 07:37:26,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2739673.3333333335, ans=0.2
2023-11-24 07:37:35,226 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2150, loss[loss=0.08126, simple_loss=0.1083, pruned_loss=0.01944, audio_tagging_loss=0.007656, over 15486.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09098, pruned_loss=0.01334, audio_tagging_loss=0.008717, over 3048961.12 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:37:37,588 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.614e+01 8.763e+01 9.326e+01 1.012e+02 1.387e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 07:37:39,540 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.07 vs. limit=15.0
2023-11-24 07:38:07,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2739873.3333333335, ans=0.0
2023-11-24 07:38:11,451 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:38:21,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411000
2023-11-24 07:38:32,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2740006.6666666665, ans=0.125
2023-11-24 07:38:37,188 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2200, loss[loss=0.08094, simple_loss=0.1094, pruned_loss=0.01747, audio_tagging_loss=0.008762, over 15854.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09171, pruned_loss=0.01349, audio_tagging_loss=0.008688, over 3043483.20 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:39:06,578 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:39:06,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2740206.6666666665, ans=0.125
2023-11-24 07:39:10,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2740206.6666666665, ans=0.95
2023-11-24 07:39:16,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2740273.3333333335, ans=0.125
2023-11-24 07:39:20,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2740273.3333333335, ans=0.1
2023-11-24 07:39:23,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411050
2023-11-24 07:39:33,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2740340.0, ans=0.125
2023-11-24 07:39:38,461 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2250, loss[loss=0.05311, simple_loss=0.06696, pruned_loss=0.007972, audio_tagging_loss=0.01166, over 13622.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09196, pruned_loss=0.01347, audio_tagging_loss=0.008763, over 3043728.64 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:39:40,799 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.314e+01 8.864e+01 9.346e+01 9.896e+01 1.400e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 07:39:48,186 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.68 vs. limit=15.0
2023-11-24 07:39:53,532 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:40:09,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2740540.0, ans=0.125
2023-11-24 07:40:23,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2740606.6666666665, ans=0.0
2023-11-24 07:40:25,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411100
2023-11-24 07:40:34,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2740673.3333333335, ans=0.0
2023-11-24 07:40:39,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2740673.3333333335, ans=0.125
2023-11-24 07:40:42,285 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2300, loss[loss=0.06281, simple_loss=0.08318, pruned_loss=0.01133, audio_tagging_loss=0.009889, over 15091.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.0915, pruned_loss=0.01349, audio_tagging_loss=0.008863, over 3047465.27 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:40:56,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2740806.6666666665, ans=0.125
2023-11-24 07:40:59,250 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0
2023-11-24 07:41:09,911 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.35 vs. limit=15.0
2023-11-24 07:41:21,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2740940.0, ans=0.125
2023-11-24 07:41:28,746 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411150
2023-11-24 07:41:31,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2741006.6666666665, ans=0.0
2023-11-24 07:41:37,506 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:41:44,675 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2350, loss[loss=0.06166, simple_loss=0.08542, pruned_loss=0.009855, audio_tagging_loss=0.009096, over 15260.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09145, pruned_loss=0.01354, audio_tagging_loss=0.00896, over 3049714.06 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:41:47,096 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.561e+01 8.408e+01 9.223e+01 9.902e+01 1.512e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 07:41:54,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2741073.3333333335, ans=0.0
2023-11-24 07:42:00,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2741140.0, ans=0.125
2023-11-24 07:42:05,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2741140.0, ans=0.125
2023-11-24 07:42:05,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2741140.0, ans=0.125
2023-11-24 07:42:23,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2741273.3333333335, ans=0.07
2023-11-24 07:42:27,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2741273.3333333335, ans=0.125
2023-11-24 07:42:31,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411200
2023-11-24 07:42:35,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2741340.0, ans=0.125
2023-11-24 07:42:43,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2741340.0, ans=0.125
2023-11-24 07:42:47,030 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2400, loss[loss=0.07449, simple_loss=0.1072, pruned_loss=0.01284, audio_tagging_loss=0.008054, over 15217.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09093, pruned_loss=0.01339, audio_tagging_loss=0.009095, over 3045761.03 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:42:47,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.27 vs. limit=22.5
2023-11-24 07:42:52,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2741406.6666666665, ans=0.125
2023-11-24 07:42:55,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2741406.6666666665, ans=0.125
2023-11-24 07:43:01,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2741473.3333333335, ans=0.125
2023-11-24 07:43:33,292 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411250
2023-11-24 07:43:49,969 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2450, loss[loss=0.06692, simple_loss=0.09455, pruned_loss=0.01198, audio_tagging_loss=0.007666, over 15091.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09133, pruned_loss=0.01331, audio_tagging_loss=0.009234, over 3045705.93 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:43:52,872 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.523e+01 9.052e+01 9.834e+01 1.481e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 07:44:01,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2741740.0, ans=0.2
2023-11-24 07:44:09,824 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.36 vs. limit=15.0
2023-11-24 07:44:13,056 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.54 vs. limit=15.0
2023-11-24 07:44:21,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2741873.3333333335, ans=0.1
2023-11-24 07:44:36,384 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411300
2023-11-24 07:44:45,590 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:44:53,073 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2500, loss[loss=0.05518, simple_loss=0.07243, pruned_loss=0.009315, audio_tagging_loss=0.009652, over 14826.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09171, pruned_loss=0.01321, audio_tagging_loss=0.009232, over 3052882.91 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:45:28,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2742206.6666666665, ans=0.07
2023-11-24 07:45:39,734 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411350
2023-11-24 07:45:55,245 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2550, loss[loss=0.04446, simple_loss=0.05273, pruned_loss=0.008346, audio_tagging_loss=0.00975, over 13579.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.0913, pruned_loss=0.01309, audio_tagging_loss=0.009114, over 3054184.59 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:45:56,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2742406.6666666665, ans=0.125
2023-11-24 07:45:56,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2742406.6666666665, ans=0.2
2023-11-24 07:45:57,674 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.339e+01 8.522e+01 9.145e+01 9.945e+01 1.198e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 07:46:15,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2742473.3333333335, ans=0.125
2023-11-24 07:46:19,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2742473.3333333335, ans=0.125
2023-11-24 07:46:34,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2742606.6666666665, ans=0.0
2023-11-24 07:46:42,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411400
2023-11-24 07:46:42,785 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.08 vs. limit=15.0
2023-11-24 07:46:47,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2742673.3333333335, ans=0.125
2023-11-24 07:46:48,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2742673.3333333335, ans=0.125
2023-11-24 07:46:58,261 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2600, loss[loss=0.06453, simple_loss=0.09058, pruned_loss=0.01131, audio_tagging_loss=0.007927, over 16191.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09051, pruned_loss=0.01304, audio_tagging_loss=0.009041, over 3051965.02 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:46:58,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2742740.0, ans=0.05
2023-11-24 07:47:03,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.63 vs. limit=22.5
2023-11-24 07:47:07,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2742740.0, ans=0.125
2023-11-24 07:47:17,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2742806.6666666665, ans=0.0
2023-11-24 07:47:23,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2742873.3333333335, ans=0.09899494936611666
2023-11-24 07:47:44,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411450
2023-11-24 07:47:51,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2743006.6666666665, ans=0.125
2023-11-24 07:48:01,088 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2650, loss[loss=0.06225, simple_loss=0.08094, pruned_loss=0.01238, audio_tagging_loss=0.009399, over 15549.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.08981, pruned_loss=0.01304, audio_tagging_loss=0.00895, over 3045519.42 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:48:03,406 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.074e+01 8.476e+01 9.136e+01 9.874e+01 1.198e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 07:48:26,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2743206.6666666665, ans=0.0
2023-11-24 07:48:45,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2743273.3333333335, ans=0.125
2023-11-24 07:48:47,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411500
2023-11-24 07:48:50,360 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-24 07:48:52,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2743340.0, ans=0.0
2023-11-24 07:49:03,300 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2700, loss[loss=0.08984, simple_loss=0.1294, pruned_loss=0.01947, audio_tagging_loss=0.005666, over 14878.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09083, pruned_loss=0.01327, audio_tagging_loss=0.008922, over 3045906.60 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:49:16,435 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.33 vs. limit=15.0
2023-11-24 07:49:18,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2743473.3333333335, ans=10.0
2023-11-24 07:49:33,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2743540.0, ans=0.125
2023-11-24 07:49:38,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2743540.0, ans=15.0
2023-11-24 07:49:46,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2743606.6666666665, ans=0.125
2023-11-24 07:49:49,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411550
2023-11-24 07:49:52,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2743673.3333333335, ans=0.07
2023-11-24 07:50:05,634 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2750, loss[loss=0.06873, simple_loss=0.09784, pruned_loss=0.01225, audio_tagging_loss=0.007563, over 16031.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09111, pruned_loss=0.01329, audio_tagging_loss=0.008908, over 3050583.01 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:50:09,739 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.955e+01 8.410e+01 9.303e+01 9.931e+01 1.664e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 07:50:10,257 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.15 vs. limit=15.0
2023-11-24 07:50:28,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2743806.6666666665, ans=0.1
2023-11-24 07:50:44,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2743940.0, ans=0.0
2023-11-24 07:50:44,463 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:50:51,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411600
2023-11-24 07:50:57,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2744006.6666666665, ans=0.2
2023-11-24 07:50:58,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2744006.6666666665, ans=0.2
2023-11-24 07:50:59,876 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:51:08,610 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2800, loss[loss=0.06545, simple_loss=0.09298, pruned_loss=0.01019, audio_tagging_loss=0.008775, over 14519.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09014, pruned_loss=0.01326, audio_tagging_loss=0.00896, over 3040882.24 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:51:29,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2744140.0, ans=0.04949747468305833
2023-11-24 07:51:47,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2744273.3333333335, ans=0.125
2023-11-24 07:51:53,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2744273.3333333335, ans=0.05
2023-11-24 07:51:54,674 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411650
2023-11-24 07:51:55,176 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.26 vs. limit=15.0
2023-11-24 07:52:03,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2744340.0, ans=0.2
2023-11-24 07:52:10,214 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2850, loss[loss=0.07999, simple_loss=0.1044, pruned_loss=0.01853, audio_tagging_loss=0.009233, over 16024.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09009, pruned_loss=0.01323, audio_tagging_loss=0.008915, over 3034148.57 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:52:14,252 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.630e+01 8.559e+01 8.889e+01 9.637e+01 1.206e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-24 07:52:21,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2744473.3333333335, ans=0.1
2023-11-24 07:52:26,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2744473.3333333335, ans=0.0
2023-11-24 07:52:28,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2744473.3333333335, ans=0.0
2023-11-24 07:52:42,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2744540.0, ans=0.07
2023-11-24 07:52:52,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2744606.6666666665, ans=0.125
2023-11-24 07:52:56,685 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411700
2023-11-24 07:53:12,538 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2900, loss[loss=0.06624, simple_loss=0.07318, pruned_loss=0.01773, audio_tagging_loss=0.01193, over 15168.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09038, pruned_loss=0.01332, audio_tagging_loss=0.008902, over 3037481.81 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:53:12,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2744740.0, ans=0.125
2023-11-24 07:53:38,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2744873.3333333335, ans=0.015
2023-11-24 07:53:38,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2744873.3333333335, ans=0.125
2023-11-24 07:53:49,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.22 vs. limit=15.0
2023-11-24 07:53:59,233 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411750
2023-11-24 07:54:16,223 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 2950, loss[loss=0.07822, simple_loss=0.106, pruned_loss=0.01727, audio_tagging_loss=0.007971, over 15465.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09096, pruned_loss=0.01346, audio_tagging_loss=0.008879, over 3040942.78 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:54:16,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2745073.3333333335, ans=0.125
2023-11-24 07:54:19,688 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.555e+01 8.832e+01 9.434e+01 1.012e+02 1.234e+02, threshold=1.887e+02, percent-clipped=0.0
2023-11-24 07:54:34,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2745140.0, ans=0.125
2023-11-24 07:54:43,872 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.00 vs. limit=10.0
2023-11-24 07:54:49,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2745206.6666666665, ans=0.2
2023-11-24 07:55:02,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411800
2023-11-24 07:55:03,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.78 vs. limit=15.0
2023-11-24 07:55:07,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2745340.0, ans=0.125
2023-11-24 07:55:16,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2745340.0, ans=0.0
2023-11-24 07:55:17,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2745406.6666666665, ans=0.2
2023-11-24 07:55:18,166 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3000, loss[loss=0.06836, simple_loss=0.09833, pruned_loss=0.01246, audio_tagging_loss=0.00673, over 15421.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09101, pruned_loss=0.01343, audio_tagging_loss=0.008945, over 3038881.94 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:55:18,167 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 07:56:00,435 INFO [train_asr.py:1253] (2/4) Epoch 35, validation: loss=0.05789, simple_loss=0.05083, pruned_loss=0.005097, audio_tagging_loss=0.02738, over 4681554.00 frames. 
2023-11-24 07:56:00,435 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 07:56:05,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2745406.6666666665, ans=0.1
2023-11-24 07:56:24,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2745540.0, ans=0.125
2023-11-24 07:56:40,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2745606.6666666665, ans=0.09899494936611666
2023-11-24 07:56:43,630 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.29 vs. limit=15.0
2023-11-24 07:56:46,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411850
2023-11-24 07:57:02,429 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3050, loss[loss=0.06709, simple_loss=0.09454, pruned_loss=0.01033, audio_tagging_loss=0.009488, over 15274.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09095, pruned_loss=0.01327, audio_tagging_loss=0.009021, over 3040465.37 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:57:07,182 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.364e+01 8.656e+01 9.299e+01 1.004e+02 2.054e+02, threshold=1.860e+02, percent-clipped=1.0
2023-11-24 07:57:15,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2745806.6666666665, ans=0.1
2023-11-24 07:57:25,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2745873.3333333335, ans=0.07
2023-11-24 07:57:38,864 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:57:44,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2745940.0, ans=0.1
2023-11-24 07:57:45,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2745940.0, ans=0.125
2023-11-24 07:57:49,053 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411900
2023-11-24 07:57:56,638 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.09 vs. limit=15.0
2023-11-24 07:58:04,305 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3100, loss[loss=0.08434, simple_loss=0.1121, pruned_loss=0.01922, audio_tagging_loss=0.009088, over 15725.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09074, pruned_loss=0.01316, audio_tagging_loss=0.009042, over 3044192.14 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:58:04,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2746073.3333333335, ans=0.125
2023-11-24 07:58:13,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.59 vs. limit=15.0
2023-11-24 07:58:37,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2746206.6666666665, ans=0.125
2023-11-24 07:58:50,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 411950
2023-11-24 07:59:05,905 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3150, loss[loss=0.08046, simple_loss=0.1111, pruned_loss=0.01794, audio_tagging_loss=0.006976, over 15315.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09114, pruned_loss=0.01302, audio_tagging_loss=0.009062, over 3043348.46 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:59:11,716 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.198e+01 8.533e+01 9.313e+01 1.001e+02 1.475e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 07:59:31,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2746540.0, ans=0.125
2023-11-24 07:59:51,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412000
2023-11-24 07:59:53,508 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.47 vs. limit=15.0
2023-11-24 08:00:02,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2746673.3333333335, ans=0.125
2023-11-24 08:00:08,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2746673.3333333335, ans=0.125
2023-11-24 08:00:09,875 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.54 vs. limit=15.0
2023-11-24 08:00:12,758 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3200, loss[loss=0.06952, simple_loss=0.08733, pruned_loss=0.01577, audio_tagging_loss=0.01009, over 15559.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09176, pruned_loss=0.01296, audio_tagging_loss=0.009126, over 3050774.51 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:00:39,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2746873.3333333335, ans=0.0
2023-11-24 08:00:51,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2746940.0, ans=0.1
2023-11-24 08:00:58,735 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412050
2023-11-24 08:01:01,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2747006.6666666665, ans=0.2
2023-11-24 08:01:14,288 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3250, loss[loss=0.05826, simple_loss=0.07223, pruned_loss=0.008991, audio_tagging_loss=0.01315, over 13985.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09198, pruned_loss=0.01298, audio_tagging_loss=0.009232, over 3047933.95 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:01:20,017 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.341e+01 8.971e+01 9.631e+01 1.385e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-24 08:01:38,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2747206.6666666665, ans=0.125
2023-11-24 08:01:43,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.55 vs. limit=6.0
2023-11-24 08:01:52,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2747273.3333333335, ans=0.125
2023-11-24 08:01:55,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2747273.3333333335, ans=10.0
2023-11-24 08:01:59,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2747273.3333333335, ans=0.2
2023-11-24 08:02:00,447 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412100
2023-11-24 08:02:11,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2747340.0, ans=0.125
2023-11-24 08:02:15,750 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3300, loss[loss=0.05093, simple_loss=0.07266, pruned_loss=0.006689, audio_tagging_loss=0.007918, over 15592.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09177, pruned_loss=0.01307, audio_tagging_loss=0.009228, over 3045556.54 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:02:17,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2747406.6666666665, ans=0.1
2023-11-24 08:02:30,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2747473.3333333335, ans=0.2
2023-11-24 08:02:45,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.80 vs. limit=15.0
2023-11-24 08:03:01,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412150
2023-11-24 08:03:09,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.25 vs. limit=15.0
2023-11-24 08:03:19,291 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3350, loss[loss=0.08591, simple_loss=0.1253, pruned_loss=0.01737, audio_tagging_loss=0.005865, over 15461.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09209, pruned_loss=0.01311, audio_tagging_loss=0.009149, over 3045020.28 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:03:21,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2747740.0, ans=0.2
2023-11-24 08:03:25,134 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.567e+01 8.617e+01 9.240e+01 1.018e+02 1.398e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 08:03:45,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2747873.3333333335, ans=0.0
2023-11-24 08:03:48,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2747873.3333333335, ans=0.0
2023-11-24 08:03:50,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2747873.3333333335, ans=0.125
2023-11-24 08:03:56,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2747940.0, ans=0.0
2023-11-24 08:04:04,471 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412200
2023-11-24 08:04:04,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2747940.0, ans=0.125
2023-11-24 08:04:10,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2748006.6666666665, ans=0.1
2023-11-24 08:04:20,964 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3400, loss[loss=0.05844, simple_loss=0.0659, pruned_loss=0.01223, audio_tagging_loss=0.01325, over 14857.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09232, pruned_loss=0.01318, audio_tagging_loss=0.009039, over 3036753.63 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:04:28,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2748073.3333333335, ans=0.0
2023-11-24 08:04:46,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2748206.6666666665, ans=0.0
2023-11-24 08:04:51,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2748206.6666666665, ans=0.0
2023-11-24 08:04:52,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2748206.6666666665, ans=6.0
2023-11-24 08:04:52,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2748206.6666666665, ans=0.1
2023-11-24 08:05:02,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2748273.3333333335, ans=0.2
2023-11-24 08:05:07,299 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412250
2023-11-24 08:05:16,327 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.12 vs. limit=15.0
2023-11-24 08:05:22,629 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3450, loss[loss=0.0658, simple_loss=0.09405, pruned_loss=0.008248, audio_tagging_loss=0.01053, over 16291.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09263, pruned_loss=0.01342, audio_tagging_loss=0.008908, over 3040982.65 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:05:25,419 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.99 vs. limit=22.5
2023-11-24 08:05:28,835 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 8.590e+01 9.237e+01 1.003e+02 1.214e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 08:05:42,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2748473.3333333335, ans=0.025
2023-11-24 08:06:08,527 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412300
2023-11-24 08:06:10,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2748606.6666666665, ans=0.125
2023-11-24 08:06:20,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.86 vs. limit=12.0
2023-11-24 08:06:25,686 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3500, loss[loss=0.05903, simple_loss=0.06956, pruned_loss=0.01494, audio_tagging_loss=0.009306, over 15985.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.0919, pruned_loss=0.01337, audio_tagging_loss=0.008768, over 3044897.06 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:06:28,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2748740.0, ans=0.1
2023-11-24 08:06:29,780 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.37 vs. limit=15.0
2023-11-24 08:06:46,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2748806.6666666665, ans=15.0
2023-11-24 08:06:57,235 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:06:57,992 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.24 vs. limit=22.5
2023-11-24 08:07:12,220 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412350
2023-11-24 08:07:12,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2748940.0, ans=0.125
2023-11-24 08:07:18,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2749006.6666666665, ans=0.125
2023-11-24 08:07:26,261 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.25 vs. limit=15.0
2023-11-24 08:07:27,855 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3550, loss[loss=0.08624, simple_loss=0.1251, pruned_loss=0.01619, audio_tagging_loss=0.007486, over 15785.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09156, pruned_loss=0.01328, audio_tagging_loss=0.008758, over 3046280.45 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:07:33,739 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.276e+01 8.953e+01 9.644e+01 1.310e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 08:07:43,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2749140.0, ans=0.125
2023-11-24 08:07:55,436 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2749206.6666666665, ans=0.125
2023-11-24 08:08:11,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2749273.3333333335, ans=0.125
2023-11-24 08:08:11,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2749273.3333333335, ans=0.0
2023-11-24 08:08:14,268 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412400
2023-11-24 08:08:15,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2749273.3333333335, ans=0.1
2023-11-24 08:08:30,088 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3600, loss[loss=0.06462, simple_loss=0.08017, pruned_loss=0.01539, audio_tagging_loss=0.009146, over 15803.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09055, pruned_loss=0.01308, audio_tagging_loss=0.008768, over 3047342.30 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:08:32,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2749406.6666666665, ans=0.0
2023-11-24 08:08:35,754 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=15.0
2023-11-24 08:08:58,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2749540.0, ans=0.0
2023-11-24 08:09:16,475 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412450
2023-11-24 08:09:19,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2749673.3333333335, ans=0.07
2023-11-24 08:09:33,399 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3650, loss[loss=0.0649, simple_loss=0.0876, pruned_loss=0.01248, audio_tagging_loss=0.008618, over 16314.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09136, pruned_loss=0.01324, audio_tagging_loss=0.008657, over 3050150.30 frames. ], batch size: 62, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:09:39,305 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.298e+01 8.982e+01 9.715e+01 1.238e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 08:09:43,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2749740.0, ans=0.125
2023-11-24 08:09:51,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2749806.6666666665, ans=0.0
2023-11-24 08:10:16,085 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.10 vs. limit=15.0
2023-11-24 08:10:19,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412500
2023-11-24 08:10:32,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2750006.6666666665, ans=0.125
2023-11-24 08:10:35,328 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3700, loss[loss=0.05858, simple_loss=0.07144, pruned_loss=0.01165, audio_tagging_loss=0.01121, over 15604.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09242, pruned_loss=0.01338, audio_tagging_loss=0.008614, over 3048627.00 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:10:49,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2750140.0, ans=0.2
2023-11-24 08:11:20,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2750273.3333333335, ans=0.125
2023-11-24 08:11:21,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412550
2023-11-24 08:11:37,811 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3750, loss[loss=0.05572, simple_loss=0.06958, pruned_loss=0.01, audio_tagging_loss=0.01093, over 15083.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09217, pruned_loss=0.01339, audio_tagging_loss=0.008745, over 3047968.99 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:11:39,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2750406.6666666665, ans=0.125
2023-11-24 08:11:44,955 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.608e+01 8.598e+01 9.101e+01 9.921e+01 1.299e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 08:11:58,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2750473.3333333335, ans=0.125
2023-11-24 08:12:03,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2750540.0, ans=0.125
2023-11-24 08:12:04,757 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:12:04,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2750540.0, ans=0.0
2023-11-24 08:12:07,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2750540.0, ans=10.0
2023-11-24 08:12:10,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2750540.0, ans=0.07
2023-11-24 08:12:12,442 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.57 vs. limit=15.0
2023-11-24 08:12:20,790 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:12:24,316 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412600
2023-11-24 08:12:37,422 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.93 vs. limit=10.0
2023-11-24 08:12:40,936 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3800, loss[loss=0.07548, simple_loss=0.1076, pruned_loss=0.01496, audio_tagging_loss=0.006725, over 16173.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09279, pruned_loss=0.01351, audio_tagging_loss=0.008811, over 3059062.45 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:13:00,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2750806.6666666665, ans=0.125
2023-11-24 08:13:11,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2750873.3333333335, ans=0.125
2023-11-24 08:13:17,204 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.77 vs. limit=15.0
2023-11-24 08:13:19,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2750940.0, ans=0.07
2023-11-24 08:13:24,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.23 vs. limit=22.5
2023-11-24 08:13:27,032 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412650
2023-11-24 08:13:31,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.29 vs. limit=15.0
2023-11-24 08:13:43,140 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3850, loss[loss=0.0521, simple_loss=0.06786, pruned_loss=0.005375, audio_tagging_loss=0.0128, over 15477.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09179, pruned_loss=0.01332, audio_tagging_loss=0.008944, over 3056400.54 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:13:50,160 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.566e+01 8.505e+01 9.405e+01 9.842e+01 1.302e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-24 08:13:55,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2751140.0, ans=0.0
2023-11-24 08:14:13,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2751206.6666666665, ans=0.125
2023-11-24 08:14:18,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2751206.6666666665, ans=0.0
2023-11-24 08:14:21,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2751273.3333333335, ans=0.0
2023-11-24 08:14:24,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2751273.3333333335, ans=0.0
2023-11-24 08:14:29,310 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412700
2023-11-24 08:14:32,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2751340.0, ans=0.1
2023-11-24 08:14:39,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2751340.0, ans=0.0
2023-11-24 08:14:45,020 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3900, loss[loss=0.09173, simple_loss=0.1323, pruned_loss=0.02006, audio_tagging_loss=0.005502, over 15527.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09123, pruned_loss=0.01314, audio_tagging_loss=0.009003, over 3049638.23 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:15:05,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2751473.3333333335, ans=0.125
2023-11-24 08:15:06,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2751473.3333333335, ans=0.0
2023-11-24 08:15:08,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2751473.3333333335, ans=0.2
2023-11-24 08:15:10,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2751540.0, ans=0.125
2023-11-24 08:15:11,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2751540.0, ans=0.125
2023-11-24 08:15:18,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2751540.0, ans=0.2
2023-11-24 08:15:30,535 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412750
2023-11-24 08:15:35,943 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:15:37,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2751673.3333333335, ans=0.125
2023-11-24 08:15:38,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2751673.3333333335, ans=0.2
2023-11-24 08:15:47,560 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 3950, loss[loss=0.07739, simple_loss=0.09991, pruned_loss=0.01648, audio_tagging_loss=0.01096, over 15635.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09116, pruned_loss=0.01316, audio_tagging_loss=0.009121, over 3051922.56 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:15:55,235 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.568e+01 8.460e+01 9.070e+01 9.889e+01 1.315e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 08:16:13,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2751873.3333333335, ans=0.125
2023-11-24 08:16:14,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2751873.3333333335, ans=0.0
2023-11-24 08:16:30,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2751940.0, ans=0.2
2023-11-24 08:16:33,314 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412800
2023-11-24 08:16:42,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2752006.6666666665, ans=0.125
2023-11-24 08:16:50,257 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4000, loss[loss=0.04262, simple_loss=0.05328, pruned_loss=0.006511, audio_tagging_loss=0.009467, over 15966.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09127, pruned_loss=0.01324, audio_tagging_loss=0.009179, over 3048688.11 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:16:51,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2752073.3333333335, ans=10.0
2023-11-24 08:16:57,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2752073.3333333335, ans=0.125
2023-11-24 08:17:05,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2752140.0, ans=0.125
2023-11-24 08:17:16,813 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:17:36,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412850
2023-11-24 08:17:51,858 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4050, loss[loss=0.05709, simple_loss=0.07279, pruned_loss=0.009253, audio_tagging_loss=0.01144, over 14793.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09077, pruned_loss=0.01299, audio_tagging_loss=0.009235, over 3050274.20 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:17:54,221 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:17:58,906 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.632e+01 8.603e+01 9.245e+01 9.992e+01 1.368e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 08:18:01,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.84 vs. limit=15.0
2023-11-24 08:18:02,567 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.73 vs. limit=22.5
2023-11-24 08:18:27,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2752540.0, ans=0.0
2023-11-24 08:18:27,763 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.72 vs. limit=22.5
2023-11-24 08:18:38,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412900
2023-11-24 08:18:39,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2752606.6666666665, ans=0.2
2023-11-24 08:18:39,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2752606.6666666665, ans=0.2
2023-11-24 08:18:48,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.73 vs. limit=22.5
2023-11-24 08:18:54,296 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4100, loss[loss=0.06082, simple_loss=0.08123, pruned_loss=0.01071, audio_tagging_loss=0.009492, over 14771.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09161, pruned_loss=0.01311, audio_tagging_loss=0.009142, over 3046335.24 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:19:31,373 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-24 08:19:35,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2752940.0, ans=0.125
2023-11-24 08:19:38,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2752940.0, ans=0.125
2023-11-24 08:19:41,085 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 412950
2023-11-24 08:19:45,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2753006.6666666665, ans=0.0
2023-11-24 08:19:57,687 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4150, loss[loss=0.06697, simple_loss=0.09156, pruned_loss=0.01292, audio_tagging_loss=0.008268, over 16086.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09192, pruned_loss=0.01326, audio_tagging_loss=0.009041, over 3045185.99 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:20:00,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=2753073.3333333335, ans=0.02
2023-11-24 08:20:04,895 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.354e+01 9.149e+01 1.001e+02 1.406e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 08:20:09,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2753140.0, ans=0.2
2023-11-24 08:20:16,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2753140.0, ans=0.125
2023-11-24 08:20:42,421 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.36 vs. limit=15.0
2023-11-24 08:20:43,162 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:20:43,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2753273.3333333335, ans=0.0
2023-11-24 08:20:44,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413000
2023-11-24 08:20:49,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=2753340.0, ans=0.1
2023-11-24 08:20:54,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2753340.0, ans=0.1
2023-11-24 08:20:59,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=15.0
2023-11-24 08:21:00,132 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4200, loss[loss=0.08676, simple_loss=0.1175, pruned_loss=0.0197, audio_tagging_loss=0.008304, over 15390.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09229, pruned_loss=0.0133, audio_tagging_loss=0.00889, over 3047376.99 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:21:08,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2753406.6666666665, ans=0.125
2023-11-24 08:21:12,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2753473.3333333335, ans=0.125
2023-11-24 08:21:35,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2753540.0, ans=0.125
2023-11-24 08:21:46,638 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413050
2023-11-24 08:22:01,973 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4250, loss[loss=0.05741, simple_loss=0.07982, pruned_loss=0.008803, audio_tagging_loss=0.0087, over 15391.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09174, pruned_loss=0.01324, audio_tagging_loss=0.008796, over 3049132.59 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:22:11,989 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.341e+01 8.897e+01 9.785e+01 1.363e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-24 08:22:37,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2753873.3333333335, ans=0.0
2023-11-24 08:22:42,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2753940.0, ans=0.2
2023-11-24 08:22:47,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.whiten.whitening_limit, batch_count=2753940.0, ans=15.0
2023-11-24 08:22:48,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413100
2023-11-24 08:23:05,756 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4300, loss[loss=0.07646, simple_loss=0.1089, pruned_loss=0.01391, audio_tagging_loss=0.008086, over 16012.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09266, pruned_loss=0.0134, audio_tagging_loss=0.00874, over 3054907.53 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:23:14,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2754073.3333333335, ans=0.1
2023-11-24 08:23:19,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.81 vs. limit=6.0
2023-11-24 08:23:52,246 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413150
2023-11-24 08:24:07,334 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4350, loss[loss=0.06009, simple_loss=0.0812, pruned_loss=0.01119, audio_tagging_loss=0.008305, over 15583.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.0926, pruned_loss=0.01349, audio_tagging_loss=0.008689, over 3055129.12 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:24:11,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2754406.6666666665, ans=0.125
2023-11-24 08:24:15,734 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.072e+01 8.729e+01 9.185e+01 1.017e+02 1.222e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 08:24:19,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2754473.3333333335, ans=0.125
2023-11-24 08:24:22,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2754473.3333333335, ans=0.1
2023-11-24 08:24:41,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2754540.0, ans=0.0
2023-11-24 08:24:53,672 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413200
2023-11-24 08:25:07,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.99 vs. limit=10.0
2023-11-24 08:25:09,259 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4400, loss[loss=0.06172, simple_loss=0.08461, pruned_loss=0.009831, audio_tagging_loss=0.009584, over 14887.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09268, pruned_loss=0.0135, audio_tagging_loss=0.008723, over 3045994.74 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:25:18,060 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.28 vs. limit=22.5
2023-11-24 08:25:36,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2754873.3333333335, ans=0.0
2023-11-24 08:25:43,762 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.53 vs. limit=15.0
2023-11-24 08:25:49,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2754940.0, ans=0.025
2023-11-24 08:25:55,010 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413250
2023-11-24 08:25:55,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2754940.0, ans=10.0
2023-11-24 08:26:12,567 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4450, loss[loss=0.0551, simple_loss=0.0784, pruned_loss=0.008252, audio_tagging_loss=0.007648, over 14390.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09267, pruned_loss=0.01347, audio_tagging_loss=0.00878, over 3043467.10 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:26:20,941 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.416e+01 8.955e+01 9.921e+01 1.330e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 08:26:43,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.44 vs. limit=15.0
2023-11-24 08:26:53,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2755273.3333333335, ans=0.125
2023-11-24 08:26:58,590 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413300
2023-11-24 08:27:14,668 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4500, loss[loss=0.07432, simple_loss=0.09072, pruned_loss=0.01738, audio_tagging_loss=0.01158, over 16820.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09282, pruned_loss=0.01334, audio_tagging_loss=0.008762, over 3054590.69 frames. ], batch size: 65, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:27:18,782 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.97 vs. limit=22.5
2023-11-24 08:27:25,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2755473.3333333335, ans=0.0
2023-11-24 08:27:36,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2755473.3333333335, ans=0.125
2023-11-24 08:27:47,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2755540.0, ans=0.125
2023-11-24 08:27:57,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2755606.6666666665, ans=0.2
2023-11-24 08:28:00,502 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413350
2023-11-24 08:28:15,966 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4550, loss[loss=0.07688, simple_loss=0.1092, pruned_loss=0.0146, audio_tagging_loss=0.007681, over 15821.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09223, pruned_loss=0.01333, audio_tagging_loss=0.00885, over 3058639.51 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:28:23,729 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5
2023-11-24 08:28:24,739 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.054e+01 8.588e+01 9.174e+01 1.003e+02 1.201e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-24 08:28:47,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2755873.3333333335, ans=0.0
2023-11-24 08:28:49,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-24 08:29:02,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413400
2023-11-24 08:29:04,026 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:29:06,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2756006.6666666665, ans=0.125
2023-11-24 08:29:07,526 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.91 vs. limit=15.0
2023-11-24 08:29:13,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2756006.6666666665, ans=0.1
2023-11-24 08:29:19,841 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4600, loss[loss=0.08441, simple_loss=0.1067, pruned_loss=0.0234, audio_tagging_loss=0.007641, over 14872.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09187, pruned_loss=0.01325, audio_tagging_loss=0.00899, over 3055857.00 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:29:20,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2756073.3333333335, ans=0.0
2023-11-24 08:29:38,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.76 vs. limit=12.0
2023-11-24 08:29:43,174 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.41 vs. limit=12.0
2023-11-24 08:29:47,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2756206.6666666665, ans=0.1
2023-11-24 08:29:54,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2756206.6666666665, ans=0.2
2023-11-24 08:30:06,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413450
2023-11-24 08:30:13,138 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2756340.0, ans=0.04949747468305833
2023-11-24 08:30:17,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2756340.0, ans=0.2
2023-11-24 08:30:22,933 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4650, loss[loss=0.05315, simple_loss=0.06325, pruned_loss=0.01027, audio_tagging_loss=0.01125, over 14519.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09075, pruned_loss=0.01324, audio_tagging_loss=0.009208, over 3047513.40 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:30:30,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2756406.6666666665, ans=0.1
2023-11-24 08:30:31,176 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.223e+01 8.404e+01 8.988e+01 9.502e+01 1.265e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 08:30:45,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2756540.0, ans=0.2
2023-11-24 08:30:49,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2756540.0, ans=0.2
2023-11-24 08:31:08,784 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413500
2023-11-24 08:31:10,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2756606.6666666665, ans=0.0
2023-11-24 08:31:13,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2756673.3333333335, ans=0.125
2023-11-24 08:31:22,225 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.49 vs. limit=15.0
2023-11-24 08:31:24,024 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4700, loss[loss=0.068, simple_loss=0.08521, pruned_loss=0.01739, audio_tagging_loss=0.008009, over 14945.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.08986, pruned_loss=0.01308, audio_tagging_loss=0.009266, over 3040614.39 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:31:34,297 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.76 vs. limit=6.0
2023-11-24 08:31:40,168 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.08 vs. limit=22.5
2023-11-24 08:31:43,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2756806.6666666665, ans=0.0
2023-11-24 08:31:58,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2756873.3333333335, ans=0.0
2023-11-24 08:32:04,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2756940.0, ans=0.125
2023-11-24 08:32:10,212 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413550
2023-11-24 08:32:26,986 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4750, loss[loss=0.0531, simple_loss=0.06453, pruned_loss=0.0116, audio_tagging_loss=0.009233, over 13274.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09006, pruned_loss=0.01319, audio_tagging_loss=0.009275, over 3046744.96 frames. ], batch size: 51, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:32:33,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2757073.3333333335, ans=0.125
2023-11-24 08:32:38,223 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.376e+01 8.531e+01 9.142e+01 9.868e+01 1.183e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 08:33:13,232 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413600
2023-11-24 08:33:18,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2757340.0, ans=0.125
2023-11-24 08:33:18,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2757340.0, ans=0.2
2023-11-24 08:33:23,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2757340.0, ans=0.07
2023-11-24 08:33:25,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2757340.0, ans=0.125
2023-11-24 08:33:29,359 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4800, loss[loss=0.08912, simple_loss=0.1206, pruned_loss=0.02224, audio_tagging_loss=0.006595, over 15778.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09092, pruned_loss=0.01338, audio_tagging_loss=0.009303, over 3049906.79 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:33:34,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2757406.6666666665, ans=0.0
2023-11-24 08:33:53,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2757540.0, ans=0.125
2023-11-24 08:33:58,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2757540.0, ans=0.125
2023-11-24 08:33:58,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.71 vs. limit=22.5
2023-11-24 08:34:00,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2757540.0, ans=0.05
2023-11-24 08:34:14,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2757606.6666666665, ans=0.0
2023-11-24 08:34:15,803 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413650
2023-11-24 08:34:15,886 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff2.min_abs, batch_count=2757606.6666666665, ans=0.1
2023-11-24 08:34:30,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2757740.0, ans=0.1
2023-11-24 08:34:31,804 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4850, loss[loss=0.05796, simple_loss=0.07834, pruned_loss=0.009932, audio_tagging_loss=0.008862, over 15308.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09139, pruned_loss=0.0134, audio_tagging_loss=0.009388, over 3049354.10 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:34:41,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2757740.0, ans=0.2
2023-11-24 08:34:42,341 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.773e+01 9.268e+01 1.019e+02 1.463e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 08:35:03,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2757873.3333333335, ans=10.0
2023-11-24 08:35:18,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413700
2023-11-24 08:35:18,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2757940.0, ans=0.125
2023-11-24 08:35:23,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2758006.6666666665, ans=0.125
2023-11-24 08:35:34,418 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4900, loss[loss=0.06399, simple_loss=0.07951, pruned_loss=0.01314, audio_tagging_loss=0.01109, over 15068.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09084, pruned_loss=0.01338, audio_tagging_loss=0.009315, over 3045327.95 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:35:51,926 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.79 vs. limit=15.0
2023-11-24 08:36:19,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2758273.3333333335, ans=0.125
2023-11-24 08:36:20,909 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413750
2023-11-24 08:36:29,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2758340.0, ans=0.125
2023-11-24 08:36:37,231 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 4950, loss[loss=0.07201, simple_loss=0.09834, pruned_loss=0.01414, audio_tagging_loss=0.008697, over 14676.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09065, pruned_loss=0.01328, audio_tagging_loss=0.009226, over 3034797.39 frames. ], batch size: 52, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:36:47,868 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.724e+01 8.295e+01 8.995e+01 9.901e+01 1.240e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 08:36:49,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2758473.3333333335, ans=0.0
2023-11-24 08:36:50,731 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0
2023-11-24 08:37:03,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2758540.0, ans=0.1
2023-11-24 08:37:24,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413800
2023-11-24 08:37:34,699 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.08 vs. limit=15.0
2023-11-24 08:37:39,928 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5000, loss[loss=0.05333, simple_loss=0.06825, pruned_loss=0.01172, audio_tagging_loss=0.007485, over 14697.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09058, pruned_loss=0.0132, audio_tagging_loss=0.008999, over 3031644.15 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:38:26,166 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413850
2023-11-24 08:38:26,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2758940.0, ans=0.125
2023-11-24 08:38:29,238 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.49 vs. limit=15.0
2023-11-24 08:38:32,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2759006.6666666665, ans=0.125
2023-11-24 08:38:42,456 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5050, loss[loss=0.0577, simple_loss=0.07657, pruned_loss=0.01037, audio_tagging_loss=0.00905, over 15620.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.08999, pruned_loss=0.01313, audio_tagging_loss=0.008968, over 3034289.84 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:38:54,964 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.375e+01 9.094e+01 9.590e+01 1.351e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 08:38:57,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2759140.0, ans=0.125
2023-11-24 08:38:58,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2759140.0, ans=0.0
2023-11-24 08:38:58,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2759140.0, ans=0.2
2023-11-24 08:39:24,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.38 vs. limit=12.0
2023-11-24 08:39:27,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2759273.3333333335, ans=0.0
2023-11-24 08:39:28,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413900
2023-11-24 08:39:42,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2759340.0, ans=0.0
2023-11-24 08:39:45,561 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5100, loss[loss=0.07011, simple_loss=0.09281, pruned_loss=0.01597, audio_tagging_loss=0.007743, over 15124.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.0895, pruned_loss=0.01302, audio_tagging_loss=0.008962, over 3038192.64 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:40:07,586 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.11 vs. limit=15.0
2023-11-24 08:40:31,756 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 413950
2023-11-24 08:40:46,922 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5150, loss[loss=0.0754, simple_loss=0.0907, pruned_loss=0.02104, audio_tagging_loss=0.009014, over 13660.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.08953, pruned_loss=0.01281, audio_tagging_loss=0.008893, over 3034687.55 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:40:58,159 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.417e+01 9.000e+01 9.770e+01 1.432e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 08:41:07,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2759806.6666666665, ans=0.0
2023-11-24 08:41:11,479 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2759873.3333333335, ans=0.125
2023-11-24 08:41:22,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2759873.3333333335, ans=0.1
2023-11-24 08:41:28,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2759940.0, ans=0.1
2023-11-24 08:41:33,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414000
2023-11-24 08:41:34,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2759940.0, ans=0.0
2023-11-24 08:41:39,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2760006.6666666665, ans=0.0
2023-11-24 08:41:49,201 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5200, loss[loss=0.07561, simple_loss=0.09571, pruned_loss=0.01704, audio_tagging_loss=0.01071, over 15275.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09083, pruned_loss=0.01321, audio_tagging_loss=0.008817, over 3032165.91 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:41:58,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2760073.3333333335, ans=0.125
2023-11-24 08:42:35,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414050
2023-11-24 08:42:36,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2760273.3333333335, ans=0.125
2023-11-24 08:42:38,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2760340.0, ans=0.0
2023-11-24 08:42:39,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.90 vs. limit=22.5
2023-11-24 08:42:52,956 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5250, loss[loss=0.06155, simple_loss=0.08037, pruned_loss=0.01277, audio_tagging_loss=0.008593, over 14693.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09132, pruned_loss=0.0132, audio_tagging_loss=0.008802, over 3038423.39 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:43:03,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.495e+01 9.165e+01 9.828e+01 1.218e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 08:43:08,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2760473.3333333335, ans=0.0
2023-11-24 08:43:12,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2760473.3333333335, ans=0.125
2023-11-24 08:43:35,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2760606.6666666665, ans=0.09899494936611666
2023-11-24 08:43:39,119 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414100
2023-11-24 08:43:45,502 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0
2023-11-24 08:43:51,569 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.12 vs. limit=15.0
2023-11-24 08:43:54,375 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5300, loss[loss=0.06502, simple_loss=0.09234, pruned_loss=0.0113, audio_tagging_loss=0.007549, over 14558.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09194, pruned_loss=0.0131, audio_tagging_loss=0.008747, over 3035831.54 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:44:26,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2760873.3333333335, ans=0.2
2023-11-24 08:44:40,578 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414150
2023-11-24 08:44:52,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2761006.6666666665, ans=0.0
2023-11-24 08:44:53,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2761006.6666666665, ans=0.0
2023-11-24 08:44:56,018 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5350, loss[loss=0.07915, simple_loss=0.1129, pruned_loss=0.01592, audio_tagging_loss=0.00677, over 15752.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09285, pruned_loss=0.01329, audio_tagging_loss=0.008723, over 3035334.25 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:44:56,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2761073.3333333335, ans=0.125
2023-11-24 08:45:07,827 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.034e+01 8.463e+01 9.143e+01 9.837e+01 1.383e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 08:45:09,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2761140.0, ans=0.125
2023-11-24 08:45:19,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2761140.0, ans=0.025
2023-11-24 08:45:22,693 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2761206.6666666665, ans=0.0
2023-11-24 08:45:33,810 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.30 vs. limit=6.0
2023-11-24 08:45:41,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414200
2023-11-24 08:45:58,641 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5400, loss[loss=0.08874, simple_loss=0.1273, pruned_loss=0.01903, audio_tagging_loss=0.006035, over 14635.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09286, pruned_loss=0.01326, audio_tagging_loss=0.008796, over 3039224.52 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:46:02,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2761406.6666666665, ans=0.125
2023-11-24 08:46:44,177 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414250
2023-11-24 08:46:48,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2761673.3333333335, ans=0.2
2023-11-24 08:46:59,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2761740.0, ans=0.1
2023-11-24 08:47:00,098 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5450, loss[loss=0.07577, simple_loss=0.09817, pruned_loss=0.01756, audio_tagging_loss=0.009124, over 15282.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09245, pruned_loss=0.01316, audio_tagging_loss=0.008866, over 3041559.13 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:47:11,909 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.777e+01 9.198e+01 9.795e+01 1.303e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 08:47:13,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2761806.6666666665, ans=0.1
2023-11-24 08:47:46,420 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414300
2023-11-24 08:47:51,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2762006.6666666665, ans=0.125
2023-11-24 08:47:59,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2762006.6666666665, ans=0.035
2023-11-24 08:48:01,961 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5500, loss[loss=0.05417, simple_loss=0.07618, pruned_loss=0.008617, audio_tagging_loss=0.007461, over 16261.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09262, pruned_loss=0.01312, audio_tagging_loss=0.008939, over 3041612.39 frames. ], batch size: 63, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:48:06,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2762073.3333333335, ans=0.125
2023-11-24 08:48:38,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.29 vs. limit=15.0
2023-11-24 08:48:48,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414350
2023-11-24 08:48:56,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2762340.0, ans=0.125
2023-11-24 08:49:00,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2762340.0, ans=0.1
2023-11-24 08:49:05,649 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5550, loss[loss=0.1063, simple_loss=0.1556, pruned_loss=0.02275, audio_tagging_loss=0.005674, over 15201.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09131, pruned_loss=0.01302, audio_tagging_loss=0.009057, over 3046493.27 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:49:11,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2762406.6666666665, ans=0.0
2023-11-24 08:49:17,949 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.668e+01 8.369e+01 9.115e+01 9.992e+01 1.269e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 08:49:28,059 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.29 vs. limit=22.5
2023-11-24 08:49:37,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2762540.0, ans=0.2
2023-11-24 08:49:51,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414400
2023-11-24 08:49:52,313 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.78 vs. limit=10.0
2023-11-24 08:49:54,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2762673.3333333335, ans=0.1
2023-11-24 08:50:08,636 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5600, loss[loss=0.09247, simple_loss=0.1335, pruned_loss=0.01896, audio_tagging_loss=0.006773, over 15182.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09126, pruned_loss=0.01297, audio_tagging_loss=0.009105, over 3039120.85 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:50:14,641 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:50:20,866 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-24 08:50:29,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2762806.6666666665, ans=0.1
2023-11-24 08:50:53,449 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:50:54,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414450
2023-11-24 08:50:54,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2762940.0, ans=0.0
2023-11-24 08:50:55,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2762940.0, ans=0.1
2023-11-24 08:51:02,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2763006.6666666665, ans=0.125
2023-11-24 08:51:10,082 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5650, loss[loss=0.06371, simple_loss=0.08261, pruned_loss=0.01105, audio_tagging_loss=0.01136, over 13917.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09066, pruned_loss=0.0129, audio_tagging_loss=0.009118, over 3044304.80 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:51:17,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2763073.3333333335, ans=0.0
2023-11-24 08:51:22,327 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.229e+01 8.298e+01 8.960e+01 9.539e+01 1.200e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 08:51:45,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=2763206.6666666665, ans=0.1
2023-11-24 08:51:47,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2763273.3333333335, ans=0.0
2023-11-24 08:51:52,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2763273.3333333335, ans=0.0
2023-11-24 08:51:56,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414500
2023-11-24 08:51:58,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2763340.0, ans=0.95
2023-11-24 08:52:03,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2763340.0, ans=0.125
2023-11-24 08:52:10,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2763340.0, ans=0.125
2023-11-24 08:52:12,848 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5700, loss[loss=0.09099, simple_loss=0.1184, pruned_loss=0.02343, audio_tagging_loss=0.008345, over 14960.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.0904, pruned_loss=0.01308, audio_tagging_loss=0.009212, over 3045144.69 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:52:13,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2763406.6666666665, ans=0.0
2023-11-24 08:52:15,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2763406.6666666665, ans=0.0
2023-11-24 08:52:52,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2763606.6666666665, ans=0.125
2023-11-24 08:52:58,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414550
2023-11-24 08:53:09,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2763673.3333333335, ans=0.09899494936611666
2023-11-24 08:53:10,336 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2763673.3333333335, ans=0.125
2023-11-24 08:53:15,303 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5750, loss[loss=0.06126, simple_loss=0.07949, pruned_loss=0.01105, audio_tagging_loss=0.01047, over 13298.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09029, pruned_loss=0.01304, audio_tagging_loss=0.009083, over 3044869.31 frames. ], batch size: 52, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:53:24,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2763740.0, ans=0.1
2023-11-24 08:53:28,331 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.460e+01 8.483e+01 9.031e+01 9.613e+01 1.136e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 08:53:30,342 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.57 vs. limit=15.0
2023-11-24 08:53:55,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2763940.0, ans=0.2
2023-11-24 08:53:58,417 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.73 vs. limit=22.5
2023-11-24 08:54:01,771 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414600
2023-11-24 08:54:03,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2763940.0, ans=0.1
2023-11-24 08:54:16,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2764073.3333333335, ans=0.0
2023-11-24 08:54:17,426 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5800, loss[loss=0.08021, simple_loss=0.1138, pruned_loss=0.01552, audio_tagging_loss=0.007784, over 15079.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09072, pruned_loss=0.01315, audio_tagging_loss=0.008943, over 3041360.10 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:54:41,309 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.61 vs. limit=22.5
2023-11-24 08:55:04,012 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414650
2023-11-24 08:55:19,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2764406.6666666665, ans=0.1
2023-11-24 08:55:20,554 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5850, loss[loss=0.04994, simple_loss=0.0616, pruned_loss=0.009259, audio_tagging_loss=0.009883, over 14282.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.08998, pruned_loss=0.01328, audio_tagging_loss=0.008927, over 3040631.30 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:55:25,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2764406.6666666665, ans=0.1
2023-11-24 08:55:36,181 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.408e+01 8.653e+01 9.084e+01 9.959e+01 1.265e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 08:55:45,616 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:55:45,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2764540.0, ans=0.125
2023-11-24 08:55:49,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2764540.0, ans=0.2
2023-11-24 08:56:02,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2764606.6666666665, ans=0.125
2023-11-24 08:56:07,455 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414700
2023-11-24 08:56:08,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2764606.6666666665, ans=0.1
2023-11-24 08:56:10,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2764673.3333333335, ans=0.0
2023-11-24 08:56:11,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2764673.3333333335, ans=0.125
2023-11-24 08:56:21,266 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.12 vs. limit=10.0
2023-11-24 08:56:23,848 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5900, loss[loss=0.06226, simple_loss=0.08349, pruned_loss=0.01283, audio_tagging_loss=0.007689, over 16465.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09105, pruned_loss=0.01344, audio_tagging_loss=0.008876, over 3043658.71 frames. ], batch size: 62, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:56:26,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2764740.0, ans=0.125
2023-11-24 08:56:27,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2764740.0, ans=0.0
2023-11-24 08:56:28,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.86 vs. limit=15.0
2023-11-24 08:56:30,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2764740.0, ans=0.125
2023-11-24 08:57:07,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2764940.0, ans=0.2
2023-11-24 08:57:10,221 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414750
2023-11-24 08:57:11,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2764940.0, ans=0.0
2023-11-24 08:57:26,296 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 5950, loss[loss=0.07131, simple_loss=0.09246, pruned_loss=0.01384, audio_tagging_loss=0.01123, over 15394.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09078, pruned_loss=0.0133, audio_tagging_loss=0.008839, over 3047973.30 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:57:40,400 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.559e+01 8.356e+01 8.865e+01 9.769e+01 1.184e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-24 08:57:42,696 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.78 vs. limit=15.0
2023-11-24 08:57:52,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2765206.6666666665, ans=0.2
2023-11-24 08:57:55,334 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:57:59,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2765206.6666666665, ans=0.5
2023-11-24 08:58:07,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2765273.3333333335, ans=0.125
2023-11-24 08:58:12,284 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414800
2023-11-24 08:58:13,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2765273.3333333335, ans=0.125
2023-11-24 08:58:21,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2765340.0, ans=0.0
2023-11-24 08:58:21,210 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:58:25,892 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2765340.0, ans=0.0
2023-11-24 08:58:27,956 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6000, loss[loss=0.04674, simple_loss=0.04505, pruned_loss=0.00815, audio_tagging_loss=0.01606, over 14535.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09077, pruned_loss=0.01328, audio_tagging_loss=0.008883, over 3043692.46 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:58:27,956 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 08:59:09,849 INFO [train_asr.py:1253] (2/4) Epoch 35, validation: loss=0.05756, simple_loss=0.05084, pruned_loss=0.005093, audio_tagging_loss=0.02705, over 4681554.00 frames. 
2023-11-24 08:59:09,850 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 08:59:25,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2765473.3333333335, ans=0.0
2023-11-24 08:59:44,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2765540.0, ans=0.1
2023-11-24 08:59:50,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2765606.6666666665, ans=0.2
2023-11-24 08:59:54,887 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:59:56,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414850
2023-11-24 09:00:02,659 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.76 vs. limit=15.0
2023-11-24 09:00:03,967 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.70 vs. limit=22.5
2023-11-24 09:00:06,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2765673.3333333335, ans=0.125
2023-11-24 09:00:11,407 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.26 vs. limit=15.0
2023-11-24 09:00:11,867 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6050, loss[loss=0.0794, simple_loss=0.1054, pruned_loss=0.016, audio_tagging_loss=0.0107, over 15207.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09047, pruned_loss=0.01315, audio_tagging_loss=0.008924, over 3056469.24 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:00:22,599 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2765806.6666666665, ans=0.0
2023-11-24 09:00:24,620 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.33 vs. limit=15.0
2023-11-24 09:00:26,526 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.506e+01 8.401e+01 9.139e+01 9.786e+01 1.325e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 09:00:38,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2765873.3333333335, ans=0.1
2023-11-24 09:00:40,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2765873.3333333335, ans=0.125
2023-11-24 09:00:41,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2765873.3333333335, ans=0.025
2023-11-24 09:00:50,403 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2765940.0, ans=0.0
2023-11-24 09:00:58,020 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414900
2023-11-24 09:00:58,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2765940.0, ans=0.1
2023-11-24 09:01:05,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2766006.6666666665, ans=0.0
2023-11-24 09:01:06,576 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2766006.6666666665, ans=0.125
2023-11-24 09:01:13,959 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6100, loss[loss=0.04645, simple_loss=0.06404, pruned_loss=0.00693, audio_tagging_loss=0.007503, over 14989.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09085, pruned_loss=0.01315, audio_tagging_loss=0.00888, over 3056802.58 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:02:00,255 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 414950
2023-11-24 09:02:16,784 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6150, loss[loss=0.06053, simple_loss=0.08368, pruned_loss=0.01046, audio_tagging_loss=0.008227, over 14034.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09096, pruned_loss=0.01318, audio_tagging_loss=0.008894, over 3042695.58 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:02:19,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2766406.6666666665, ans=0.125
2023-11-24 09:02:21,008 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0
2023-11-24 09:02:31,220 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.378e+01 9.060e+01 9.653e+01 1.339e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 09:02:36,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2766473.3333333335, ans=0.125
2023-11-24 09:02:42,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2766540.0, ans=0.0
2023-11-24 09:02:47,494 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.36 vs. limit=12.0
2023-11-24 09:02:48,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2766540.0, ans=0.125
2023-11-24 09:02:59,972 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.69 vs. limit=12.0
2023-11-24 09:03:03,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415000
2023-11-24 09:03:06,556 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.27 vs. limit=6.0
2023-11-24 09:03:18,779 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6200, loss[loss=0.05956, simple_loss=0.08289, pruned_loss=0.007244, audio_tagging_loss=0.01087, over 15580.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09022, pruned_loss=0.01293, audio_tagging_loss=0.009009, over 3042015.96 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:03:22,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2766740.0, ans=0.125
2023-11-24 09:03:39,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2766806.6666666665, ans=15.0
2023-11-24 09:03:47,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2766873.3333333335, ans=0.0
2023-11-24 09:04:04,863 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415050
2023-11-24 09:04:08,983 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.46 vs. limit=15.0
2023-11-24 09:04:13,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2767006.6666666665, ans=0.125
2023-11-24 09:04:16,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.47 vs. limit=15.0
2023-11-24 09:04:21,390 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6250, loss[loss=0.08284, simple_loss=0.1067, pruned_loss=0.02018, audio_tagging_loss=0.009319, over 14517.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09005, pruned_loss=0.01287, audio_tagging_loss=0.009064, over 3039001.76 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:04:31,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.56 vs. limit=10.0
2023-11-24 09:04:32,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.70 vs. limit=15.0
2023-11-24 09:04:38,115 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.234e+01 8.847e+01 9.550e+01 1.216e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-24 09:04:57,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2767273.3333333335, ans=0.0
2023-11-24 09:05:05,465 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.45 vs. limit=15.0
2023-11-24 09:05:07,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415100
2023-11-24 09:05:07,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2767273.3333333335, ans=0.0
2023-11-24 09:05:07,936 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.10 vs. limit=15.0
2023-11-24 09:05:18,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2767340.0, ans=0.125
2023-11-24 09:05:24,273 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6300, loss[loss=0.06159, simple_loss=0.08717, pruned_loss=0.008922, audio_tagging_loss=0.009085, over 14725.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.0907, pruned_loss=0.01297, audio_tagging_loss=0.009123, over 3040276.56 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:05:36,359 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2767473.3333333335, ans=0.125
2023-11-24 09:05:56,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2767540.0, ans=0.125
2023-11-24 09:06:10,474 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415150
2023-11-24 09:06:20,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2767673.3333333335, ans=0.1
2023-11-24 09:06:26,028 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6350, loss[loss=0.07233, simple_loss=0.09028, pruned_loss=0.0158, audio_tagging_loss=0.01139, over 15221.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09032, pruned_loss=0.0129, audio_tagging_loss=0.009159, over 3045513.22 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:06:33,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2767740.0, ans=0.0
2023-11-24 09:06:41,161 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.597e+01 9.129e+01 9.711e+01 1.215e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 09:06:59,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2767873.3333333335, ans=0.07
2023-11-24 09:07:03,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2767940.0, ans=0.125
2023-11-24 09:07:03,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2767940.0, ans=0.125
2023-11-24 09:07:04,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2767940.0, ans=0.0
2023-11-24 09:07:10,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2767940.0, ans=0.125
2023-11-24 09:07:11,384 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415200
2023-11-24 09:07:17,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2768006.6666666665, ans=0.0
2023-11-24 09:07:17,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2768006.6666666665, ans=0.125
2023-11-24 09:07:27,184 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6400, loss[loss=0.06622, simple_loss=0.08437, pruned_loss=0.0133, audio_tagging_loss=0.01074, over 14221.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.0901, pruned_loss=0.01297, audio_tagging_loss=0.009207, over 3042891.11 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:08:12,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2768273.3333333335, ans=0.125
2023-11-24 09:08:12,973 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415250
2023-11-24 09:08:17,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2768340.0, ans=0.125
2023-11-24 09:08:30,060 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0
2023-11-24 09:08:30,522 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6450, loss[loss=0.06261, simple_loss=0.08641, pruned_loss=0.01226, audio_tagging_loss=0.007149, over 14568.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09004, pruned_loss=0.01294, audio_tagging_loss=0.009211, over 3041432.70 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:08:34,846 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.37 vs. limit=15.0
2023-11-24 09:08:46,597 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.512e+01 9.076e+01 9.568e+01 1.780e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 09:09:07,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2768606.6666666665, ans=0.0
2023-11-24 09:09:17,354 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415300
2023-11-24 09:09:26,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2768673.3333333335, ans=0.0
2023-11-24 09:09:32,750 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6500, loss[loss=0.06383, simple_loss=0.0886, pruned_loss=0.01166, audio_tagging_loss=0.007867, over 15203.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.08944, pruned_loss=0.01286, audio_tagging_loss=0.009294, over 3045275.41 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:09:57,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2768873.3333333335, ans=0.0
2023-11-24 09:10:06,895 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.52 vs. limit=22.5
2023-11-24 09:10:11,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2768940.0, ans=0.125
2023-11-24 09:10:12,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2768940.0, ans=0.0
2023-11-24 09:10:17,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415350
2023-11-24 09:10:26,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2769006.6666666665, ans=0.1
2023-11-24 09:10:33,288 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6550, loss[loss=0.07304, simple_loss=0.1089, pruned_loss=0.01151, audio_tagging_loss=0.007089, over 15734.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.08952, pruned_loss=0.01286, audio_tagging_loss=0.009205, over 3048110.85 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:10:48,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2769140.0, ans=0.0
2023-11-24 09:10:50,882 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.510e+01 8.986e+01 9.947e+01 1.296e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 09:11:02,409 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.09 vs. limit=10.0
2023-11-24 09:11:18,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2769273.3333333335, ans=0.125
2023-11-24 09:11:19,775 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415400
2023-11-24 09:11:37,602 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6600, loss[loss=0.06855, simple_loss=0.08853, pruned_loss=0.01452, audio_tagging_loss=0.009759, over 14834.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.08986, pruned_loss=0.01302, audio_tagging_loss=0.009013, over 3041151.41 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:11:37,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2769406.6666666665, ans=0.05
2023-11-24 09:11:47,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2769406.6666666665, ans=0.0
2023-11-24 09:12:05,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2769540.0, ans=0.125
2023-11-24 09:12:05,784 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-24 09:12:06,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2769540.0, ans=0.0
2023-11-24 09:12:14,997 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.16 vs. limit=22.5
2023-11-24 09:12:17,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2769606.6666666665, ans=0.125
2023-11-24 09:12:23,233 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415450
2023-11-24 09:12:30,009 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2769673.3333333335, ans=0.0
2023-11-24 09:12:38,687 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.49 vs. limit=15.0
2023-11-24 09:12:39,121 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6650, loss[loss=0.05782, simple_loss=0.06962, pruned_loss=0.008955, audio_tagging_loss=0.01405, over 15649.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.0911, pruned_loss=0.01319, audio_tagging_loss=0.008987, over 3037092.71 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:12:54,691 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.324e+01 8.429e+01 8.957e+01 9.773e+01 1.353e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 09:13:01,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.38 vs. limit=15.0
2023-11-24 09:13:09,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.25 vs. limit=12.0
2023-11-24 09:13:12,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2769873.3333333335, ans=0.05
2023-11-24 09:13:18,089 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.12 vs. limit=15.0
2023-11-24 09:13:25,859 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415500
2023-11-24 09:13:39,473 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.14 vs. limit=6.0
2023-11-24 09:13:41,132 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6700, loss[loss=0.04452, simple_loss=0.05825, pruned_loss=0.004918, audio_tagging_loss=0.01048, over 14765.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09123, pruned_loss=0.01329, audio_tagging_loss=0.008895, over 3035022.67 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:13:55,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2770140.0, ans=0.2
2023-11-24 09:14:01,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2770140.0, ans=0.125
2023-11-24 09:14:02,221 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.08 vs. limit=22.5
2023-11-24 09:14:16,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2770206.6666666665, ans=0.2
2023-11-24 09:14:27,380 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415550
2023-11-24 09:14:43,946 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6750, loss[loss=0.07033, simple_loss=0.0976, pruned_loss=0.01383, audio_tagging_loss=0.007705, over 14574.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09054, pruned_loss=0.01323, audio_tagging_loss=0.008857, over 3032027.10 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:14:57,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2770473.3333333335, ans=0.125
2023-11-24 09:15:00,429 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.302e+01 8.550e+01 9.289e+01 9.814e+01 1.264e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-24 09:15:25,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2770606.6666666665, ans=0.0
2023-11-24 09:15:30,362 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415600
2023-11-24 09:15:33,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2770673.3333333335, ans=0.95
2023-11-24 09:15:47,285 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6800, loss[loss=0.05534, simple_loss=0.0694, pruned_loss=0.01023, audio_tagging_loss=0.01041, over 14533.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09035, pruned_loss=0.01325, audio_tagging_loss=0.008937, over 3036232.20 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:15:48,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2770740.0, ans=0.125
2023-11-24 09:16:02,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2770806.6666666665, ans=0.125
2023-11-24 09:16:03,447 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:16:10,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2770873.3333333335, ans=0.0
2023-11-24 09:16:26,531 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.56 vs. limit=15.0
2023-11-24 09:16:33,500 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415650
2023-11-24 09:16:46,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2771006.6666666665, ans=0.035
2023-11-24 09:16:48,153 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.17 vs. limit=6.0
2023-11-24 09:16:48,697 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6850, loss[loss=0.04724, simple_loss=0.06798, pruned_loss=0.006221, audio_tagging_loss=0.007031, over 14582.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09099, pruned_loss=0.01339, audio_tagging_loss=0.008835, over 3041560.37 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:17:06,380 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.373e+01 8.924e+01 9.619e+01 1.192e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-24 09:17:18,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2771206.6666666665, ans=0.125
2023-11-24 09:17:19,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2771206.6666666665, ans=0.125
2023-11-24 09:17:22,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2771206.6666666665, ans=0.0
2023-11-24 09:17:32,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2771273.3333333335, ans=0.125
2023-11-24 09:17:34,744 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415700
2023-11-24 09:17:47,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.01 vs. limit=15.0
2023-11-24 09:17:50,597 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6900, loss[loss=0.0717, simple_loss=0.09877, pruned_loss=0.01347, audio_tagging_loss=0.008841, over 14213.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09018, pruned_loss=0.01309, audio_tagging_loss=0.008827, over 3040592.26 frames. ], batch size: 52, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:17:56,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2771406.6666666665, ans=0.125
2023-11-24 09:17:58,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2771406.6666666665, ans=0.125
2023-11-24 09:18:06,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.96 vs. limit=15.0
2023-11-24 09:18:36,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415750
2023-11-24 09:18:37,855 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:18:45,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2771673.3333333335, ans=0.125
2023-11-24 09:18:53,281 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 6950, loss[loss=0.0747, simple_loss=0.1007, pruned_loss=0.01675, audio_tagging_loss=0.007614, over 15117.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09099, pruned_loss=0.01324, audio_tagging_loss=0.008864, over 3043200.98 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:19:01,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2771740.0, ans=0.2
2023-11-24 09:19:10,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.481e+01 9.077e+01 9.564e+01 1.596e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 09:19:27,874 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.87 vs. limit=22.5
2023-11-24 09:19:39,847 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415800
2023-11-24 09:19:41,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2771940.0, ans=0.125
2023-11-24 09:19:48,388 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.27 vs. limit=22.5
2023-11-24 09:19:55,975 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7000, loss[loss=0.0929, simple_loss=0.1175, pruned_loss=0.02612, audio_tagging_loss=0.008014, over 15379.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09077, pruned_loss=0.01304, audio_tagging_loss=0.008929, over 3045499.97 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:19:57,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2772073.3333333335, ans=0.125
2023-11-24 09:20:04,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2772073.3333333335, ans=0.035
2023-11-24 09:20:14,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2772140.0, ans=0.1
2023-11-24 09:20:28,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2772206.6666666665, ans=0.0
2023-11-24 09:20:28,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2772206.6666666665, ans=0.125
2023-11-24 09:20:42,071 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415850
2023-11-24 09:20:53,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.93 vs. limit=22.5
2023-11-24 09:20:57,814 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7050, loss[loss=0.07132, simple_loss=0.1048, pruned_loss=0.01229, audio_tagging_loss=0.006637, over 15206.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09151, pruned_loss=0.01323, audio_tagging_loss=0.008853, over 3047363.51 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:21:00,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2772406.6666666665, ans=0.0
2023-11-24 09:21:15,711 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.245e+01 8.341e+01 9.106e+01 9.596e+01 1.490e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 09:21:25,262 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.66 vs. limit=22.5
2023-11-24 09:21:43,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2772606.6666666665, ans=0.025
2023-11-24 09:21:44,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415900
2023-11-24 09:21:45,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2772606.6666666665, ans=0.125
2023-11-24 09:21:48,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2772673.3333333335, ans=22.5
2023-11-24 09:22:00,637 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7100, loss[loss=0.07069, simple_loss=0.0891, pruned_loss=0.01539, audio_tagging_loss=0.01075, over 15140.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09181, pruned_loss=0.0133, audio_tagging_loss=0.008974, over 3050581.37 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:22:46,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 415950
2023-11-24 09:22:53,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2773006.6666666665, ans=0.2
2023-11-24 09:22:54,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2773006.6666666665, ans=0.125
2023-11-24 09:23:02,690 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7150, loss[loss=0.06574, simple_loss=0.09045, pruned_loss=0.01161, audio_tagging_loss=0.0089, over 15168.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09169, pruned_loss=0.01318, audio_tagging_loss=0.009065, over 3054481.69 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:23:18,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2773140.0, ans=0.5
2023-11-24 09:23:19,803 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.644e+01 9.348e+01 1.022e+02 1.334e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-24 09:23:20,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2773140.0, ans=0.125
2023-11-24 09:23:29,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2773206.6666666665, ans=0.125
2023-11-24 09:23:48,065 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416000
2023-11-24 09:23:48,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2773273.3333333335, ans=0.2
2023-11-24 09:23:48,689 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.18 vs. limit=15.0
2023-11-24 09:23:49,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2773273.3333333335, ans=0.125
2023-11-24 09:23:58,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2773340.0, ans=0.125
2023-11-24 09:24:01,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2773340.0, ans=0.0
2023-11-24 09:24:09,343 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7200, loss[loss=0.05057, simple_loss=0.07077, pruned_loss=0.006399, audio_tagging_loss=0.008788, over 15175.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09233, pruned_loss=0.01327, audio_tagging_loss=0.009132, over 3048464.63 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:24:38,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2773540.0, ans=0.0
2023-11-24 09:24:38,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.82 vs. limit=15.0
2023-11-24 09:24:49,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2773606.6666666665, ans=0.05
2023-11-24 09:24:55,818 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416050
2023-11-24 09:25:09,386 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.25 vs. limit=15.0
2023-11-24 09:25:12,604 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7250, loss[loss=0.06967, simple_loss=0.1011, pruned_loss=0.01331, audio_tagging_loss=0.005806, over 15199.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09109, pruned_loss=0.01305, audio_tagging_loss=0.009247, over 3045684.86 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:25:29,220 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.544e+01 8.813e+01 9.334e+01 1.034e+02 1.372e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-24 09:25:45,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2773873.3333333335, ans=0.0
2023-11-24 09:25:58,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416100
2023-11-24 09:26:00,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2773940.0, ans=0.125
2023-11-24 09:26:13,969 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7300, loss[loss=0.07981, simple_loss=0.1081, pruned_loss=0.01689, audio_tagging_loss=0.008863, over 13957.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09176, pruned_loss=0.01308, audio_tagging_loss=0.009149, over 3039333.44 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:26:24,432 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.74 vs. limit=15.0
2023-11-24 09:26:41,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2774206.6666666665, ans=0.125
2023-11-24 09:26:51,309 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.19 vs. limit=15.0
2023-11-24 09:26:53,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2774273.3333333335, ans=0.07
2023-11-24 09:27:00,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416150
2023-11-24 09:27:00,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2774273.3333333335, ans=0.2
2023-11-24 09:27:06,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2774340.0, ans=0.125
2023-11-24 09:27:06,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2774340.0, ans=0.125
2023-11-24 09:27:12,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2774340.0, ans=0.125
2023-11-24 09:27:16,243 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7350, loss[loss=0.07169, simple_loss=0.1023, pruned_loss=0.01463, audio_tagging_loss=0.005929, over 15157.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09105, pruned_loss=0.01313, audio_tagging_loss=0.008966, over 3032061.58 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:27:21,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2774406.6666666665, ans=0.125
2023-11-24 09:27:21,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2774406.6666666665, ans=0.125
2023-11-24 09:27:24,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2774406.6666666665, ans=0.1
2023-11-24 09:27:28,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2774473.3333333335, ans=0.0
2023-11-24 09:27:28,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2774473.3333333335, ans=0.125
2023-11-24 09:27:34,371 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.604e+01 9.108e+01 1.008e+02 1.406e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 09:27:41,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2774540.0, ans=0.1
2023-11-24 09:27:42,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2774540.0, ans=0.07
2023-11-24 09:27:45,194 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:27:51,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2774540.0, ans=0.125
2023-11-24 09:28:02,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416200
2023-11-24 09:28:11,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2774673.3333333335, ans=0.125
2023-11-24 09:28:15,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2774673.3333333335, ans=0.1
2023-11-24 09:28:19,625 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7400, loss[loss=0.07943, simple_loss=0.1111, pruned_loss=0.01758, audio_tagging_loss=0.006322, over 14754.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09127, pruned_loss=0.01316, audio_tagging_loss=0.008829, over 3029240.33 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:28:29,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2774740.0, ans=0.125
2023-11-24 09:28:42,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2774873.3333333335, ans=0.125
2023-11-24 09:28:49,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2774873.3333333335, ans=0.125
2023-11-24 09:29:03,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2774940.0, ans=0.125
2023-11-24 09:29:05,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416250
2023-11-24 09:29:10,995 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.07 vs. limit=15.0
2023-11-24 09:29:21,041 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7450, loss[loss=0.05118, simple_loss=0.06083, pruned_loss=0.00931, audio_tagging_loss=0.01146, over 14456.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09099, pruned_loss=0.01315, audio_tagging_loss=0.00887, over 3030058.96 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:29:32,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2775140.0, ans=0.125
2023-11-24 09:29:38,157 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 8.229e+01 9.115e+01 9.575e+01 1.366e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 09:29:45,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2775206.6666666665, ans=0.0
2023-11-24 09:29:48,566 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0
2023-11-24 09:30:05,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2775273.3333333335, ans=0.2
2023-11-24 09:30:07,431 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416300
2023-11-24 09:30:22,754 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7500, loss[loss=0.0693, simple_loss=0.09668, pruned_loss=0.01593, audio_tagging_loss=0.005033, over 14903.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09113, pruned_loss=0.01309, audio_tagging_loss=0.008814, over 3037286.87 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:31:01,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.58 vs. limit=15.0
2023-11-24 09:31:08,196 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416350
2023-11-24 09:31:13,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2775673.3333333335, ans=0.0
2023-11-24 09:31:17,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2775673.3333333335, ans=0.1
2023-11-24 09:31:25,166 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7550, loss[loss=0.06975, simple_loss=0.09346, pruned_loss=0.01479, audio_tagging_loss=0.008229, over 15361.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09173, pruned_loss=0.01327, audio_tagging_loss=0.008719, over 3043163.08 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:31:27,227 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.85 vs. limit=22.5
2023-11-24 09:31:34,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2775740.0, ans=0.0
2023-11-24 09:31:41,459 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.230e+01 8.548e+01 9.078e+01 9.572e+01 1.867e+02, threshold=1.816e+02, percent-clipped=1.0
2023-11-24 09:31:46,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.28 vs. limit=15.0
2023-11-24 09:31:54,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2775873.3333333335, ans=0.125
2023-11-24 09:32:11,179 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416400
2023-11-24 09:32:19,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2776006.6666666665, ans=0.0
2023-11-24 09:32:22,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2776006.6666666665, ans=0.2
2023-11-24 09:32:26,775 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7600, loss[loss=0.06826, simple_loss=0.09459, pruned_loss=0.01183, audio_tagging_loss=0.009134, over 16563.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09084, pruned_loss=0.01311, audio_tagging_loss=0.008825, over 3048037.52 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:32:30,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2776073.3333333335, ans=0.2
2023-11-24 09:32:37,850 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.26 vs. limit=22.5
2023-11-24 09:32:38,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2776140.0, ans=0.125
2023-11-24 09:33:07,197 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.63 vs. limit=15.0
2023-11-24 09:33:12,563 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416450
2023-11-24 09:33:17,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2776340.0, ans=0.125
2023-11-24 09:33:27,747 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7650, loss[loss=0.07607, simple_loss=0.1077, pruned_loss=0.01695, audio_tagging_loss=0.005251, over 14572.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09048, pruned_loss=0.01295, audio_tagging_loss=0.008729, over 3044127.57 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:33:44,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2776473.3333333335, ans=0.125
2023-11-24 09:33:48,915 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.062e+01 8.355e+01 9.042e+01 9.809e+01 1.325e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 09:33:50,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2776473.3333333335, ans=0.0
2023-11-24 09:33:51,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2776473.3333333335, ans=0.0
2023-11-24 09:34:13,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416500
2023-11-24 09:34:26,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2776673.3333333335, ans=0.1
2023-11-24 09:34:27,683 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.35 vs. limit=15.0
2023-11-24 09:34:30,123 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7700, loss[loss=0.08092, simple_loss=0.1161, pruned_loss=0.01501, audio_tagging_loss=0.007849, over 15494.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.09009, pruned_loss=0.01291, audio_tagging_loss=0.008682, over 3046232.00 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:34:31,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2776740.0, ans=0.125
2023-11-24 09:34:37,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2776740.0, ans=0.125
2023-11-24 09:34:45,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-24 09:34:55,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.58 vs. limit=15.0
2023-11-24 09:35:14,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416550
2023-11-24 09:35:14,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2776940.0, ans=0.2
2023-11-24 09:35:24,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2777006.6666666665, ans=0.125
2023-11-24 09:35:30,955 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.88 vs. limit=15.0
2023-11-24 09:35:31,200 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7750, loss[loss=0.05599, simple_loss=0.07366, pruned_loss=0.01067, audio_tagging_loss=0.008485, over 15329.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09068, pruned_loss=0.01309, audio_tagging_loss=0.008788, over 3042322.51 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:35:37,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2777073.3333333335, ans=0.0
2023-11-24 09:35:39,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2777073.3333333335, ans=0.125
2023-11-24 09:35:43,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2777140.0, ans=0.125
2023-11-24 09:35:50,350 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.304e+01 9.008e+01 9.844e+01 1.304e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 09:35:55,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2777206.6666666665, ans=0.2
2023-11-24 09:36:15,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2777273.3333333335, ans=0.125
2023-11-24 09:36:17,185 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416600
2023-11-24 09:36:31,430 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0
2023-11-24 09:36:33,298 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7800, loss[loss=0.05808, simple_loss=0.08316, pruned_loss=0.007901, audio_tagging_loss=0.008597, over 14438.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09155, pruned_loss=0.01318, audio_tagging_loss=0.008871, over 3042785.98 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:36:37,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.75 vs. limit=6.0
2023-11-24 09:36:39,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2777406.6666666665, ans=0.1
2023-11-24 09:36:44,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2777473.3333333335, ans=10.0
2023-11-24 09:36:49,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2777473.3333333335, ans=0.07
2023-11-24 09:36:54,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2777473.3333333335, ans=0.5
2023-11-24 09:37:19,359 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416650
2023-11-24 09:37:35,057 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7850, loss[loss=0.07289, simple_loss=0.0949, pruned_loss=0.01336, audio_tagging_loss=0.01207, over 15644.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.0915, pruned_loss=0.01317, audio_tagging_loss=0.008939, over 3045154.71 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:37:37,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.51 vs. limit=15.0
2023-11-24 09:37:49,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2777806.6666666665, ans=0.0
2023-11-24 09:37:53,641 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0
2023-11-24 09:37:55,546 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.418e+01 9.005e+01 9.795e+01 1.227e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 09:38:05,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2777873.3333333335, ans=0.125
2023-11-24 09:38:06,635 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:38:15,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2777940.0, ans=15.0
2023-11-24 09:38:15,049 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0
2023-11-24 09:38:20,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2777940.0, ans=0.025
2023-11-24 09:38:21,176 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416700
2023-11-24 09:38:26,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2778006.6666666665, ans=0.125
2023-11-24 09:38:37,570 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7900, loss[loss=0.0545, simple_loss=0.07597, pruned_loss=0.008357, audio_tagging_loss=0.008155, over 15199.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09091, pruned_loss=0.013, audio_tagging_loss=0.009083, over 3041169.62 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:38:46,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2778073.3333333335, ans=0.1
2023-11-24 09:38:50,705 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2778140.0, ans=0.1
2023-11-24 09:38:57,128 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0
2023-11-24 09:38:59,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2778140.0, ans=0.09899494936611666
2023-11-24 09:39:02,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2778206.6666666665, ans=0.1
2023-11-24 09:39:04,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2778206.6666666665, ans=0.125
2023-11-24 09:39:22,829 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.60 vs. limit=15.0
2023-11-24 09:39:23,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416750
2023-11-24 09:39:26,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2778340.0, ans=0.1
2023-11-24 09:39:33,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2778340.0, ans=0.0
2023-11-24 09:39:34,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2778340.0, ans=0.125
2023-11-24 09:39:37,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2778406.6666666665, ans=0.025
2023-11-24 09:39:38,825 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 7950, loss[loss=0.08632, simple_loss=0.1286, pruned_loss=0.01524, audio_tagging_loss=0.006782, over 15995.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09093, pruned_loss=0.01305, audio_tagging_loss=0.009061, over 3041581.29 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:39:42,669 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:39:53,587 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:39:58,841 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 8.587e+01 9.115e+01 9.680e+01 1.233e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 09:39:59,486 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.78 vs. limit=15.0
2023-11-24 09:40:00,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2778473.3333333335, ans=0.015
2023-11-24 09:40:11,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2778540.0, ans=0.0
2023-11-24 09:40:24,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2778606.6666666665, ans=0.125
2023-11-24 09:40:25,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416800
2023-11-24 09:40:26,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2778606.6666666665, ans=0.125
2023-11-24 09:40:35,528 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.63 vs. limit=15.0
2023-11-24 09:40:40,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2778740.0, ans=0.125
2023-11-24 09:40:41,545 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8000, loss[loss=0.06726, simple_loss=0.08752, pruned_loss=0.01155, audio_tagging_loss=0.01195, over 13681.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.0908, pruned_loss=0.01306, audio_tagging_loss=0.00917, over 3033558.45 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:41:27,976 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416850
2023-11-24 09:41:34,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2779006.6666666665, ans=0.0
2023-11-24 09:41:44,420 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8050, loss[loss=0.06363, simple_loss=0.08462, pruned_loss=0.009722, audio_tagging_loss=0.0116, over 14955.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09021, pruned_loss=0.01298, audio_tagging_loss=0.009269, over 3037048.16 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:42:00,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2779140.0, ans=0.125
2023-11-24 09:42:03,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2779140.0, ans=0.0
2023-11-24 09:42:05,149 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.535e+01 9.115e+01 9.754e+01 2.166e+02, threshold=1.823e+02, percent-clipped=1.0
2023-11-24 09:42:30,561 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416900
2023-11-24 09:42:42,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.14 vs. limit=15.0
2023-11-24 09:42:46,443 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8100, loss[loss=0.08354, simple_loss=0.1096, pruned_loss=0.01767, audio_tagging_loss=0.01107, over 16143.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09148, pruned_loss=0.01314, audio_tagging_loss=0.009096, over 3037614.16 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:42:51,496 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2779406.6666666665, ans=0.125
2023-11-24 09:42:53,882 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2779406.6666666665, ans=0.0
2023-11-24 09:42:56,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2779406.6666666665, ans=0.1
2023-11-24 09:43:03,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2779473.3333333335, ans=0.1
2023-11-24 09:43:12,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2779540.0, ans=0.1
2023-11-24 09:43:21,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2779540.0, ans=0.125
2023-11-24 09:43:30,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2779606.6666666665, ans=0.125
2023-11-24 09:43:32,669 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 416950
2023-11-24 09:43:42,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2779673.3333333335, ans=0.125
2023-11-24 09:43:47,835 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8150, loss[loss=0.09072, simple_loss=0.1278, pruned_loss=0.02011, audio_tagging_loss=0.006727, over 14946.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09107, pruned_loss=0.01318, audio_tagging_loss=0.008934, over 3038705.33 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:43:53,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2779740.0, ans=0.125
2023-11-24 09:44:09,534 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.453e+01 8.620e+01 9.310e+01 1.002e+02 1.330e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 09:44:18,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2779873.3333333335, ans=0.125
2023-11-24 09:44:20,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0
2023-11-24 09:44:34,048 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417000
2023-11-24 09:44:44,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2780006.6666666665, ans=0.1
2023-11-24 09:44:50,741 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8200, loss[loss=0.05214, simple_loss=0.06919, pruned_loss=0.008187, audio_tagging_loss=0.009358, over 14763.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09138, pruned_loss=0.01319, audio_tagging_loss=0.008783, over 3037170.09 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:44:51,957 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:45:37,029 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417050
2023-11-24 09:45:44,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2780340.0, ans=0.2
2023-11-24 09:45:46,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.28 vs. limit=12.0
2023-11-24 09:45:52,416 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8250, loss[loss=0.08814, simple_loss=0.1186, pruned_loss=0.02266, audio_tagging_loss=0.006175, over 15543.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09178, pruned_loss=0.01342, audio_tagging_loss=0.008703, over 3033210.08 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:46:13,778 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.658e+01 9.148e+01 9.760e+01 1.278e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 09:46:26,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.83 vs. limit=15.0
2023-11-24 09:46:31,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2780606.6666666665, ans=0.125
2023-11-24 09:46:38,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417100
2023-11-24 09:46:48,222 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-24 09:46:54,582 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8300, loss[loss=0.04959, simple_loss=0.06108, pruned_loss=0.01013, audio_tagging_loss=0.008913, over 15009.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09239, pruned_loss=0.0134, audio_tagging_loss=0.008676, over 3035606.01 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:46:56,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2780740.0, ans=0.0
2023-11-24 09:47:08,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2780806.6666666665, ans=0.04949747468305833
2023-11-24 09:47:08,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2780806.6666666665, ans=0.2
2023-11-24 09:47:11,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=2780806.6666666665, ans=0.02
2023-11-24 09:47:19,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2780873.3333333335, ans=0.0
2023-11-24 09:47:41,306 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417150
2023-11-24 09:47:42,788 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2780940.0, ans=0.125
2023-11-24 09:47:58,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-24 09:47:58,465 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8350, loss[loss=0.07096, simple_loss=0.09028, pruned_loss=0.01581, audio_tagging_loss=0.01001, over 14904.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09215, pruned_loss=0.01328, audio_tagging_loss=0.008687, over 3031570.12 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:48:01,125 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2781073.3333333335, ans=0.2
2023-11-24 09:48:14,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2781140.0, ans=0.1
2023-11-24 09:48:16,558 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.43 vs. limit=12.0
2023-11-24 09:48:18,480 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.379e+01 9.010e+01 9.696e+01 1.228e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 09:48:21,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2781206.6666666665, ans=0.125
2023-11-24 09:48:28,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2781206.6666666665, ans=0.09899494936611666
2023-11-24 09:48:28,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2781206.6666666665, ans=0.1
2023-11-24 09:48:43,748 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.12 vs. limit=12.0
2023-11-24 09:48:44,398 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417200
2023-11-24 09:48:57,172 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.83 vs. limit=10.0
2023-11-24 09:49:00,073 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8400, loss[loss=0.07887, simple_loss=0.09456, pruned_loss=0.02008, audio_tagging_loss=0.01151, over 15392.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09126, pruned_loss=0.01315, audio_tagging_loss=0.008717, over 3037946.72 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:49:07,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2781406.6666666665, ans=0.125
2023-11-24 09:49:07,657 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.16 vs. limit=22.5
2023-11-24 09:49:31,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2781540.0, ans=0.2
2023-11-24 09:49:35,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2781540.0, ans=0.2
2023-11-24 09:49:40,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2781606.6666666665, ans=0.125
2023-11-24 09:49:46,343 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417250
2023-11-24 09:49:47,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2781606.6666666665, ans=0.1
2023-11-24 09:50:02,428 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8450, loss[loss=0.08391, simple_loss=0.1073, pruned_loss=0.01682, audio_tagging_loss=0.01344, over 13697.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09112, pruned_loss=0.0131, audio_tagging_loss=0.008735, over 3034093.12 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:50:03,171 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.38 vs. limit=10.0
2023-11-24 09:50:07,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2781740.0, ans=0.0
2023-11-24 09:50:10,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2781740.0, ans=0.0
2023-11-24 09:50:11,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2781740.0, ans=0.125
2023-11-24 09:50:21,234 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2781806.6666666665, ans=0.125
2023-11-24 09:50:24,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.178e+01 8.593e+01 9.155e+01 9.677e+01 1.308e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 09:50:28,755 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.17 vs. limit=15.0
2023-11-24 09:50:35,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2781873.3333333335, ans=0.2
2023-11-24 09:50:49,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417300
2023-11-24 09:51:06,341 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8500, loss[loss=0.07734, simple_loss=0.1115, pruned_loss=0.01393, audio_tagging_loss=0.007681, over 15755.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09223, pruned_loss=0.01332, audio_tagging_loss=0.008687, over 3047057.87 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:51:06,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2782073.3333333335, ans=0.125
2023-11-24 09:51:13,142 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.23 vs. limit=15.0
2023-11-24 09:51:18,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2782140.0, ans=0.125
2023-11-24 09:51:21,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2782140.0, ans=0.5
2023-11-24 09:51:25,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2782140.0, ans=0.125
2023-11-24 09:51:31,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2782206.6666666665, ans=0.1
2023-11-24 09:51:32,139 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.72 vs. limit=15.0
2023-11-24 09:51:40,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2782206.6666666665, ans=0.125
2023-11-24 09:51:51,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2782273.3333333335, ans=0.2
2023-11-24 09:51:52,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417350
2023-11-24 09:52:07,812 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8550, loss[loss=0.05908, simple_loss=0.07472, pruned_loss=0.008932, audio_tagging_loss=0.01279, over 14513.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09223, pruned_loss=0.01327, audio_tagging_loss=0.008728, over 3049209.31 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:52:12,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2782406.6666666665, ans=0.125
2023-11-24 09:52:20,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2782473.3333333335, ans=0.0
2023-11-24 09:52:28,051 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.57 vs. limit=22.5
2023-11-24 09:52:29,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.685e+01 9.326e+01 1.025e+02 1.164e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 09:52:36,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2782540.0, ans=0.125
2023-11-24 09:52:45,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.15 vs. limit=15.0
2023-11-24 09:52:52,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2782606.6666666665, ans=0.0
2023-11-24 09:52:54,228 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417400
2023-11-24 09:52:57,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2782673.3333333335, ans=0.1
2023-11-24 09:52:59,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2782673.3333333335, ans=0.125
2023-11-24 09:53:02,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.00 vs. limit=15.0
2023-11-24 09:53:10,015 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8600, loss[loss=0.06995, simple_loss=0.09155, pruned_loss=0.01558, audio_tagging_loss=0.008605, over 15677.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09167, pruned_loss=0.0133, audio_tagging_loss=0.008841, over 3051179.95 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:53:10,835 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.54 vs. limit=22.5
2023-11-24 09:53:56,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417450
2023-11-24 09:54:03,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2783006.6666666665, ans=0.0
2023-11-24 09:54:03,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2783006.6666666665, ans=0.125
2023-11-24 09:54:12,819 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8650, loss[loss=0.08738, simple_loss=0.1173, pruned_loss=0.02104, audio_tagging_loss=0.007698, over 15161.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09196, pruned_loss=0.01328, audio_tagging_loss=0.00889, over 3056524.06 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:54:12,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2783073.3333333335, ans=0.125
2023-11-24 09:54:35,509 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.880e+01 8.560e+01 9.076e+01 1.003e+02 1.218e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 09:54:43,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2783206.6666666665, ans=0.125
2023-11-24 09:54:49,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2783273.3333333335, ans=0.2
2023-11-24 09:54:49,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2783273.3333333335, ans=0.125
2023-11-24 09:54:53,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2783273.3333333335, ans=0.1
2023-11-24 09:54:58,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2783273.3333333335, ans=0.125
2023-11-24 09:54:59,534 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417500
2023-11-24 09:55:16,172 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8700, loss[loss=0.07999, simple_loss=0.1142, pruned_loss=0.01488, audio_tagging_loss=0.007986, over 15629.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09063, pruned_loss=0.01306, audio_tagging_loss=0.00895, over 3058608.00 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:55:23,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2783406.6666666665, ans=0.125
2023-11-24 09:55:24,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2783406.6666666665, ans=0.0
2023-11-24 09:55:27,453 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.48 vs. limit=15.0
2023-11-24 09:55:46,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2783540.0, ans=0.0
2023-11-24 09:55:54,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2783606.6666666665, ans=0.125
2023-11-24 09:55:57,027 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.98 vs. limit=15.0
2023-11-24 09:55:59,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2783606.6666666665, ans=0.125
2023-11-24 09:55:59,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2783606.6666666665, ans=0.0
2023-11-24 09:56:02,413 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417550
2023-11-24 09:56:13,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.42 vs. limit=15.0
2023-11-24 09:56:17,913 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8750, loss[loss=0.06217, simple_loss=0.08742, pruned_loss=0.007645, audio_tagging_loss=0.01081, over 15331.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09186, pruned_loss=0.01334, audio_tagging_loss=0.008945, over 3055360.94 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:56:24,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2783740.0, ans=0.09899494936611666
2023-11-24 09:56:37,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2783806.6666666665, ans=0.0
2023-11-24 09:56:40,384 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.442e+01 8.767e+01 9.281e+01 1.037e+02 1.287e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 09:57:03,163 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417600
2023-11-24 09:57:20,250 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8800, loss[loss=0.07312, simple_loss=0.0971, pruned_loss=0.01547, audio_tagging_loss=0.009101, over 14365.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09314, pruned_loss=0.01354, audio_tagging_loss=0.008939, over 3058441.87 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:57:33,328 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:57:35,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2784140.0, ans=0.125
2023-11-24 09:57:46,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2784206.6666666665, ans=0.125
2023-11-24 09:57:55,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2784273.3333333335, ans=0.125
2023-11-24 09:57:57,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2784273.3333333335, ans=0.125
2023-11-24 09:57:58,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten.whitening_limit, batch_count=2784273.3333333335, ans=22.5
2023-11-24 09:57:59,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.31 vs. limit=22.5
2023-11-24 09:58:01,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2784273.3333333335, ans=0.125
2023-11-24 09:58:03,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.90 vs. limit=6.0
2023-11-24 09:58:05,337 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417650
2023-11-24 09:58:14,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2784340.0, ans=0.125
2023-11-24 09:58:21,981 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8850, loss[loss=0.07723, simple_loss=0.1101, pruned_loss=0.01357, audio_tagging_loss=0.008587, over 16312.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09303, pruned_loss=0.01351, audio_tagging_loss=0.008999, over 3062780.90 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:58:22,745 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0
2023-11-24 09:58:23,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2784406.6666666665, ans=0.0
2023-11-24 09:58:32,825 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:58:41,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2784473.3333333335, ans=0.09899494936611666
2023-11-24 09:58:41,415 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2784473.3333333335, ans=0.025
2023-11-24 09:58:43,363 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.550e+01 9.128e+01 9.722e+01 1.464e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 09:58:59,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff3.min_abs, batch_count=2784606.6666666665, ans=0.2
2023-11-24 09:59:07,661 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417700
2023-11-24 09:59:18,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2784673.3333333335, ans=0.125
2023-11-24 09:59:22,986 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8900, loss[loss=0.06322, simple_loss=0.07913, pruned_loss=0.009363, audio_tagging_loss=0.01429, over 15620.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09281, pruned_loss=0.01353, audio_tagging_loss=0.008953, over 3061960.63 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:59:29,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2784740.0, ans=0.0
2023-11-24 09:59:31,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2784740.0, ans=0.125
2023-11-24 09:59:59,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2784940.0, ans=0.125
2023-11-24 10:00:08,501 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417750
2023-11-24 10:00:09,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2784940.0, ans=0.125
2023-11-24 10:00:17,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2785006.6666666665, ans=0.2
2023-11-24 10:00:24,333 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 8950, loss[loss=0.05102, simple_loss=0.06843, pruned_loss=0.009465, audio_tagging_loss=0.007343, over 15778.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09282, pruned_loss=0.01365, audio_tagging_loss=0.008835, over 3061240.84 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:00:47,815 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.675e+01 8.510e+01 9.195e+01 1.012e+02 1.504e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 10:01:09,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417800
2023-11-24 10:01:10,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2785273.3333333335, ans=0.125
2023-11-24 10:01:15,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2785340.0, ans=0.125
2023-11-24 10:01:26,223 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9000, loss[loss=0.04517, simple_loss=0.05374, pruned_loss=0.005462, audio_tagging_loss=0.01284, over 14042.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09259, pruned_loss=0.01355, audio_tagging_loss=0.008809, over 3060528.64 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:01:26,224 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 10:01:57,943 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.3382, 2.9838, 3.3138, 2.9382, 3.6989, 3.7388, 3.2641, 3.2156],
       device='cuda:2')
2023-11-24 10:02:10,237 INFO [train_asr.py:1253] (2/4) Epoch 35, validation: loss=0.05875, simple_loss=0.05079, pruned_loss=0.005122, audio_tagging_loss=0.02823, over 4681554.00 frames. 
2023-11-24 10:02:10,237 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 10:02:12,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2785406.6666666665, ans=0.125
2023-11-24 10:02:23,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.01 vs. limit=22.5
2023-11-24 10:02:24,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2785473.3333333335, ans=0.125
2023-11-24 10:02:49,632 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.34 vs. limit=15.0
2023-11-24 10:02:50,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2785606.6666666665, ans=0.125
2023-11-24 10:02:56,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417850
2023-11-24 10:02:58,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2785673.3333333335, ans=0.2
2023-11-24 10:03:02,926 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.11 vs. limit=15.0
2023-11-24 10:03:12,443 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9050, loss[loss=0.06858, simple_loss=0.1043, pruned_loss=0.00979, audio_tagging_loss=0.006656, over 15126.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09252, pruned_loss=0.0134, audio_tagging_loss=0.00875, over 3058368.67 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:03:23,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2785740.0, ans=0.1
2023-11-24 10:03:36,292 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.575e+01 9.161e+01 9.743e+01 1.218e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 10:03:58,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417900
2023-11-24 10:04:06,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2786006.6666666665, ans=0.125
2023-11-24 10:04:14,921 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9100, loss[loss=0.06223, simple_loss=0.08386, pruned_loss=0.01133, audio_tagging_loss=0.008973, over 15779.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09184, pruned_loss=0.01324, audio_tagging_loss=0.008747, over 3051070.30 frames. ], batch size: 63, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:04:34,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2786140.0, ans=0.125
2023-11-24 10:04:48,499 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=17.83 vs. limit=22.5
2023-11-24 10:04:55,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2786273.3333333335, ans=0.0
2023-11-24 10:05:00,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 417950
2023-11-24 10:05:15,920 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9150, loss[loss=0.08106, simple_loss=0.1102, pruned_loss=0.0166, audio_tagging_loss=0.009362, over 15027.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09209, pruned_loss=0.0133, audio_tagging_loss=0.008669, over 3047679.58 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:05:24,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2786406.6666666665, ans=0.0
2023-11-24 10:05:39,645 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.404e+01 9.029e+01 9.911e+01 1.244e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 10:05:46,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2786540.0, ans=0.125
2023-11-24 10:05:59,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2786606.6666666665, ans=0.125
2023-11-24 10:06:01,938 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418000
2023-11-24 10:06:07,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2786673.3333333335, ans=0.125
2023-11-24 10:06:10,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2786673.3333333335, ans=0.125
2023-11-24 10:06:18,304 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9200, loss[loss=0.06686, simple_loss=0.08651, pruned_loss=0.01156, audio_tagging_loss=0.01204, over 14764.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09284, pruned_loss=0.01346, audio_tagging_loss=0.008549, over 3050251.97 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:06:40,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2786806.6666666665, ans=0.125
2023-11-24 10:06:59,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2786940.0, ans=0.1
2023-11-24 10:07:04,179 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418050
2023-11-24 10:07:20,416 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9250, loss[loss=0.06073, simple_loss=0.07175, pruned_loss=0.01076, audio_tagging_loss=0.01409, over 16743.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09204, pruned_loss=0.01334, audio_tagging_loss=0.00864, over 3053683.40 frames. ], batch size: 64, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:07:40,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2787140.0, ans=0.1
2023-11-24 10:07:43,436 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.443e+01 8.965e+01 9.843e+01 1.266e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 10:07:47,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2787206.6666666665, ans=0.0
2023-11-24 10:08:06,257 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418100
2023-11-24 10:08:15,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2787340.0, ans=0.0
2023-11-24 10:08:16,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.05 vs. limit=15.0
2023-11-24 10:08:22,358 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9300, loss[loss=0.06077, simple_loss=0.07831, pruned_loss=0.0113, audio_tagging_loss=0.01032, over 14600.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09184, pruned_loss=0.01333, audio_tagging_loss=0.00861, over 3054581.09 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:08:46,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2787540.0, ans=0.2
2023-11-24 10:08:48,135 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.11 vs. limit=12.0
2023-11-24 10:09:00,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2787606.6666666665, ans=0.1
2023-11-24 10:09:08,336 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418150
2023-11-24 10:09:13,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2787673.3333333335, ans=0.2
2023-11-24 10:09:15,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2787673.3333333335, ans=0.0
2023-11-24 10:09:23,905 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9350, loss[loss=0.07472, simple_loss=0.1078, pruned_loss=0.01348, audio_tagging_loss=0.007366, over 15733.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09202, pruned_loss=0.01328, audio_tagging_loss=0.008742, over 3059886.94 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:09:25,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2787740.0, ans=0.025
2023-11-24 10:09:42,923 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.47 vs. limit=22.5
2023-11-24 10:09:48,170 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.653e+01 8.452e+01 8.975e+01 9.526e+01 2.573e+02, threshold=1.795e+02, percent-clipped=1.0
2023-11-24 10:09:59,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2787940.0, ans=0.0
2023-11-24 10:09:59,951 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.10 vs. limit=15.0
2023-11-24 10:10:03,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.12 vs. limit=22.5
2023-11-24 10:10:09,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418200
2023-11-24 10:10:15,974 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:10:26,212 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9400, loss[loss=0.07272, simple_loss=0.09838, pruned_loss=0.01467, audio_tagging_loss=0.008864, over 16305.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09172, pruned_loss=0.01325, audio_tagging_loss=0.008959, over 3056074.24 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:10:31,690 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.84 vs. limit=10.0
2023-11-24 10:10:38,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2788140.0, ans=0.04949747468305833
2023-11-24 10:10:41,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2788140.0, ans=0.1
2023-11-24 10:11:12,710 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418250
2023-11-24 10:11:15,921 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.12 vs. limit=15.0
2023-11-24 10:11:20,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2788340.0, ans=0.125
2023-11-24 10:11:25,991 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:11:28,916 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9450, loss[loss=0.06317, simple_loss=0.08096, pruned_loss=0.01574, audio_tagging_loss=0.006945, over 15462.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09185, pruned_loss=0.01337, audio_tagging_loss=0.008995, over 3056817.41 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:11:51,347 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:11:54,121 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.573e+01 9.142e+01 1.013e+02 1.307e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 10:12:01,276 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.63 vs. limit=10.0
2023-11-24 10:12:14,915 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418300
2023-11-24 10:12:20,703 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.68 vs. limit=15.0
2023-11-24 10:12:28,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=2788673.3333333335, ans=0.2
2023-11-24 10:12:30,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2788740.0, ans=15.0
2023-11-24 10:12:30,838 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9500, loss[loss=0.04719, simple_loss=0.05188, pruned_loss=0.009665, audio_tagging_loss=0.01158, over 14412.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09154, pruned_loss=0.01332, audio_tagging_loss=0.009143, over 3058503.60 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:12:40,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2788740.0, ans=0.1
2023-11-24 10:12:43,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2788806.6666666665, ans=0.1
2023-11-24 10:13:04,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2788873.3333333335, ans=0.125
2023-11-24 10:13:06,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2788873.3333333335, ans=10.0
2023-11-24 10:13:17,141 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418350
2023-11-24 10:13:30,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2789006.6666666665, ans=0.125
2023-11-24 10:13:32,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2789006.6666666665, ans=0.125
2023-11-24 10:13:34,256 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9550, loss[loss=0.0622, simple_loss=0.08964, pruned_loss=0.009593, audio_tagging_loss=0.007783, over 15105.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09178, pruned_loss=0.01335, audio_tagging_loss=0.009086, over 3048070.29 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:13:37,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2789073.3333333335, ans=0.125
2023-11-24 10:13:44,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2789073.3333333335, ans=0.125
2023-11-24 10:13:51,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2789140.0, ans=0.025
2023-11-24 10:13:58,337 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.515e+01 8.574e+01 9.293e+01 1.001e+02 1.205e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 10:14:19,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2789273.3333333335, ans=0.0
2023-11-24 10:14:20,446 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418400
2023-11-24 10:14:36,090 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9600, loss[loss=0.05176, simple_loss=0.07108, pruned_loss=0.005523, audio_tagging_loss=0.0107, over 14667.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09206, pruned_loss=0.01336, audio_tagging_loss=0.009224, over 3050944.69 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:14:38,181 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.51 vs. limit=15.0
2023-11-24 10:14:38,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2789406.6666666665, ans=0.1
2023-11-24 10:14:50,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2789473.3333333335, ans=0.0
2023-11-24 10:14:52,282 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2789473.3333333335, ans=0.125
2023-11-24 10:15:22,482 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418450
2023-11-24 10:15:25,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2789673.3333333335, ans=0.2
2023-11-24 10:15:29,662 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:15:38,365 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9650, loss[loss=0.07403, simple_loss=0.0941, pruned_loss=0.0178, audio_tagging_loss=0.009185, over 15050.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09095, pruned_loss=0.01318, audio_tagging_loss=0.009268, over 3051436.94 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:16:04,132 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.403e+01 8.395e+01 8.995e+01 9.967e+01 1.274e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 10:16:25,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418500
2023-11-24 10:16:27,909 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2790006.6666666665, ans=0.2
2023-11-24 10:16:33,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2790006.6666666665, ans=0.2
2023-11-24 10:16:34,990 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.67 vs. limit=15.0
2023-11-24 10:16:42,581 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9700, loss[loss=0.05874, simple_loss=0.07829, pruned_loss=0.008259, audio_tagging_loss=0.01134, over 15816.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09111, pruned_loss=0.01315, audio_tagging_loss=0.009055, over 3050948.38 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:16:48,696 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2790073.3333333335, ans=0.0
2023-11-24 10:16:49,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2790073.3333333335, ans=0.1
2023-11-24 10:17:05,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2790206.6666666665, ans=0.0
2023-11-24 10:17:22,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2790273.3333333335, ans=0.07
2023-11-24 10:17:25,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.53 vs. limit=15.0
2023-11-24 10:17:26,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2790273.3333333335, ans=0.125
2023-11-24 10:17:27,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418550
2023-11-24 10:17:27,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2790273.3333333335, ans=0.2
2023-11-24 10:17:30,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2790340.0, ans=0.125
2023-11-24 10:17:36,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2790340.0, ans=0.1
2023-11-24 10:17:43,642 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9750, loss[loss=0.09353, simple_loss=0.1265, pruned_loss=0.02295, audio_tagging_loss=0.007311, over 16215.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09105, pruned_loss=0.01305, audio_tagging_loss=0.008982, over 3053864.36 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:17:47,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2790406.6666666665, ans=0.0
2023-11-24 10:17:52,123 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:17:54,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2790473.3333333335, ans=0.0
2023-11-24 10:18:08,322 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.283e+01 8.345e+01 8.930e+01 9.592e+01 1.228e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 10:18:17,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2790540.0, ans=0.2
2023-11-24 10:18:24,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2790606.6666666665, ans=0.1
2023-11-24 10:18:29,683 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418600
2023-11-24 10:18:40,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2790673.3333333335, ans=0.125
2023-11-24 10:18:45,251 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9800, loss[loss=0.04801, simple_loss=0.06334, pruned_loss=0.005925, audio_tagging_loss=0.01042, over 15487.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09077, pruned_loss=0.01315, audio_tagging_loss=0.008865, over 3052404.92 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:18:51,348 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.44 vs. limit=22.5
2023-11-24 10:18:55,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2790740.0, ans=0.1
2023-11-24 10:19:04,619 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.97 vs. limit=15.0
2023-11-24 10:19:10,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2790873.3333333335, ans=0.1
2023-11-24 10:19:12,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2790873.3333333335, ans=0.125
2023-11-24 10:19:16,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2790873.3333333335, ans=0.0
2023-11-24 10:19:31,355 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418650
2023-11-24 10:19:36,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2791006.6666666665, ans=0.0
2023-11-24 10:19:39,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2791006.6666666665, ans=0.125
2023-11-24 10:19:40,631 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:19:48,362 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9850, loss[loss=0.05971, simple_loss=0.08772, pruned_loss=0.008772, audio_tagging_loss=0.007081, over 14969.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09124, pruned_loss=0.01302, audio_tagging_loss=0.008819, over 3048513.87 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:19:51,948 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=12.0
2023-11-24 10:19:55,583 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.41 vs. limit=12.0
2023-11-24 10:20:12,661 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.621e+01 9.085e+01 9.764e+01 1.357e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 10:20:23,637 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2791273.3333333335, ans=0.0
2023-11-24 10:20:34,210 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418700
2023-11-24 10:20:47,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2791340.0, ans=0.2
2023-11-24 10:20:50,731 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9900, loss[loss=0.08138, simple_loss=0.1249, pruned_loss=0.01351, audio_tagging_loss=0.005426, over 14958.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09108, pruned_loss=0.01297, audio_tagging_loss=0.008722, over 3041653.56 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:21:12,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2791473.3333333335, ans=0.0
2023-11-24 10:21:26,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2791540.0, ans=0.125
2023-11-24 10:21:36,765 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418750
2023-11-24 10:21:52,131 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 9950, loss[loss=0.06833, simple_loss=0.08738, pruned_loss=0.01282, audio_tagging_loss=0.01182, over 14754.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.09058, pruned_loss=0.01284, audio_tagging_loss=0.008747, over 3040286.04 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:21:58,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2791740.0, ans=0.2
2023-11-24 10:22:06,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2791806.6666666665, ans=0.1
2023-11-24 10:22:18,409 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.444e+01 9.082e+01 9.757e+01 1.187e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 10:22:28,775 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.96 vs. limit=10.0
2023-11-24 10:22:39,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418800
2023-11-24 10:22:46,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2792006.6666666665, ans=0.025
2023-11-24 10:22:55,823 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10000, loss[loss=0.08177, simple_loss=0.1237, pruned_loss=0.01405, audio_tagging_loss=0.005868, over 15463.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.09037, pruned_loss=0.01277, audio_tagging_loss=0.008758, over 3044044.96 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:22:56,351 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.05 vs. limit=6.0
2023-11-24 10:22:59,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2792073.3333333335, ans=0.0
2023-11-24 10:23:01,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2792073.3333333335, ans=0.125
2023-11-24 10:23:02,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2792073.3333333335, ans=0.125
2023-11-24 10:23:02,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2792073.3333333335, ans=0.2
2023-11-24 10:23:03,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2792073.3333333335, ans=0.0
2023-11-24 10:23:30,094 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.74 vs. limit=15.0
2023-11-24 10:23:30,824 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2792206.6666666665, ans=0.125
2023-11-24 10:23:42,082 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418850
2023-11-24 10:23:47,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2792340.0, ans=0.125
2023-11-24 10:23:59,927 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10050, loss[loss=0.08308, simple_loss=0.1145, pruned_loss=0.01888, audio_tagging_loss=0.006959, over 15630.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09101, pruned_loss=0.01282, audio_tagging_loss=0.008765, over 3046627.12 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:24:24,438 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:24:25,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.576e+01 9.106e+01 9.929e+01 1.520e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 10:24:26,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2792540.0, ans=0.125
2023-11-24 10:24:28,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2792540.0, ans=0.0
2023-11-24 10:24:29,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2792540.0, ans=0.0
2023-11-24 10:24:37,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2792606.6666666665, ans=0.0
2023-11-24 10:24:38,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2792606.6666666665, ans=0.2
2023-11-24 10:24:39,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2792606.6666666665, ans=0.125
2023-11-24 10:24:41,239 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2792606.6666666665, ans=0.125
2023-11-24 10:24:43,001 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.03 vs. limit=10.0
2023-11-24 10:24:47,104 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418900
2023-11-24 10:24:52,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2792673.3333333335, ans=0.125
2023-11-24 10:25:02,628 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10100, loss[loss=0.09467, simple_loss=0.1216, pruned_loss=0.02692, audio_tagging_loss=0.006927, over 16173.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09131, pruned_loss=0.01292, audio_tagging_loss=0.008878, over 3045423.02 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:25:15,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2792806.6666666665, ans=0.0
2023-11-24 10:25:19,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2792806.6666666665, ans=0.0
2023-11-24 10:25:32,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2792873.3333333335, ans=0.0
2023-11-24 10:25:48,981 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 418950
2023-11-24 10:25:51,287 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:26:05,129 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10150, loss[loss=0.07649, simple_loss=0.1026, pruned_loss=0.01713, audio_tagging_loss=0.008042, over 15973.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09045, pruned_loss=0.0128, audio_tagging_loss=0.008918, over 3040654.53 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:26:10,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2793073.3333333335, ans=0.125
2023-11-24 10:26:14,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2793073.3333333335, ans=0.125
2023-11-24 10:26:14,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2793073.3333333335, ans=0.125
2023-11-24 10:26:31,167 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.136e+01 8.585e+01 9.069e+01 9.759e+01 1.192e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 10:26:33,576 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:26:50,933 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419000
2023-11-24 10:27:08,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2793406.6666666665, ans=0.125
2023-11-24 10:27:08,926 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10200, loss[loss=0.04849, simple_loss=0.06256, pruned_loss=0.005945, audio_tagging_loss=0.01126, over 15645.00 frames. ], tot_loss[loss=0.06625, simple_loss=0.08914, pruned_loss=0.01259, audio_tagging_loss=0.009084, over 3039940.66 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:27:30,991 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:27:40,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.68 vs. limit=15.0
2023-11-24 10:27:46,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2793606.6666666665, ans=0.0
2023-11-24 10:27:55,676 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419050
2023-11-24 10:28:04,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2793673.3333333335, ans=0.0
2023-11-24 10:28:10,833 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10250, loss[loss=0.07089, simple_loss=0.08948, pruned_loss=0.01505, audio_tagging_loss=0.0111, over 14730.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08956, pruned_loss=0.01282, audio_tagging_loss=0.009079, over 3046099.74 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:28:37,729 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.437e+01 9.083e+01 9.683e+01 1.798e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 10:28:57,497 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419100
2023-11-24 10:29:07,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2794006.6666666665, ans=0.05
2023-11-24 10:29:11,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2794006.6666666665, ans=0.015
2023-11-24 10:29:13,554 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10300, loss[loss=0.07093, simple_loss=0.09191, pruned_loss=0.0171, audio_tagging_loss=0.007873, over 16061.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.08976, pruned_loss=0.01306, audio_tagging_loss=0.009037, over 3050679.84 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:29:20,026 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.86 vs. limit=15.0
2023-11-24 10:29:35,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2794140.0, ans=0.125
2023-11-24 10:29:37,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2794140.0, ans=0.125
2023-11-24 10:29:59,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2794273.3333333335, ans=0.125
2023-11-24 10:30:00,373 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419150
2023-11-24 10:30:06,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2794340.0, ans=0.125
2023-11-24 10:30:16,909 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10350, loss[loss=0.05465, simple_loss=0.07004, pruned_loss=0.009877, audio_tagging_loss=0.00975, over 16196.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.08963, pruned_loss=0.01311, audio_tagging_loss=0.009077, over 3055420.64 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:30:35,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2794473.3333333335, ans=0.07
2023-11-24 10:30:35,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2794473.3333333335, ans=0.125
2023-11-24 10:30:42,291 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.615e+01 9.196e+01 1.008e+02 1.353e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 10:30:49,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2794540.0, ans=0.1
2023-11-24 10:31:00,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2794606.6666666665, ans=0.0
2023-11-24 10:31:00,833 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:31:03,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419200
2023-11-24 10:31:19,394 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10400, loss[loss=0.05801, simple_loss=0.08071, pruned_loss=0.01036, audio_tagging_loss=0.007293, over 14418.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09009, pruned_loss=0.0133, audio_tagging_loss=0.009109, over 3049149.30 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:31:19,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2794740.0, ans=0.125
2023-11-24 10:31:36,096 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.27 vs. limit=15.0
2023-11-24 10:32:00,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2794940.0, ans=0.0
2023-11-24 10:32:05,594 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419250
2023-11-24 10:32:05,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2794940.0, ans=0.0
2023-11-24 10:32:19,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.84 vs. limit=12.0
2023-11-24 10:32:21,632 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10450, loss[loss=0.07864, simple_loss=0.1055, pruned_loss=0.01763, audio_tagging_loss=0.00825, over 15322.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09019, pruned_loss=0.01338, audio_tagging_loss=0.00919, over 3049637.83 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:32:22,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.35 vs. limit=6.0
2023-11-24 10:32:46,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2795206.6666666665, ans=0.0
2023-11-24 10:32:49,179 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.462e+01 9.061e+01 9.658e+01 1.208e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 10:32:55,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2795206.6666666665, ans=0.125
2023-11-24 10:32:59,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.48 vs. limit=15.0
2023-11-24 10:33:01,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2795273.3333333335, ans=0.125
2023-11-24 10:33:07,148 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419300
2023-11-24 10:33:15,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2795340.0, ans=0.125
2023-11-24 10:33:24,644 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10500, loss[loss=0.07067, simple_loss=0.09424, pruned_loss=0.01481, audio_tagging_loss=0.008736, over 14554.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09007, pruned_loss=0.01322, audio_tagging_loss=0.009013, over 3051623.91 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:33:54,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2795540.0, ans=0.2
2023-11-24 10:34:00,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.58 vs. limit=22.5
2023-11-24 10:34:11,666 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419350
2023-11-24 10:34:13,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2795606.6666666665, ans=0.125
2023-11-24 10:34:19,441 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.27 vs. limit=15.0
2023-11-24 10:34:27,459 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10550, loss[loss=0.05895, simple_loss=0.07669, pruned_loss=0.01066, audio_tagging_loss=0.009944, over 15690.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09146, pruned_loss=0.01345, audio_tagging_loss=0.008881, over 3051668.34 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:34:31,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2795740.0, ans=0.05
2023-11-24 10:34:31,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2795740.0, ans=0.125
2023-11-24 10:34:40,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2795806.6666666665, ans=0.125
2023-11-24 10:34:52,027 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2795873.3333333335, ans=0.125
2023-11-24 10:34:53,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2795873.3333333335, ans=0.125
2023-11-24 10:34:54,571 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.441e+01 8.781e+01 9.299e+01 1.006e+02 1.266e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 10:35:07,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2795940.0, ans=0.125
2023-11-24 10:35:13,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419400
2023-11-24 10:35:23,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2796006.6666666665, ans=0.0
2023-11-24 10:35:29,253 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10600, loss[loss=0.06546, simple_loss=0.09108, pruned_loss=0.01283, audio_tagging_loss=0.007092, over 16006.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09186, pruned_loss=0.01339, audio_tagging_loss=0.008864, over 3048630.87 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:35:33,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2796073.3333333335, ans=0.0
2023-11-24 10:35:38,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.88 vs. limit=12.0
2023-11-24 10:35:39,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2796073.3333333335, ans=0.125
2023-11-24 10:35:56,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2796206.6666666665, ans=0.0
2023-11-24 10:35:59,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2796206.6666666665, ans=0.0
2023-11-24 10:36:02,042 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2796206.6666666665, ans=0.125
2023-11-24 10:36:15,407 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419450
2023-11-24 10:36:26,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2796340.0, ans=0.035
2023-11-24 10:36:32,459 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10650, loss[loss=0.07193, simple_loss=0.09793, pruned_loss=0.01109, audio_tagging_loss=0.01187, over 14139.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09137, pruned_loss=0.01333, audio_tagging_loss=0.008919, over 3041208.61 frames. ], batch size: 52, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:36:32,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2796406.6666666665, ans=0.0
2023-11-24 10:36:36,964 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.30 vs. limit=15.0
2023-11-24 10:36:59,535 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.535e+01 9.205e+01 9.977e+01 1.279e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 10:37:04,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2796540.0, ans=0.2
2023-11-24 10:37:04,976 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.70 vs. limit=22.5
2023-11-24 10:37:18,809 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419500
2023-11-24 10:37:29,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2796673.3333333335, ans=0.2
2023-11-24 10:37:34,635 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10700, loss[loss=0.05835, simple_loss=0.08346, pruned_loss=0.008955, audio_tagging_loss=0.007671, over 15784.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09147, pruned_loss=0.01326, audio_tagging_loss=0.008899, over 3041635.39 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:37:36,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2796740.0, ans=0.2
2023-11-24 10:37:53,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2796806.6666666665, ans=0.125
2023-11-24 10:37:57,951 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.76 vs. limit=15.0
2023-11-24 10:38:20,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2796940.0, ans=0.95
2023-11-24 10:38:21,151 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419550
2023-11-24 10:38:35,106 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2797006.6666666665, ans=0.125
2023-11-24 10:38:37,184 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10750, loss[loss=0.07366, simple_loss=0.09556, pruned_loss=0.01757, audio_tagging_loss=0.008313, over 15578.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.0916, pruned_loss=0.01332, audio_tagging_loss=0.008766, over 3042466.60 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:39:03,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2797206.6666666665, ans=0.125
2023-11-24 10:39:05,296 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.423e+01 9.138e+01 9.778e+01 1.311e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 10:39:21,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2797273.3333333335, ans=0.0
2023-11-24 10:39:24,010 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419600
2023-11-24 10:39:40,627 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10800, loss[loss=0.09334, simple_loss=0.1294, pruned_loss=0.01971, audio_tagging_loss=0.008901, over 15369.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.0911, pruned_loss=0.01319, audio_tagging_loss=0.008783, over 3044538.09 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:39:42,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2797406.6666666665, ans=0.125
2023-11-24 10:40:02,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2797473.3333333335, ans=0.125
2023-11-24 10:40:05,705 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.61 vs. limit=15.0
2023-11-24 10:40:15,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.17 vs. limit=6.0
2023-11-24 10:40:26,298 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419650
2023-11-24 10:40:42,943 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10850, loss[loss=0.09847, simple_loss=0.137, pruned_loss=0.02305, audio_tagging_loss=0.006933, over 15392.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09144, pruned_loss=0.01328, audio_tagging_loss=0.008729, over 3039738.61 frames. ], batch size: 54, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:40:44,387 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2797740.0, ans=0.125
2023-11-24 10:41:09,949 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.591e+01 9.237e+01 1.014e+02 1.325e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 10:41:20,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-24 10:41:27,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.47 vs. limit=10.0
2023-11-24 10:41:29,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419700
2023-11-24 10:41:35,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.35 vs. limit=22.5
2023-11-24 10:41:39,722 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:41:44,408 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10900, loss[loss=0.06269, simple_loss=0.08751, pruned_loss=0.01064, audio_tagging_loss=0.008295, over 14804.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09065, pruned_loss=0.01313, audio_tagging_loss=0.00889, over 3037486.19 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:41:53,411 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.66 vs. limit=22.5
2023-11-24 10:42:03,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2798140.0, ans=0.125
2023-11-24 10:42:30,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419750
2023-11-24 10:42:47,317 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 10950, loss[loss=0.05881, simple_loss=0.07188, pruned_loss=0.01078, audio_tagging_loss=0.01209, over 15003.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09148, pruned_loss=0.01329, audio_tagging_loss=0.008831, over 3043440.92 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:43:14,656 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.263e+01 9.239e+01 9.893e+01 2.160e+02, threshold=1.848e+02, percent-clipped=1.0
2023-11-24 10:43:15,248 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.77 vs. limit=15.0
2023-11-24 10:43:23,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2798606.6666666665, ans=0.1
2023-11-24 10:43:33,334 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.12 vs. limit=22.5
2023-11-24 10:43:33,884 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419800
2023-11-24 10:43:45,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2798673.3333333335, ans=0.2
2023-11-24 10:43:50,667 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11000, loss[loss=0.05306, simple_loss=0.06918, pruned_loss=0.009241, audio_tagging_loss=0.009228, over 14372.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09077, pruned_loss=0.013, audio_tagging_loss=0.008853, over 3044672.30 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:43:55,016 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.53 vs. limit=12.0
2023-11-24 10:43:58,912 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:44:21,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.46 vs. limit=15.0
2023-11-24 10:44:32,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2798940.0, ans=0.1
2023-11-24 10:44:37,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419850
2023-11-24 10:44:39,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2799006.6666666665, ans=0.125
2023-11-24 10:44:52,417 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11050, loss[loss=0.0802, simple_loss=0.1104, pruned_loss=0.01657, audio_tagging_loss=0.008418, over 15781.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.0904, pruned_loss=0.01303, audio_tagging_loss=0.008913, over 3038932.45 frames. ], batch size: 59, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:45:20,196 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.86 vs. limit=10.0
2023-11-24 10:45:21,959 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.396e+01 8.466e+01 9.230e+01 9.960e+01 1.939e+02, threshold=1.846e+02, percent-clipped=1.0
2023-11-24 10:45:23,827 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.88 vs. limit=6.0
2023-11-24 10:45:25,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2799206.6666666665, ans=0.0
2023-11-24 10:45:31,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2799273.3333333335, ans=0.125
2023-11-24 10:45:38,866 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419900
2023-11-24 10:45:55,569 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11100, loss[loss=0.08844, simple_loss=0.1299, pruned_loss=0.01802, audio_tagging_loss=0.005449, over 14979.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09001, pruned_loss=0.01305, audio_tagging_loss=0.009015, over 3039676.74 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:46:26,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2799540.0, ans=0.0
2023-11-24 10:46:41,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 419950
2023-11-24 10:46:44,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2799673.3333333335, ans=0.09899494936611666
2023-11-24 10:46:50,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.12 vs. limit=22.5
2023-11-24 10:46:52,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2799673.3333333335, ans=0.125
2023-11-24 10:46:52,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2799673.3333333335, ans=0.125
2023-11-24 10:46:54,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2799673.3333333335, ans=0.025
2023-11-24 10:46:58,126 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11150, loss[loss=0.07738, simple_loss=0.1068, pruned_loss=0.01427, audio_tagging_loss=0.009692, over 13937.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09039, pruned_loss=0.01315, audio_tagging_loss=0.009075, over 3041700.16 frames. ], batch size: 52, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:46:58,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2799740.0, ans=0.125
2023-11-24 10:47:13,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2799806.6666666665, ans=0.07
2023-11-24 10:47:15,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2799806.6666666665, ans=0.09899494936611666
2023-11-24 10:47:17,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=2799806.6666666665, ans=0.02
2023-11-24 10:47:25,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2799873.3333333335, ans=15.0
2023-11-24 10:47:26,046 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.463e+01 9.073e+01 9.645e+01 1.170e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 10:47:44,398 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420000
2023-11-24 10:48:03,616 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11200, loss[loss=0.04937, simple_loss=0.06486, pruned_loss=0.006339, audio_tagging_loss=0.01059, over 15075.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09053, pruned_loss=0.01313, audio_tagging_loss=0.009209, over 3039231.08 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:48:07,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2800073.3333333335, ans=0.125
2023-11-24 10:48:09,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2800073.3333333335, ans=0.125
2023-11-24 10:48:11,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2800073.3333333335, ans=0.2
2023-11-24 10:48:20,197 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.15 vs. limit=15.0
2023-11-24 10:48:25,569 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.10 vs. limit=12.0
2023-11-24 10:48:50,452 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420050
2023-11-24 10:48:53,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2800340.0, ans=0.125
2023-11-24 10:48:59,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2800340.0, ans=0.125
2023-11-24 10:49:06,601 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11250, loss[loss=0.07564, simple_loss=0.1024, pruned_loss=0.01601, audio_tagging_loss=0.008422, over 14648.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09018, pruned_loss=0.01305, audio_tagging_loss=0.00921, over 3048168.07 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:49:22,283 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.58 vs. limit=15.0
2023-11-24 10:49:34,244 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.75 vs. limit=15.0
2023-11-24 10:49:35,710 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.479e+01 9.057e+01 9.752e+01 1.909e+02, threshold=1.811e+02, percent-clipped=1.0
2023-11-24 10:49:35,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2800540.0, ans=0.04949747468305833
2023-11-24 10:49:49,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.44 vs. limit=15.0
2023-11-24 10:49:52,792 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420100
2023-11-24 10:50:09,688 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11300, loss[loss=0.07291, simple_loss=0.09636, pruned_loss=0.01467, audio_tagging_loss=0.01006, over 15383.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.08987, pruned_loss=0.01291, audio_tagging_loss=0.009073, over 3049572.22 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:50:22,311 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.28 vs. limit=10.0
2023-11-24 10:50:47,883 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.27 vs. limit=10.0
2023-11-24 10:50:55,473 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420150
2023-11-24 10:50:55,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2800940.0, ans=0.125
2023-11-24 10:50:55,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2800940.0, ans=0.0
2023-11-24 10:50:59,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2801006.6666666665, ans=0.0
2023-11-24 10:51:06,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2801006.6666666665, ans=0.125
2023-11-24 10:51:10,848 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11350, loss[loss=0.06491, simple_loss=0.09361, pruned_loss=0.01057, audio_tagging_loss=0.007538, over 15894.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09109, pruned_loss=0.01318, audio_tagging_loss=0.008917, over 3053998.36 frames. ], batch size: 60, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:51:14,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2801073.3333333335, ans=0.125
2023-11-24 10:51:37,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2801206.6666666665, ans=0.125
2023-11-24 10:51:40,023 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.605e+01 8.545e+01 9.244e+01 9.787e+01 1.195e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 10:51:56,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420200
2023-11-24 10:52:04,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2801340.0, ans=10.0
2023-11-24 10:52:13,247 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11400, loss[loss=0.07147, simple_loss=0.08921, pruned_loss=0.01951, audio_tagging_loss=0.007363, over 15408.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09125, pruned_loss=0.01324, audio_tagging_loss=0.00876, over 3050261.45 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:52:17,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.68 vs. limit=22.5
2023-11-24 10:52:34,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2801473.3333333335, ans=0.0
2023-11-24 10:52:46,177 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.23 vs. limit=22.5
2023-11-24 10:52:49,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2801540.0, ans=0.125
2023-11-24 10:52:50,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2801606.6666666665, ans=0.2
2023-11-24 10:52:58,859 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2801606.6666666665, ans=0.125
2023-11-24 10:53:00,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420250
2023-11-24 10:53:00,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2801606.6666666665, ans=0.1
2023-11-24 10:53:00,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2801606.6666666665, ans=0.2
2023-11-24 10:53:17,817 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11450, loss[loss=0.05782, simple_loss=0.07569, pruned_loss=0.01005, audio_tagging_loss=0.009924, over 15195.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09184, pruned_loss=0.01333, audio_tagging_loss=0.008728, over 3045181.84 frames. ], batch size: 59, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:53:26,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2801740.0, ans=0.1
2023-11-24 10:53:27,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.96 vs. limit=22.5
2023-11-24 10:53:38,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2801806.6666666665, ans=0.07
2023-11-24 10:53:46,377 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.482e+01 8.489e+01 9.216e+01 1.000e+02 1.338e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 10:54:03,635 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420300
2023-11-24 10:54:07,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.30 vs. limit=15.0
2023-11-24 10:54:10,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.08 vs. limit=15.0
2023-11-24 10:54:16,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2802006.6666666665, ans=0.1
2023-11-24 10:54:19,459 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11500, loss[loss=0.08456, simple_loss=0.1165, pruned_loss=0.01671, audio_tagging_loss=0.009603, over 14720.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09177, pruned_loss=0.01338, audio_tagging_loss=0.00871, over 3049149.00 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:54:34,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2802140.0, ans=0.1
2023-11-24 10:54:41,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2802140.0, ans=0.0
2023-11-24 10:55:05,366 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420350
2023-11-24 10:55:10,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2802340.0, ans=0.1
2023-11-24 10:55:20,738 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11550, loss[loss=0.08127, simple_loss=0.1149, pruned_loss=0.01675, audio_tagging_loss=0.007048, over 14683.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09174, pruned_loss=0.01337, audio_tagging_loss=0.008771, over 3052963.26 frames. ], batch size: 52, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:55:20,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2802406.6666666665, ans=0.1
2023-11-24 10:55:50,994 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.625e+01 9.353e+01 9.944e+01 1.426e+02, threshold=1.871e+02, percent-clipped=0.0
2023-11-24 10:55:51,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2802540.0, ans=0.1
2023-11-24 10:55:53,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2802540.0, ans=0.0
2023-11-24 10:55:56,970 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:55:57,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2802606.6666666665, ans=0.0
2023-11-24 10:56:06,441 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420400
2023-11-24 10:56:06,913 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.29 vs. limit=22.5
2023-11-24 10:56:10,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2802673.3333333335, ans=0.125
2023-11-24 10:56:10,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2802673.3333333335, ans=0.1
2023-11-24 10:56:13,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2802673.3333333335, ans=0.2
2023-11-24 10:56:22,930 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11600, loss[loss=0.07911, simple_loss=0.1096, pruned_loss=0.01561, audio_tagging_loss=0.008698, over 15916.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09248, pruned_loss=0.01353, audio_tagging_loss=0.008771, over 3059726.53 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:56:24,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.82 vs. limit=12.0
2023-11-24 10:56:39,446 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.44 vs. limit=15.0
2023-11-24 10:56:45,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2802806.6666666665, ans=0.0
2023-11-24 10:56:49,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2802873.3333333335, ans=0.0
2023-11-24 10:56:49,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2802873.3333333335, ans=0.125
2023-11-24 10:56:54,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2802873.3333333335, ans=0.125
2023-11-24 10:57:08,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420450
2023-11-24 10:57:18,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2803006.6666666665, ans=0.125
2023-11-24 10:57:24,768 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11650, loss[loss=0.07501, simple_loss=0.1052, pruned_loss=0.01339, audio_tagging_loss=0.009014, over 15566.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09201, pruned_loss=0.01338, audio_tagging_loss=0.008797, over 3058037.37 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:57:30,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2803073.3333333335, ans=0.125
2023-11-24 10:57:30,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2803073.3333333335, ans=0.1
2023-11-24 10:57:46,255 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.27 vs. limit=15.0
2023-11-24 10:57:54,137 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.520e+01 9.268e+01 1.018e+02 1.416e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 10:58:03,334 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.72 vs. limit=15.0
2023-11-24 10:58:04,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2803273.3333333335, ans=0.125
2023-11-24 10:58:10,502 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420500
2023-11-24 10:58:20,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2803340.0, ans=0.0
2023-11-24 10:58:21,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2803340.0, ans=0.125
2023-11-24 10:58:25,799 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11700, loss[loss=0.08965, simple_loss=0.1337, pruned_loss=0.01656, audio_tagging_loss=0.006244, over 15658.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.0916, pruned_loss=0.01325, audio_tagging_loss=0.008824, over 3053318.14 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:58:59,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2803540.0, ans=0.125
2023-11-24 10:59:03,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2803606.6666666665, ans=0.2
2023-11-24 10:59:11,901 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420550
2023-11-24 10:59:19,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2803673.3333333335, ans=0.0
2023-11-24 10:59:27,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2803740.0, ans=0.2
2023-11-24 10:59:28,189 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0
2023-11-24 10:59:28,515 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11750, loss[loss=0.05138, simple_loss=0.06942, pruned_loss=0.008263, audio_tagging_loss=0.008406, over 15095.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.0909, pruned_loss=0.0131, audio_tagging_loss=0.008836, over 3051462.42 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:59:30,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2803740.0, ans=0.04949747468305833
2023-11-24 10:59:57,407 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.255e+01 8.962e+01 9.569e+01 1.238e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 11:00:13,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420600
2023-11-24 11:00:14,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2803940.0, ans=0.1
2023-11-24 11:00:24,530 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.12 vs. limit=15.0
2023-11-24 11:00:29,778 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11800, loss[loss=0.07901, simple_loss=0.1076, pruned_loss=0.0164, audio_tagging_loss=0.008837, over 15587.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09174, pruned_loss=0.01319, audio_tagging_loss=0.008755, over 3049697.78 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:00:36,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2804073.3333333335, ans=0.05
2023-11-24 11:01:16,283 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420650
2023-11-24 11:01:24,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2804340.0, ans=0.1
2023-11-24 11:01:26,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2804340.0, ans=0.125
2023-11-24 11:01:32,373 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11850, loss[loss=0.06202, simple_loss=0.07247, pruned_loss=0.01155, audio_tagging_loss=0.01423, over 15455.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09131, pruned_loss=0.01306, audio_tagging_loss=0.008896, over 3045139.27 frames. ], batch size: 59, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:01:32,610 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:01:32,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2804406.6666666665, ans=0.0
2023-11-24 11:01:59,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2804540.0, ans=0.125
2023-11-24 11:02:02,503 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.352e+01 9.070e+01 9.952e+01 1.461e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 11:02:18,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420700
2023-11-24 11:02:19,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2804606.6666666665, ans=0.125
2023-11-24 11:02:27,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2804673.3333333335, ans=0.125
2023-11-24 11:02:32,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2804673.3333333335, ans=0.1
2023-11-24 11:02:34,396 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11900, loss[loss=0.0655, simple_loss=0.08054, pruned_loss=0.01548, audio_tagging_loss=0.009748, over 15496.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09138, pruned_loss=0.01327, audio_tagging_loss=0.009089, over 3048586.69 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:02:46,713 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.18 vs. limit=15.0
2023-11-24 11:02:58,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2804873.3333333335, ans=0.1
2023-11-24 11:03:20,546 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420750
2023-11-24 11:03:37,231 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 11950, loss[loss=0.0594, simple_loss=0.07301, pruned_loss=0.01234, audio_tagging_loss=0.01056, over 15119.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09033, pruned_loss=0.01309, audio_tagging_loss=0.009172, over 3050375.65 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:03:37,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2805073.3333333335, ans=0.125
2023-11-24 11:03:37,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2805073.3333333335, ans=0.1
2023-11-24 11:04:03,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2805206.6666666665, ans=0.0
2023-11-24 11:04:07,047 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.367e+01 9.062e+01 9.589e+01 1.237e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 11:04:11,685 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2805206.6666666665, ans=0.0
2023-11-24 11:04:15,398 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.99 vs. limit=15.0
2023-11-24 11:04:22,819 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420800
2023-11-24 11:04:37,826 INFO [train_asr.py:1221] (2/4) Epoch 35, batch 12000, loss[loss=0.05971, simple_loss=0.07982, pruned_loss=0.01138, audio_tagging_loss=0.00842, over 15751.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09011, pruned_loss=0.01311, audio_tagging_loss=0.009206, over 3045629.05 frames. ], batch size: 61, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:04:37,827 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 11:05:10,163 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.9295, 1.5946, 3.5083, 3.0863, 3.0455, 3.0783, 3.2012, 3.2304],
       device='cuda:2')
2023-11-24 11:05:19,866 INFO [train_asr.py:1253] (2/4) Epoch 35, validation: loss=0.0585, simple_loss=0.05078, pruned_loss=0.005085, audio_tagging_loss=0.02803, over 4681554.00 frames. 
2023-11-24 11:05:19,867 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 11:05:24,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2805406.6666666665, ans=0.05
2023-11-24 11:05:26,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2805406.6666666665, ans=0.1
2023-11-24 11:05:28,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0
2023-11-24 11:05:34,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=2805473.3333333335, ans=15.0
2023-11-24 11:05:42,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2805540.0, ans=0.1
2023-11-24 11:06:25,322 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 0, loss[loss=0.07335, simple_loss=0.1002, pruned_loss=0.006639, audio_tagging_loss=0.01659, over 14831.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.1002, pruned_loss=0.006639, audio_tagging_loss=0.01659, over 14831.00 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:06:25,322 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 11:06:44,482 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.1813, 4.9592, 4.3436, 4.8606], device='cuda:2')
2023-11-24 11:07:04,363 INFO [train_asr.py:1253] (2/4) Epoch 36, validation: loss=0.05761, simple_loss=0.0508, pruned_loss=0.005078, audio_tagging_loss=0.02713, over 4681554.00 frames. 
2023-11-24 11:07:04,364 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 11:07:08,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2805553.3333333335, ans=0.0
2023-11-24 11:07:08,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2805553.3333333335, ans=0.125
2023-11-24 11:07:14,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2805553.3333333335, ans=0.125
2023-11-24 11:07:22,249 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420850
2023-11-24 11:07:58,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2805820.0, ans=0.1
2023-11-24 11:08:06,407 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 50, loss[loss=0.07666, simple_loss=0.1007, pruned_loss=0.01329, audio_tagging_loss=0.013, over 14962.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09004, pruned_loss=0.01247, audio_tagging_loss=0.01721, over 680545.49 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:08:07,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2805886.6666666665, ans=0.2
2023-11-24 11:08:09,899 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.471e+01 9.077e+01 9.826e+01 1.067e+02 1.319e+02, threshold=1.965e+02, percent-clipped=0.0
2023-11-24 11:08:14,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2805886.6666666665, ans=0.125
2023-11-24 11:08:15,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2805886.6666666665, ans=15.0
2023-11-24 11:08:21,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2805953.3333333335, ans=0.125
2023-11-24 11:08:25,423 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420900
2023-11-24 11:08:39,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2806020.0, ans=0.1
2023-11-24 11:08:43,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2806086.6666666665, ans=0.125
2023-11-24 11:09:07,570 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.28 vs. limit=6.0
2023-11-24 11:09:08,023 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 100, loss[loss=0.07165, simple_loss=0.09073, pruned_loss=0.0142, audio_tagging_loss=0.01209, over 14318.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09147, pruned_loss=0.01277, audio_tagging_loss=0.01649, over 1202658.90 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:09:11,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2806220.0, ans=0.0
2023-11-24 11:09:12,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2806220.0, ans=0.1
2023-11-24 11:09:16,845 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.81 vs. limit=15.0
2023-11-24 11:09:24,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2806286.6666666665, ans=0.09899494936611666
2023-11-24 11:09:27,616 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 420950
2023-11-24 11:09:27,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2806286.6666666665, ans=0.0
2023-11-24 11:09:43,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2806353.3333333335, ans=0.125
2023-11-24 11:10:00,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2806486.6666666665, ans=0.0
2023-11-24 11:10:05,356 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2806486.6666666665, ans=0.125
2023-11-24 11:10:06,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2806486.6666666665, ans=0.0
2023-11-24 11:10:11,678 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 150, loss[loss=0.069, simple_loss=0.08465, pruned_loss=0.01064, audio_tagging_loss=0.01603, over 14665.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09101, pruned_loss=0.01276, audio_tagging_loss=0.01481, over 1610083.72 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:10:15,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 8.014e+01 9.024e+01 9.585e+01 1.046e+02 2.115e+02, threshold=1.917e+02, percent-clipped=1.0
2023-11-24 11:10:20,726 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.01 vs. limit=15.0
2023-11-24 11:10:22,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2806620.0, ans=0.035
2023-11-24 11:10:29,472 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421000
2023-11-24 11:10:34,034 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.73 vs. limit=15.0
2023-11-24 11:11:04,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2806820.0, ans=0.0
2023-11-24 11:11:13,392 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 200, loss[loss=0.05432, simple_loss=0.06736, pruned_loss=0.01182, audio_tagging_loss=0.008823, over 15228.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09256, pruned_loss=0.01309, audio_tagging_loss=0.01307, over 1926212.00 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:11:31,238 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421050
2023-11-24 11:11:59,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2807086.6666666665, ans=0.125
2023-11-24 11:12:14,916 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 250, loss[loss=0.06692, simple_loss=0.08089, pruned_loss=0.01339, audio_tagging_loss=0.01308, over 14170.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09167, pruned_loss=0.01296, audio_tagging_loss=0.01186, over 2171615.28 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:12:18,505 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.652e+01 9.389e+01 9.985e+01 1.276e+02, threshold=1.878e+02, percent-clipped=0.0
2023-11-24 11:12:18,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2807220.0, ans=0.125
2023-11-24 11:12:34,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421100
2023-11-24 11:12:41,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2807353.3333333335, ans=0.07
2023-11-24 11:12:43,350 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.56 vs. limit=15.0
2023-11-24 11:12:43,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=12.0
2023-11-24 11:13:02,073 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2807420.0, ans=0.125
2023-11-24 11:13:03,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2807486.6666666665, ans=0.125
2023-11-24 11:13:18,263 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 300, loss[loss=0.07043, simple_loss=0.09627, pruned_loss=0.01471, audio_tagging_loss=0.007591, over 15446.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09179, pruned_loss=0.01303, audio_tagging_loss=0.01106, over 2365306.38 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:13:19,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2807553.3333333335, ans=0.0
2023-11-24 11:13:31,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2807620.0, ans=0.125
2023-11-24 11:13:36,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421150
2023-11-24 11:13:38,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2807620.0, ans=0.125
2023-11-24 11:13:39,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2807620.0, ans=0.1
2023-11-24 11:13:48,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.28 vs. limit=15.0
2023-11-24 11:14:02,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2807753.3333333335, ans=0.0
2023-11-24 11:14:11,832 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2807820.0, ans=0.0
2023-11-24 11:14:19,892 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 350, loss[loss=0.05614, simple_loss=0.0726, pruned_loss=0.009046, audio_tagging_loss=0.01079, over 15308.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09189, pruned_loss=0.01303, audio_tagging_loss=0.01046, over 2517207.31 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:14:23,373 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.480e+01 9.093e+01 9.744e+01 1.336e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 11:14:30,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2807953.3333333335, ans=0.125
2023-11-24 11:14:32,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2807953.3333333335, ans=0.0
2023-11-24 11:14:38,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421200
2023-11-24 11:14:47,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2808020.0, ans=0.2
2023-11-24 11:15:21,386 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 400, loss[loss=0.06479, simple_loss=0.1, pruned_loss=0.008793, audio_tagging_loss=0.005984, over 15413.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09209, pruned_loss=0.013, audio_tagging_loss=0.01008, over 2635034.74 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:15:22,850 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2808220.0, ans=0.2
2023-11-24 11:15:30,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2808220.0, ans=0.125
2023-11-24 11:15:37,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2808286.6666666665, ans=0.025
2023-11-24 11:15:41,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421250
2023-11-24 11:15:48,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2808353.3333333335, ans=0.0
2023-11-24 11:16:04,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2808420.0, ans=0.1
2023-11-24 11:16:13,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.32 vs. limit=10.0
2023-11-24 11:16:23,730 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 450, loss[loss=0.06821, simple_loss=0.08927, pruned_loss=0.01251, audio_tagging_loss=0.01107, over 14367.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09215, pruned_loss=0.01311, audio_tagging_loss=0.009685, over 2728220.78 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:16:27,929 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.299e+01 9.052e+01 9.986e+01 1.410e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 11:16:40,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2808620.0, ans=0.0
2023-11-24 11:16:42,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421300
2023-11-24 11:17:15,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2808820.0, ans=0.125
2023-11-24 11:17:22,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2808820.0, ans=0.015
2023-11-24 11:17:26,671 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 500, loss[loss=0.08194, simple_loss=0.1149, pruned_loss=0.0162, audio_tagging_loss=0.008288, over 14411.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09136, pruned_loss=0.01304, audio_tagging_loss=0.00946, over 2796474.48 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:17:32,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2808886.6666666665, ans=10.0
2023-11-24 11:17:44,584 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421350
2023-11-24 11:17:48,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2808953.3333333335, ans=0.1
2023-11-24 11:18:10,914 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2809086.6666666665, ans=0.0
2023-11-24 11:18:24,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2809153.3333333335, ans=0.125
2023-11-24 11:18:28,539 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 550, loss[loss=0.08083, simple_loss=0.1189, pruned_loss=0.0147, audio_tagging_loss=0.006674, over 15297.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09156, pruned_loss=0.01297, audio_tagging_loss=0.009311, over 2859083.91 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:18:32,038 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.126e+01 8.386e+01 8.873e+01 9.796e+01 1.246e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 11:18:43,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2809286.6666666665, ans=0.5
2023-11-24 11:18:47,300 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421400
2023-11-24 11:18:47,809 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.17 vs. limit=22.5
2023-11-24 11:19:06,636 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2809420.0, ans=0.1
2023-11-24 11:19:11,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2809420.0, ans=0.09899494936611666
2023-11-24 11:19:17,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2809486.6666666665, ans=0.0
2023-11-24 11:19:23,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2809486.6666666665, ans=0.125
2023-11-24 11:19:27,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2809486.6666666665, ans=0.0
2023-11-24 11:19:31,296 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 600, loss[loss=0.06633, simple_loss=0.09399, pruned_loss=0.0116, audio_tagging_loss=0.007739, over 14689.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09203, pruned_loss=0.01308, audio_tagging_loss=0.009197, over 2899152.07 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:19:33,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2809553.3333333335, ans=10.0
2023-11-24 11:19:35,542 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.90 vs. limit=22.5
2023-11-24 11:19:50,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421450
2023-11-24 11:19:50,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2809620.0, ans=0.125
2023-11-24 11:20:02,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2809686.6666666665, ans=0.0
2023-11-24 11:20:07,337 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2809753.3333333335, ans=0.125
2023-11-24 11:20:33,874 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 650, loss[loss=0.06751, simple_loss=0.1004, pruned_loss=0.009126, audio_tagging_loss=0.008176, over 15385.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09209, pruned_loss=0.0131, audio_tagging_loss=0.009207, over 2937862.02 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:20:37,349 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.541e+01 9.260e+01 1.007e+02 1.291e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 11:20:51,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421500
2023-11-24 11:21:03,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2810020.0, ans=0.07
2023-11-24 11:21:06,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2810020.0, ans=0.1
2023-11-24 11:21:14,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.42 vs. limit=15.0
2023-11-24 11:21:23,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.98 vs. limit=10.0
2023-11-24 11:21:23,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2810153.3333333335, ans=0.04949747468305833
2023-11-24 11:21:35,321 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 700, loss[loss=0.0704, simple_loss=0.09199, pruned_loss=0.01375, audio_tagging_loss=0.01066, over 14299.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09144, pruned_loss=0.01313, audio_tagging_loss=0.00917, over 2963786.89 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:21:54,198 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421550
2023-11-24 11:21:58,408 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2810286.6666666665, ans=0.025
2023-11-24 11:22:09,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2810353.3333333335, ans=0.125
2023-11-24 11:22:17,964 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:22:20,633 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.61 vs. limit=15.0
2023-11-24 11:22:36,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2810553.3333333335, ans=0.1
2023-11-24 11:22:36,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2810553.3333333335, ans=0.0
2023-11-24 11:22:37,456 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 750, loss[loss=0.07389, simple_loss=0.1002, pruned_loss=0.01553, audio_tagging_loss=0.008247, over 14076.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09158, pruned_loss=0.01308, audio_tagging_loss=0.009117, over 2980073.83 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:22:38,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2810553.3333333335, ans=0.125
2023-11-24 11:22:41,366 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.522e+01 9.060e+01 9.732e+01 1.237e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 11:22:45,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2810553.3333333335, ans=0.125
2023-11-24 11:22:56,230 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421600
2023-11-24 11:23:01,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2810686.6666666665, ans=0.0
2023-11-24 11:23:05,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0
2023-11-24 11:23:15,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2810753.3333333335, ans=0.125
2023-11-24 11:23:19,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2810753.3333333335, ans=0.125
2023-11-24 11:23:21,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2810753.3333333335, ans=0.1
2023-11-24 11:23:21,309 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.41 vs. limit=15.0
2023-11-24 11:23:40,409 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 800, loss[loss=0.08441, simple_loss=0.1232, pruned_loss=0.01728, audio_tagging_loss=0.005558, over 15474.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09163, pruned_loss=0.01318, audio_tagging_loss=0.009162, over 2998440.63 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:23:40,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2810886.6666666665, ans=0.0
2023-11-24 11:23:54,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2810953.3333333335, ans=0.2
2023-11-24 11:23:58,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421650
2023-11-24 11:24:25,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2811086.6666666665, ans=0.125
2023-11-24 11:24:27,720 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2811086.6666666665, ans=0.125
2023-11-24 11:24:36,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2811153.3333333335, ans=0.125
2023-11-24 11:24:42,464 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 850, loss[loss=0.04433, simple_loss=0.05164, pruned_loss=0.006311, audio_tagging_loss=0.0122, over 14362.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09093, pruned_loss=0.01285, audio_tagging_loss=0.009256, over 3011348.85 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:24:47,201 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.597e+01 8.758e+01 9.246e+01 1.015e+02 2.108e+02, threshold=1.849e+02, percent-clipped=1.0
2023-11-24 11:25:01,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421700
2023-11-24 11:25:21,081 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.86 vs. limit=10.0
2023-11-24 11:25:27,580 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.19 vs. limit=15.0
2023-11-24 11:25:44,017 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=15.0
2023-11-24 11:25:45,249 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 900, loss[loss=0.06639, simple_loss=0.08634, pruned_loss=0.01345, audio_tagging_loss=0.009778, over 14196.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09032, pruned_loss=0.01292, audio_tagging_loss=0.009374, over 3012365.54 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:25:53,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2811553.3333333335, ans=0.125
2023-11-24 11:26:04,196 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421750
2023-11-24 11:26:16,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2811686.6666666665, ans=0.0
2023-11-24 11:26:22,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2811753.3333333335, ans=0.2
2023-11-24 11:26:47,789 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 950, loss[loss=0.06675, simple_loss=0.08833, pruned_loss=0.01558, audio_tagging_loss=0.007008, over 15381.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.08964, pruned_loss=0.01281, audio_tagging_loss=0.009261, over 3020897.35 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:26:52,396 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.504e+01 9.122e+01 9.832e+01 1.245e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 11:26:59,003 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=5.73 vs. limit=12.0
2023-11-24 11:27:06,038 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421800
2023-11-24 11:27:16,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2812020.0, ans=0.125
2023-11-24 11:27:17,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2812020.0, ans=0.125
2023-11-24 11:27:21,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2812020.0, ans=0.1
2023-11-24 11:27:32,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2812086.6666666665, ans=0.0
2023-11-24 11:27:39,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2812153.3333333335, ans=0.05
2023-11-24 11:27:39,229 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:27:49,562 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1000, loss[loss=0.07127, simple_loss=0.09418, pruned_loss=0.01645, audio_tagging_loss=0.007724, over 14819.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.08945, pruned_loss=0.01282, audio_tagging_loss=0.0091, over 3025309.86 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:27:53,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2812220.0, ans=0.125
2023-11-24 11:28:00,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.72 vs. limit=22.5
2023-11-24 11:28:03,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2812286.6666666665, ans=0.0
2023-11-24 11:28:08,616 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421850
2023-11-24 11:28:15,003 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:28:18,171 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2812353.3333333335, ans=0.2
2023-11-24 11:28:26,764 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.81 vs. limit=15.0
2023-11-24 11:28:42,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2812486.6666666665, ans=0.125
2023-11-24 11:28:51,811 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1050, loss[loss=0.05922, simple_loss=0.0763, pruned_loss=0.01055, audio_tagging_loss=0.01053, over 15074.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09059, pruned_loss=0.01297, audio_tagging_loss=0.008975, over 3033719.71 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:28:57,202 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.446e+01 9.134e+01 9.863e+01 1.540e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 11:29:05,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2812620.0, ans=0.125
2023-11-24 11:29:11,712 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421900
2023-11-24 11:29:27,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2812686.6666666665, ans=0.125
2023-11-24 11:29:38,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2812753.3333333335, ans=0.125
2023-11-24 11:29:39,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2812753.3333333335, ans=0.2
2023-11-24 11:29:55,484 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1100, loss[loss=0.05538, simple_loss=0.0755, pruned_loss=0.009579, audio_tagging_loss=0.008047, over 15704.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.0908, pruned_loss=0.01307, audio_tagging_loss=0.008823, over 3036413.77 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:29:57,912 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:29:59,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2812886.6666666665, ans=0.125
2023-11-24 11:30:13,221 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 421950
2023-11-24 11:30:25,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2813020.0, ans=0.035
2023-11-24 11:30:47,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2813153.3333333335, ans=0.125
2023-11-24 11:30:56,384 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1150, loss[loss=0.06902, simple_loss=0.08784, pruned_loss=0.01609, audio_tagging_loss=0.009006, over 16648.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09098, pruned_loss=0.01319, audio_tagging_loss=0.008755, over 3040698.61 frames. ], batch size: 64, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:31:01,046 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.442e+01 9.003e+01 9.685e+01 1.147e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 11:31:01,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2813220.0, ans=0.125
2023-11-24 11:31:03,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2813220.0, ans=0.125
2023-11-24 11:31:15,483 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422000
2023-11-24 11:31:19,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2813286.6666666665, ans=0.125
2023-11-24 11:31:41,733 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=2813420.0, ans=0.2
2023-11-24 11:31:56,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2813553.3333333335, ans=0.0
2023-11-24 11:31:58,433 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1200, loss[loss=0.05197, simple_loss=0.0663, pruned_loss=0.008144, audio_tagging_loss=0.01068, over 15533.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09072, pruned_loss=0.01322, audio_tagging_loss=0.008752, over 3032792.26 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:32:01,144 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2813553.3333333335, ans=0.125
2023-11-24 11:32:08,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2813553.3333333335, ans=0.1
2023-11-24 11:32:17,870 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422050
2023-11-24 11:32:35,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2813753.3333333335, ans=0.125
2023-11-24 11:32:36,022 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.80 vs. limit=15.0
2023-11-24 11:33:00,677 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1250, loss[loss=0.06948, simple_loss=0.09066, pruned_loss=0.01444, audio_tagging_loss=0.009711, over 16486.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09144, pruned_loss=0.0134, audio_tagging_loss=0.008757, over 3038852.20 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:33:07,042 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.529e+01 8.565e+01 9.272e+01 1.023e+02 1.182e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 11:33:08,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=12.61 vs. limit=22.5
2023-11-24 11:33:18,919 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422100
2023-11-24 11:33:25,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2814020.0, ans=0.2
2023-11-24 11:33:28,070 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.47 vs. limit=15.0
2023-11-24 11:33:37,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2814086.6666666665, ans=0.125
2023-11-24 11:33:38,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2814086.6666666665, ans=0.0
2023-11-24 11:33:50,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2814153.3333333335, ans=0.2
2023-11-24 11:34:01,804 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1300, loss[loss=0.0498, simple_loss=0.06582, pruned_loss=0.007724, audio_tagging_loss=0.009168, over 15228.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09076, pruned_loss=0.01325, audio_tagging_loss=0.008774, over 3031719.81 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:34:13,134 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.66 vs. limit=22.5
2023-11-24 11:34:19,680 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422150
2023-11-24 11:34:32,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2814353.3333333335, ans=0.1
2023-11-24 11:34:37,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2814353.3333333335, ans=0.2
2023-11-24 11:34:41,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2814420.0, ans=0.2
2023-11-24 11:34:48,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=22.5
2023-11-24 11:35:04,250 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1350, loss[loss=0.07976, simple_loss=0.1067, pruned_loss=0.01662, audio_tagging_loss=0.009782, over 15601.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09015, pruned_loss=0.01314, audio_tagging_loss=0.00882, over 3032389.38 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:35:10,798 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.582e+01 9.240e+01 9.907e+01 1.176e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 11:35:21,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2814620.0, ans=0.125
2023-11-24 11:35:22,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2814620.0, ans=0.125
2023-11-24 11:35:24,589 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422200
2023-11-24 11:35:42,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2814753.3333333335, ans=0.0
2023-11-24 11:35:49,781 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:36:01,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2814820.0, ans=0.0
2023-11-24 11:36:04,318 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:36:08,387 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1400, loss[loss=0.07819, simple_loss=0.104, pruned_loss=0.01658, audio_tagging_loss=0.009617, over 15871.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09047, pruned_loss=0.01301, audio_tagging_loss=0.008872, over 3038030.55 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:36:10,055 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.61 vs. limit=15.0
2023-11-24 11:36:19,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2814953.3333333335, ans=0.0
2023-11-24 11:36:22,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2814953.3333333335, ans=0.0
2023-11-24 11:36:24,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2814953.3333333335, ans=0.125
2023-11-24 11:36:26,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422250
2023-11-24 11:36:26,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2814953.3333333335, ans=0.2
2023-11-24 11:36:58,228 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2815153.3333333335, ans=0.0
2023-11-24 11:37:02,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2815153.3333333335, ans=0.1
2023-11-24 11:37:02,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2815153.3333333335, ans=0.0
2023-11-24 11:37:02,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.80 vs. limit=22.5
2023-11-24 11:37:10,646 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1450, loss[loss=0.07061, simple_loss=0.08887, pruned_loss=0.01334, audio_tagging_loss=0.01283, over 15958.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09113, pruned_loss=0.01311, audio_tagging_loss=0.008919, over 3037618.58 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:37:13,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.49 vs. limit=22.5
2023-11-24 11:37:15,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2815220.0, ans=0.125
2023-11-24 11:37:16,388 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.536e+01 9.281e+01 1.016e+02 1.384e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 11:37:28,222 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422300
2023-11-24 11:37:30,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2815286.6666666665, ans=0.0
2023-11-24 11:37:36,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2815353.3333333335, ans=0.1
2023-11-24 11:37:50,920 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.49 vs. limit=15.0
2023-11-24 11:38:11,603 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1500, loss[loss=0.1079, simple_loss=0.1451, pruned_loss=0.02936, audio_tagging_loss=0.006026, over 16094.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09189, pruned_loss=0.01344, audio_tagging_loss=0.00901, over 3036181.15 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:38:17,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2815553.3333333335, ans=0.0
2023-11-24 11:38:19,231 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.28 vs. limit=15.0
2023-11-24 11:38:27,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2815620.0, ans=0.1
2023-11-24 11:38:31,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422350
2023-11-24 11:38:31,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.63 vs. limit=15.0
2023-11-24 11:38:47,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2815686.6666666665, ans=0.1
2023-11-24 11:39:13,898 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1550, loss[loss=0.08139, simple_loss=0.1163, pruned_loss=0.01556, audio_tagging_loss=0.007661, over 15995.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.0915, pruned_loss=0.0134, audio_tagging_loss=0.00904, over 3030737.79 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:39:20,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.494e+01 9.231e+01 9.838e+01 1.696e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 11:39:32,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422400
2023-11-24 11:39:51,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2816086.6666666665, ans=0.0
2023-11-24 11:39:52,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.14 vs. limit=22.5
2023-11-24 11:39:56,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2816086.6666666665, ans=0.125
2023-11-24 11:40:01,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2816086.6666666665, ans=0.2
2023-11-24 11:40:14,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2816153.3333333335, ans=0.125
2023-11-24 11:40:17,307 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1600, loss[loss=0.08262, simple_loss=0.1112, pruned_loss=0.01849, audio_tagging_loss=0.008525, over 15196.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09119, pruned_loss=0.0134, audio_tagging_loss=0.00922, over 3033269.81 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:40:35,173 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422450
2023-11-24 11:40:43,516 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.65 vs. limit=15.0
2023-11-24 11:40:52,230 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0
2023-11-24 11:40:56,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2816420.0, ans=0.1
2023-11-24 11:41:03,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2816420.0, ans=0.0
2023-11-24 11:41:05,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2816420.0, ans=0.1
2023-11-24 11:41:18,984 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1650, loss[loss=0.07464, simple_loss=0.1022, pruned_loss=0.01495, audio_tagging_loss=0.008566, over 14921.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09154, pruned_loss=0.01347, audio_tagging_loss=0.009258, over 3037564.34 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:41:25,966 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.620e+01 9.407e+01 1.041e+02 1.351e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-24 11:41:37,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422500
2023-11-24 11:42:20,989 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1700, loss[loss=0.05774, simple_loss=0.08104, pruned_loss=0.008495, audio_tagging_loss=0.008728, over 15051.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09136, pruned_loss=0.01338, audio_tagging_loss=0.009239, over 3036773.88 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:42:21,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2816886.6666666665, ans=0.035
2023-11-24 11:42:26,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2816886.6666666665, ans=0.125
2023-11-24 11:42:40,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422550
2023-11-24 11:42:56,126 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:43:00,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2817086.6666666665, ans=0.125
2023-11-24 11:43:18,117 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2817153.3333333335, ans=0.0
2023-11-24 11:43:24,389 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1750, loss[loss=0.0796, simple_loss=0.1026, pruned_loss=0.02064, audio_tagging_loss=0.007643, over 14484.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09104, pruned_loss=0.01328, audio_tagging_loss=0.00913, over 3041026.38 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:43:24,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2817220.0, ans=0.125
2023-11-24 11:43:27,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2817220.0, ans=0.2
2023-11-24 11:43:28,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.59 vs. limit=15.0
2023-11-24 11:43:29,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2817220.0, ans=0.125
2023-11-24 11:43:31,537 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.461e+01 8.743e+01 9.239e+01 9.812e+01 1.237e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 11:43:42,389 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422600
2023-11-24 11:43:47,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2817353.3333333335, ans=0.125
2023-11-24 11:44:01,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2817420.0, ans=0.0
2023-11-24 11:44:11,900 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.33 vs. limit=15.0
2023-11-24 11:44:18,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2817486.6666666665, ans=0.125
2023-11-24 11:44:26,846 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1800, loss[loss=0.07021, simple_loss=0.08952, pruned_loss=0.01462, audio_tagging_loss=0.01083, over 15373.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09062, pruned_loss=0.01312, audio_tagging_loss=0.009127, over 3045582.15 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:44:27,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2817553.3333333335, ans=0.2
2023-11-24 11:44:45,800 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422650
2023-11-24 11:44:46,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2817620.0, ans=0.0
2023-11-24 11:45:22,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2817820.0, ans=0.0
2023-11-24 11:45:22,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2817820.0, ans=0.125
2023-11-24 11:45:23,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2817820.0, ans=0.0
2023-11-24 11:45:28,828 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1850, loss[loss=0.06592, simple_loss=0.09632, pruned_loss=0.009996, audio_tagging_loss=0.007762, over 16735.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.08978, pruned_loss=0.01291, audio_tagging_loss=0.009129, over 3049737.95 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:45:36,361 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.377e+01 8.376e+01 8.960e+01 9.835e+01 1.450e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 11:45:40,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2817886.6666666665, ans=15.0
2023-11-24 11:45:48,724 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422700
2023-11-24 11:46:00,757 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2818020.0, ans=0.125
2023-11-24 11:46:02,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2818020.0, ans=0.125
2023-11-24 11:46:10,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2818086.6666666665, ans=0.1
2023-11-24 11:46:31,722 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1900, loss[loss=0.06638, simple_loss=0.08306, pruned_loss=0.01737, audio_tagging_loss=0.007478, over 14064.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08884, pruned_loss=0.01282, audio_tagging_loss=0.009113, over 3048789.56 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 11:46:50,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422750
2023-11-24 11:46:53,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2818286.6666666665, ans=0.125
2023-11-24 11:47:02,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2818353.3333333335, ans=0.125
2023-11-24 11:47:18,216 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.45 vs. limit=15.0
2023-11-24 11:47:32,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2818553.3333333335, ans=0.125
2023-11-24 11:47:33,685 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 1950, loss[loss=0.0665, simple_loss=0.0861, pruned_loss=0.0128, audio_tagging_loss=0.01065, over 15458.00 frames. ], tot_loss[loss=0.06653, simple_loss=0.08944, pruned_loss=0.01272, audio_tagging_loss=0.009089, over 3046431.78 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 11:47:40,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2818553.3333333335, ans=0.125
2023-11-24 11:47:41,771 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.719e+01 9.311e+01 9.932e+01 1.598e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 11:47:51,885 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422800
2023-11-24 11:47:56,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2818620.0, ans=0.1
2023-11-24 11:47:56,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2818620.0, ans=0.125
2023-11-24 11:47:57,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2818686.6666666665, ans=0.1
2023-11-24 11:47:58,919 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.16 vs. limit=15.0
2023-11-24 11:48:14,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2818753.3333333335, ans=0.0
2023-11-24 11:48:16,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2818753.3333333335, ans=0.125
2023-11-24 11:48:19,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2818753.3333333335, ans=0.05
2023-11-24 11:48:20,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2818753.3333333335, ans=0.125
2023-11-24 11:48:28,103 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2818820.0, ans=0.0
2023-11-24 11:48:34,863 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2000, loss[loss=0.05955, simple_loss=0.07558, pruned_loss=0.01144, audio_tagging_loss=0.01032, over 15488.00 frames. ], tot_loss[loss=0.06636, simple_loss=0.08896, pruned_loss=0.01285, audio_tagging_loss=0.009036, over 3047799.87 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:48:54,375 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422850
2023-11-24 11:49:08,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2819020.0, ans=0.2
2023-11-24 11:49:12,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2819086.6666666665, ans=0.125
2023-11-24 11:49:16,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2819086.6666666665, ans=0.07
2023-11-24 11:49:25,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2819153.3333333335, ans=0.125
2023-11-24 11:49:33,514 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.75 vs. limit=12.0
2023-11-24 11:49:38,122 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2050, loss[loss=0.06608, simple_loss=0.0919, pruned_loss=0.01227, audio_tagging_loss=0.007865, over 15039.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09054, pruned_loss=0.01307, audio_tagging_loss=0.009002, over 3047463.38 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:49:42,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2819220.0, ans=0.125
2023-11-24 11:49:43,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2819220.0, ans=0.125
2023-11-24 11:49:46,320 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.671e+01 8.908e+01 9.390e+01 1.017e+02 1.320e+02, threshold=1.878e+02, percent-clipped=0.0
2023-11-24 11:49:51,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2819286.6666666665, ans=0.07
2023-11-24 11:49:56,495 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422900
2023-11-24 11:50:29,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2819486.6666666665, ans=0.0
2023-11-24 11:50:35,376 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2819486.6666666665, ans=0.0
2023-11-24 11:50:40,064 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2100, loss[loss=0.07931, simple_loss=0.09712, pruned_loss=0.01789, audio_tagging_loss=0.01286, over 14063.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.0903, pruned_loss=0.01307, audio_tagging_loss=0.008935, over 3037974.28 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:50:47,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2819553.3333333335, ans=0.125
2023-11-24 11:50:54,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2819620.0, ans=0.125
2023-11-24 11:50:58,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 422950
2023-11-24 11:51:02,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2819620.0, ans=0.125
2023-11-24 11:51:21,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2819753.3333333335, ans=0.125
2023-11-24 11:51:25,315 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2819753.3333333335, ans=0.2
2023-11-24 11:51:26,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2819753.3333333335, ans=0.125
2023-11-24 11:51:33,472 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.35 vs. limit=15.0
2023-11-24 11:51:42,200 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2150, loss[loss=0.06579, simple_loss=0.09075, pruned_loss=0.01147, audio_tagging_loss=0.008953, over 13889.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09015, pruned_loss=0.01301, audio_tagging_loss=0.008868, over 3032652.94 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:51:51,143 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.463e+01 9.382e+01 1.013e+02 1.297e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 11:51:51,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2819886.6666666665, ans=0.1
2023-11-24 11:51:56,978 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.09 vs. limit=15.0
2023-11-24 11:52:01,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423000
2023-11-24 11:52:09,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2820020.0, ans=0.0
2023-11-24 11:52:18,810 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:52:31,733 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.79 vs. limit=12.0
2023-11-24 11:52:32,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.25 vs. limit=15.0
2023-11-24 11:52:45,856 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2200, loss[loss=0.06251, simple_loss=0.08497, pruned_loss=0.009168, audio_tagging_loss=0.01085, over 15537.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.08988, pruned_loss=0.0129, audio_tagging_loss=0.0089, over 3042016.47 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:52:50,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2820220.0, ans=0.0
2023-11-24 11:53:03,690 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423050
2023-11-24 11:53:05,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2820286.6666666665, ans=0.125
2023-11-24 11:53:07,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2820286.6666666665, ans=0.125
2023-11-24 11:53:12,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2820353.3333333335, ans=0.125
2023-11-24 11:53:16,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2820353.3333333335, ans=0.0
2023-11-24 11:53:16,619 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.46 vs. limit=10.0
2023-11-24 11:53:42,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=2820486.6666666665, ans=10.0
2023-11-24 11:53:47,542 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2250, loss[loss=0.06676, simple_loss=0.09589, pruned_loss=0.01064, audio_tagging_loss=0.00818, over 15479.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09048, pruned_loss=0.0129, audio_tagging_loss=0.008879, over 3045621.66 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:53:50,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2820553.3333333335, ans=0.0
2023-11-24 11:53:55,796 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.635e+01 8.644e+01 9.254e+01 1.001e+02 1.291e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 11:54:02,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2820620.0, ans=0.0
2023-11-24 11:54:05,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2820620.0, ans=10.0
2023-11-24 11:54:06,252 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423100
2023-11-24 11:54:06,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2820620.0, ans=0.09899494936611666
2023-11-24 11:54:22,781 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.54 vs. limit=15.0
2023-11-24 11:54:49,830 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2300, loss[loss=0.04908, simple_loss=0.05701, pruned_loss=0.006515, audio_tagging_loss=0.01406, over 15669.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09058, pruned_loss=0.01284, audio_tagging_loss=0.008975, over 3045021.35 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:55:09,880 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423150
2023-11-24 11:55:26,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2821086.6666666665, ans=0.0
2023-11-24 11:55:45,407 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:55:52,930 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2350, loss[loss=0.07834, simple_loss=0.1092, pruned_loss=0.01618, audio_tagging_loss=0.007547, over 14900.00 frames. ], tot_loss[loss=0.06652, simple_loss=0.08963, pruned_loss=0.01267, audio_tagging_loss=0.009029, over 3037844.84 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:56:00,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2821220.0, ans=0.1
2023-11-24 11:56:01,695 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.333e+01 8.924e+01 9.646e+01 1.536e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-24 11:56:06,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2821286.6666666665, ans=0.2
2023-11-24 11:56:09,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2821286.6666666665, ans=0.0
2023-11-24 11:56:11,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423200
2023-11-24 11:56:32,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2821420.0, ans=0.125
2023-11-24 11:56:45,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2821486.6666666665, ans=0.0
2023-11-24 11:56:55,521 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2400, loss[loss=0.06563, simple_loss=0.09263, pruned_loss=0.01148, audio_tagging_loss=0.007828, over 15733.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08988, pruned_loss=0.01267, audio_tagging_loss=0.009052, over 3041782.49 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:56:55,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2821553.3333333335, ans=0.0
2023-11-24 11:57:05,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2821553.3333333335, ans=0.125
2023-11-24 11:57:10,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2821620.0, ans=0.0
2023-11-24 11:57:13,502 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423250
2023-11-24 11:57:23,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2821686.6666666665, ans=0.1
2023-11-24 11:57:27,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2821686.6666666665, ans=0.125
2023-11-24 11:57:32,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2821753.3333333335, ans=0.0
2023-11-24 11:57:57,320 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2450, loss[loss=0.06306, simple_loss=0.08385, pruned_loss=0.01246, audio_tagging_loss=0.008678, over 16132.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.08972, pruned_loss=0.0126, audio_tagging_loss=0.009095, over 3039476.63 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:58:06,725 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.416e+01 8.359e+01 8.901e+01 9.611e+01 1.267e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-24 11:58:16,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423300
2023-11-24 11:58:26,227 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:58:29,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2822020.0, ans=0.0
2023-11-24 11:58:43,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.70 vs. limit=15.0
2023-11-24 11:58:55,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2822153.3333333335, ans=0.2
2023-11-24 11:59:00,576 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2500, loss[loss=0.09267, simple_loss=0.125, pruned_loss=0.02188, audio_tagging_loss=0.008311, over 14965.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08975, pruned_loss=0.01264, audio_tagging_loss=0.009164, over 3041217.65 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:59:18,844 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423350
2023-11-24 11:59:18,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2822286.6666666665, ans=0.125
2023-11-24 11:59:20,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2822286.6666666665, ans=0.125
2023-11-24 11:59:20,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2822286.6666666665, ans=0.125
2023-11-24 11:59:24,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.80 vs. limit=22.5
2023-11-24 11:59:34,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=2822353.3333333335, ans=0.2
2023-11-24 11:59:41,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2822420.0, ans=0.125
2023-11-24 11:59:56,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2822486.6666666665, ans=0.0
2023-11-24 12:00:02,394 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2550, loss[loss=0.07767, simple_loss=0.09769, pruned_loss=0.01802, audio_tagging_loss=0.01081, over 15165.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08964, pruned_loss=0.01276, audio_tagging_loss=0.009084, over 3042572.03 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:00:08,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2822553.3333333335, ans=0.05
2023-11-24 12:00:10,606 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.497e+01 9.131e+01 9.713e+01 1.199e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 12:00:16,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2822620.0, ans=0.0
2023-11-24 12:00:20,159 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423400
2023-11-24 12:00:21,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2822620.0, ans=0.025
2023-11-24 12:00:25,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2822686.6666666665, ans=0.125
2023-11-24 12:00:31,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2822686.6666666665, ans=0.0
2023-11-24 12:00:32,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2822686.6666666665, ans=0.125
2023-11-24 12:00:53,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2822820.0, ans=0.125
2023-11-24 12:00:56,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2822820.0, ans=0.0
2023-11-24 12:01:04,225 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2600, loss[loss=0.07056, simple_loss=0.09033, pruned_loss=0.01677, audio_tagging_loss=0.008627, over 14697.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.08995, pruned_loss=0.01283, audio_tagging_loss=0.008962, over 3045299.03 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:01:11,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2822886.6666666665, ans=0.95
2023-11-24 12:01:23,836 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423450
2023-11-24 12:01:48,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2823086.6666666665, ans=0.125
2023-11-24 12:02:01,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2823153.3333333335, ans=0.125
2023-11-24 12:02:07,931 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2650, loss[loss=0.06542, simple_loss=0.07816, pruned_loss=0.01738, audio_tagging_loss=0.00896, over 15169.00 frames. ], tot_loss[loss=0.06652, simple_loss=0.08958, pruned_loss=0.01283, audio_tagging_loss=0.008902, over 3042742.85 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:02:14,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2823220.0, ans=0.0
2023-11-24 12:02:18,453 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.376e+01 8.976e+01 9.750e+01 1.206e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-24 12:02:23,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2823286.6666666665, ans=0.125
2023-11-24 12:02:23,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2823286.6666666665, ans=0.2
2023-11-24 12:02:27,110 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423500
2023-11-24 12:02:39,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2823353.3333333335, ans=0.07
2023-11-24 12:03:01,910 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2823486.6666666665, ans=0.0
2023-11-24 12:03:02,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2823486.6666666665, ans=0.1
2023-11-24 12:03:10,460 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2700, loss[loss=0.05791, simple_loss=0.07091, pruned_loss=0.008214, audio_tagging_loss=0.01424, over 14514.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.09001, pruned_loss=0.01278, audio_tagging_loss=0.00888, over 3038554.08 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:03:20,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2823553.3333333335, ans=0.125
2023-11-24 12:03:26,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2823620.0, ans=0.0
2023-11-24 12:03:28,260 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423550
2023-11-24 12:03:34,487 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2823686.6666666665, ans=0.125
2023-11-24 12:03:36,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2823686.6666666665, ans=0.0
2023-11-24 12:04:11,632 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2750, loss[loss=0.0481, simple_loss=0.06107, pruned_loss=0.005173, audio_tagging_loss=0.01239, over 14721.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09027, pruned_loss=0.01287, audio_tagging_loss=0.008864, over 3032409.97 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:04:20,992 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.328e+01 8.931e+01 9.724e+01 1.117e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 12:04:30,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423600
2023-11-24 12:05:00,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2824153.3333333335, ans=0.2
2023-11-24 12:05:03,794 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:05:05,754 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.96 vs. limit=15.0
2023-11-24 12:05:13,995 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2800, loss[loss=0.08798, simple_loss=0.1264, pruned_loss=0.01634, audio_tagging_loss=0.008422, over 15225.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09084, pruned_loss=0.01301, audio_tagging_loss=0.008718, over 3037731.45 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:05:33,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423650
2023-11-24 12:05:40,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2824353.3333333335, ans=0.1
2023-11-24 12:05:55,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2824420.0, ans=0.0
2023-11-24 12:06:08,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2824486.6666666665, ans=0.0
2023-11-24 12:06:17,161 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2850, loss[loss=0.08102, simple_loss=0.113, pruned_loss=0.01453, audio_tagging_loss=0.009979, over 14823.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09156, pruned_loss=0.013, audio_tagging_loss=0.008633, over 3042372.70 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:06:19,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2824553.3333333335, ans=0.0
2023-11-24 12:06:26,591 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.601e+01 9.263e+01 1.001e+02 1.378e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 12:06:35,065 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423700
2023-11-24 12:06:59,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2824753.3333333335, ans=0.1
2023-11-24 12:07:05,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.05 vs. limit=12.0
2023-11-24 12:07:15,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2824820.0, ans=0.125
2023-11-24 12:07:18,748 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2900, loss[loss=0.08113, simple_loss=0.1109, pruned_loss=0.01833, audio_tagging_loss=0.007358, over 16549.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09236, pruned_loss=0.01328, audio_tagging_loss=0.008588, over 3040604.62 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:07:21,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2824886.6666666665, ans=0.1
2023-11-24 12:07:37,115 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423750
2023-11-24 12:07:45,155 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2825020.0, ans=0.025
2023-11-24 12:07:46,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2825020.0, ans=0.0
2023-11-24 12:08:00,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2825086.6666666665, ans=0.125
2023-11-24 12:08:06,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2825086.6666666665, ans=0.05
2023-11-24 12:08:21,109 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 2950, loss[loss=0.06485, simple_loss=0.08646, pruned_loss=0.01142, audio_tagging_loss=0.0102, over 14901.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09181, pruned_loss=0.01336, audio_tagging_loss=0.008713, over 3036492.21 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:08:31,478 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.518e+01 9.259e+01 1.017e+02 1.246e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 12:08:41,108 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423800
2023-11-24 12:09:05,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0
2023-11-24 12:09:08,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2825420.0, ans=0.1
2023-11-24 12:09:24,612 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3000, loss[loss=0.08133, simple_loss=0.09195, pruned_loss=0.02099, audio_tagging_loss=0.01436, over 15979.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09239, pruned_loss=0.01351, audio_tagging_loss=0.008776, over 3040735.86 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:09:24,613 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 12:10:06,945 INFO [train_asr.py:1253] (2/4) Epoch 36, validation: loss=0.05726, simple_loss=0.05083, pruned_loss=0.005098, audio_tagging_loss=0.02675, over 4681554.00 frames. 
2023-11-24 12:10:06,946 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 12:10:26,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423850
2023-11-24 12:10:45,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2825753.3333333335, ans=0.125
2023-11-24 12:10:52,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2825753.3333333335, ans=0.0
2023-11-24 12:10:53,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff3.min_abs, batch_count=2825753.3333333335, ans=0.2
2023-11-24 12:10:56,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2825820.0, ans=0.0
2023-11-24 12:11:05,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2825820.0, ans=0.0
2023-11-24 12:11:10,537 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3050, loss[loss=0.06068, simple_loss=0.08762, pruned_loss=0.009236, audio_tagging_loss=0.007637, over 14293.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09284, pruned_loss=0.01343, audio_tagging_loss=0.008767, over 3042449.75 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:11:20,549 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.127e+01 8.750e+01 9.306e+01 1.016e+02 1.344e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 12:11:28,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2825953.3333333335, ans=0.07
2023-11-24 12:11:29,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423900
2023-11-24 12:11:31,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2825953.3333333335, ans=0.1
2023-11-24 12:11:46,846 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:11:47,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.51 vs. limit=15.0
2023-11-24 12:11:48,744 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.67 vs. limit=6.0
2023-11-24 12:12:07,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2826153.3333333335, ans=0.1
2023-11-24 12:12:13,624 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3100, loss[loss=0.07443, simple_loss=0.1046, pruned_loss=0.01363, audio_tagging_loss=0.008494, over 15268.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09309, pruned_loss=0.01346, audio_tagging_loss=0.008841, over 3042892.60 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:12:32,254 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 423950
2023-11-24 12:12:50,692 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2826420.0, ans=0.09899494936611666
2023-11-24 12:13:08,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2826486.6666666665, ans=0.0
2023-11-24 12:13:16,553 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3150, loss[loss=0.08504, simple_loss=0.1194, pruned_loss=0.0207, audio_tagging_loss=0.004641, over 14301.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09248, pruned_loss=0.01343, audio_tagging_loss=0.00897, over 3035526.58 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:13:25,948 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.600e+01 9.080e+01 9.861e+01 1.236e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 12:13:34,977 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424000
2023-11-24 12:14:15,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2826820.0, ans=0.2
2023-11-24 12:14:15,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2826820.0, ans=0.05
2023-11-24 12:14:21,340 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3200, loss[loss=0.08058, simple_loss=0.09615, pruned_loss=0.02149, audio_tagging_loss=0.01101, over 14721.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09262, pruned_loss=0.01353, audio_tagging_loss=0.009068, over 3038310.72 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:14:31,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2826886.6666666665, ans=0.1
2023-11-24 12:14:38,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2826953.3333333335, ans=0.1
2023-11-24 12:14:41,531 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424050
2023-11-24 12:14:54,262 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2827020.0, ans=0.0
2023-11-24 12:14:55,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2827020.0, ans=0.125
2023-11-24 12:14:55,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2827020.0, ans=0.1
2023-11-24 12:14:56,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2827020.0, ans=0.125
2023-11-24 12:14:59,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2827086.6666666665, ans=0.125
2023-11-24 12:15:22,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2827153.3333333335, ans=0.125
2023-11-24 12:15:25,381 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3250, loss[loss=0.03566, simple_loss=0.03799, pruned_loss=0.004371, audio_tagging_loss=0.0123, over 13649.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09153, pruned_loss=0.01335, audio_tagging_loss=0.009195, over 3037101.07 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:15:25,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2827220.0, ans=0.0
2023-11-24 12:15:34,951 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.544e+01 9.313e+01 9.948e+01 1.268e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 12:15:43,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424100
2023-11-24 12:15:47,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2827286.6666666665, ans=0.125
2023-11-24 12:15:53,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2827353.3333333335, ans=0.0
2023-11-24 12:16:11,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2827420.0, ans=0.125
2023-11-24 12:16:26,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2827553.3333333335, ans=0.0
2023-11-24 12:16:27,556 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3300, loss[loss=0.07221, simple_loss=0.1053, pruned_loss=0.01068, audio_tagging_loss=0.00886, over 15010.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09157, pruned_loss=0.01326, audio_tagging_loss=0.00918, over 3039949.21 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:16:29,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2827553.3333333335, ans=0.125
2023-11-24 12:16:36,589 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.69 vs. limit=15.0
2023-11-24 12:16:45,701 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424150
2023-11-24 12:16:47,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2827620.0, ans=0.125
2023-11-24 12:17:11,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2827753.3333333335, ans=0.125
2023-11-24 12:17:26,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2827820.0, ans=0.2
2023-11-24 12:17:28,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.86 vs. limit=22.5
2023-11-24 12:17:30,008 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3350, loss[loss=0.07678, simple_loss=0.1021, pruned_loss=0.01837, audio_tagging_loss=0.007353, over 15077.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09205, pruned_loss=0.01323, audio_tagging_loss=0.009018, over 3041807.37 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:17:31,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2827886.6666666665, ans=0.0
2023-11-24 12:17:40,614 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.571e+01 9.014e+01 9.710e+01 1.291e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 12:17:47,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2827953.3333333335, ans=0.125
2023-11-24 12:17:49,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424200
2023-11-24 12:18:22,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2828153.3333333335, ans=0.125
2023-11-24 12:18:23,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2828153.3333333335, ans=0.0
2023-11-24 12:18:33,250 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3400, loss[loss=0.06651, simple_loss=0.08486, pruned_loss=0.01384, audio_tagging_loss=0.01024, over 15490.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.0912, pruned_loss=0.01306, audio_tagging_loss=0.008976, over 3049369.00 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:18:33,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2828220.0, ans=0.1
2023-11-24 12:18:38,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2828220.0, ans=0.125
2023-11-24 12:18:40,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2828220.0, ans=0.125
2023-11-24 12:18:49,630 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2828286.6666666665, ans=0.125
2023-11-24 12:18:51,910 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424250
2023-11-24 12:18:58,899 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0
2023-11-24 12:19:04,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2828353.3333333335, ans=0.0
2023-11-24 12:19:32,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.86 vs. limit=15.0
2023-11-24 12:19:33,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0
2023-11-24 12:19:35,825 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3450, loss[loss=0.08891, simple_loss=0.1242, pruned_loss=0.01971, audio_tagging_loss=0.007096, over 16349.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09164, pruned_loss=0.01325, audio_tagging_loss=0.00882, over 3047160.21 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:19:47,009 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.469e+01 9.145e+01 9.844e+01 1.227e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 12:19:49,152 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.33 vs. limit=5.0
2023-11-24 12:19:53,625 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.92 vs. limit=12.0
2023-11-24 12:19:54,186 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424300
2023-11-24 12:20:12,856 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2828753.3333333335, ans=0.125
2023-11-24 12:20:38,412 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3500, loss[loss=0.07443, simple_loss=0.09968, pruned_loss=0.01618, audio_tagging_loss=0.008411, over 15386.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09147, pruned_loss=0.0132, audio_tagging_loss=0.008746, over 3046825.87 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:20:41,080 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:20:58,012 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424350
2023-11-24 12:21:00,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2828953.3333333335, ans=0.2
2023-11-24 12:21:11,071 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:21:19,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2829086.6666666665, ans=0.125
2023-11-24 12:21:22,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2829086.6666666665, ans=0.95
2023-11-24 12:21:25,484 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.41 vs. limit=10.0
2023-11-24 12:21:29,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2829153.3333333335, ans=0.2
2023-11-24 12:21:31,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2829153.3333333335, ans=0.125
2023-11-24 12:21:31,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.18 vs. limit=15.0
2023-11-24 12:21:35,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2829153.3333333335, ans=0.125
2023-11-24 12:21:41,494 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3550, loss[loss=0.05343, simple_loss=0.06695, pruned_loss=0.01049, audio_tagging_loss=0.009462, over 14685.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09078, pruned_loss=0.01308, audio_tagging_loss=0.008676, over 3047500.33 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:21:49,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2829220.0, ans=0.025
2023-11-24 12:21:52,757 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 8.449e+01 8.981e+01 9.607e+01 1.173e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 12:21:58,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2829286.6666666665, ans=0.015
2023-11-24 12:22:00,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424400
2023-11-24 12:22:03,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.59 vs. limit=10.0
2023-11-24 12:22:06,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2829353.3333333335, ans=0.125
2023-11-24 12:22:35,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2829486.6666666665, ans=0.2
2023-11-24 12:22:36,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2829486.6666666665, ans=0.2
2023-11-24 12:22:44,315 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3600, loss[loss=0.0603, simple_loss=0.07936, pruned_loss=0.01277, audio_tagging_loss=0.007848, over 14716.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.09001, pruned_loss=0.01296, audio_tagging_loss=0.00875, over 3047415.78 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:23:00,961 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.75 vs. limit=15.0
2023-11-24 12:23:03,167 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424450
2023-11-24 12:23:04,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2829620.0, ans=0.0
2023-11-24 12:23:12,090 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.77 vs. limit=15.0
2023-11-24 12:23:13,434 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.56 vs. limit=15.0
2023-11-24 12:23:20,482 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.06 vs. limit=22.5
2023-11-24 12:23:34,218 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.30 vs. limit=12.0
2023-11-24 12:23:46,404 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3650, loss[loss=0.07533, simple_loss=0.09461, pruned_loss=0.01579, audio_tagging_loss=0.01223, over 15138.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09065, pruned_loss=0.01318, audio_tagging_loss=0.008814, over 3054488.57 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:23:49,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2829886.6666666665, ans=0.0
2023-11-24 12:23:59,993 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.262e+01 8.304e+01 8.721e+01 9.501e+01 1.428e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-24 12:24:05,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2829953.3333333335, ans=0.125
2023-11-24 12:24:06,038 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424500
2023-11-24 12:24:12,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2830020.0, ans=0.125
2023-11-24 12:24:17,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2830020.0, ans=0.5
2023-11-24 12:24:18,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2830020.0, ans=0.125
2023-11-24 12:24:20,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.17 vs. limit=15.0
2023-11-24 12:24:23,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2830086.6666666665, ans=0.2
2023-11-24 12:24:23,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2830086.6666666665, ans=0.125
2023-11-24 12:24:26,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2830086.6666666665, ans=0.125
2023-11-24 12:24:30,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2830086.6666666665, ans=0.0
2023-11-24 12:24:46,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2830153.3333333335, ans=0.0
2023-11-24 12:24:47,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2830153.3333333335, ans=0.2
2023-11-24 12:24:49,724 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3700, loss[loss=0.06941, simple_loss=0.09444, pruned_loss=0.01125, audio_tagging_loss=0.01094, over 14580.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09163, pruned_loss=0.01336, audio_tagging_loss=0.008752, over 3059463.35 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:24:57,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2830220.0, ans=0.125
2023-11-24 12:25:05,259 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.59 vs. limit=22.5
2023-11-24 12:25:08,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424550
2023-11-24 12:25:09,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2830286.6666666665, ans=0.125
2023-11-24 12:25:35,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2830420.0, ans=0.125
2023-11-24 12:25:37,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2830420.0, ans=0.0
2023-11-24 12:25:51,803 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3750, loss[loss=0.07295, simple_loss=0.09697, pruned_loss=0.0136, audio_tagging_loss=0.01086, over 15528.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09112, pruned_loss=0.01325, audio_tagging_loss=0.008807, over 3057064.92 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:25:58,377 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.49 vs. limit=15.0
2023-11-24 12:26:03,766 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.375e+01 8.897e+01 9.293e+01 9.941e+01 1.332e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 12:26:09,863 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424600
2023-11-24 12:26:10,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2830620.0, ans=0.1
2023-11-24 12:26:18,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2830686.6666666665, ans=0.1
2023-11-24 12:26:26,323 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.83 vs. limit=15.0
2023-11-24 12:26:35,386 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:26:42,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2830820.0, ans=0.2
2023-11-24 12:26:53,216 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3800, loss[loss=0.06943, simple_loss=0.09487, pruned_loss=0.01114, audio_tagging_loss=0.01086, over 15597.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09169, pruned_loss=0.01339, audio_tagging_loss=0.008837, over 3058217.96 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:27:12,739 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424650
2023-11-24 12:27:12,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2830953.3333333335, ans=0.09899494936611666
2023-11-24 12:27:28,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2831020.0, ans=0.125
2023-11-24 12:27:29,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2831020.0, ans=0.125
2023-11-24 12:27:31,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2831086.6666666665, ans=0.1
2023-11-24 12:27:47,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2831153.3333333335, ans=0.025
2023-11-24 12:27:56,864 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3850, loss[loss=0.06816, simple_loss=0.09004, pruned_loss=0.01381, audio_tagging_loss=0.009336, over 14618.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.0921, pruned_loss=0.01356, audio_tagging_loss=0.00885, over 3057938.11 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:28:00,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2831220.0, ans=0.0
2023-11-24 12:28:05,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2831220.0, ans=0.1
2023-11-24 12:28:08,605 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.03 vs. limit=15.0
2023-11-24 12:28:09,276 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.465e+01 9.054e+01 9.529e+01 1.182e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 12:28:10,986 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.92 vs. limit=15.0
2023-11-24 12:28:15,417 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424700
2023-11-24 12:28:20,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2831353.3333333335, ans=0.125
2023-11-24 12:28:21,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2831353.3333333335, ans=0.125
2023-11-24 12:28:25,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2831353.3333333335, ans=0.125
2023-11-24 12:28:25,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2831353.3333333335, ans=0.125
2023-11-24 12:28:39,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2831420.0, ans=0.125
2023-11-24 12:28:58,698 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3900, loss[loss=0.05692, simple_loss=0.07431, pruned_loss=0.01141, audio_tagging_loss=0.00835, over 14131.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09167, pruned_loss=0.01352, audio_tagging_loss=0.008863, over 3041709.12 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:29:04,870 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2831553.3333333335, ans=0.125
2023-11-24 12:29:13,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2831620.0, ans=0.0
2023-11-24 12:29:16,395 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424750
2023-11-24 12:29:26,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2831686.6666666665, ans=0.09899494936611666
2023-11-24 12:29:49,552 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=12.0
2023-11-24 12:29:54,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2831820.0, ans=0.125
2023-11-24 12:29:57,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2831820.0, ans=0.09899494936611666
2023-11-24 12:30:00,558 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 3950, loss[loss=0.06458, simple_loss=0.0682, pruned_loss=0.01669, audio_tagging_loss=0.01379, over 14411.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09091, pruned_loss=0.01336, audio_tagging_loss=0.00899, over 3039803.70 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:30:00,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2831886.6666666665, ans=0.125
2023-11-24 12:30:14,457 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.446e+01 8.514e+01 9.052e+01 9.990e+01 1.654e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 12:30:19,983 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424800
2023-11-24 12:30:25,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2832020.0, ans=0.015
2023-11-24 12:30:52,118 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-24 12:31:01,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2832153.3333333335, ans=0.0
2023-11-24 12:31:03,781 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4000, loss[loss=0.06033, simple_loss=0.08145, pruned_loss=0.009484, audio_tagging_loss=0.01012, over 15383.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09061, pruned_loss=0.01321, audio_tagging_loss=0.009022, over 3045750.38 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:31:23,637 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424850
2023-11-24 12:31:24,234 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.01 vs. limit=15.0
2023-11-24 12:31:35,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2832353.3333333335, ans=0.2
2023-11-24 12:31:39,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2832353.3333333335, ans=0.125
2023-11-24 12:31:39,522 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0
2023-11-24 12:31:41,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2832420.0, ans=0.125
2023-11-24 12:31:51,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2832420.0, ans=0.0
2023-11-24 12:31:59,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2832486.6666666665, ans=0.125
2023-11-24 12:32:07,617 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4050, loss[loss=0.08175, simple_loss=0.1115, pruned_loss=0.01885, audio_tagging_loss=0.00717, over 15222.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09105, pruned_loss=0.01318, audio_tagging_loss=0.009082, over 3044760.61 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:32:11,198 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:32:12,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2832553.3333333335, ans=0.125
2023-11-24 12:32:21,751 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.652e+01 8.871e+01 9.409e+01 1.005e+02 1.323e+02, threshold=1.882e+02, percent-clipped=0.0
2023-11-24 12:32:23,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2832620.0, ans=0.1
2023-11-24 12:32:24,561 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2832620.0, ans=0.07
2023-11-24 12:32:25,521 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424900
2023-11-24 12:32:27,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2832620.0, ans=0.125
2023-11-24 12:32:35,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2832686.6666666665, ans=0.125
2023-11-24 12:33:04,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2832820.0, ans=0.0
2023-11-24 12:33:09,315 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4100, loss[loss=0.06951, simple_loss=0.09063, pruned_loss=0.01342, audio_tagging_loss=0.01077, over 15417.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09029, pruned_loss=0.0131, audio_tagging_loss=0.009198, over 3041662.78 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:33:16,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2832886.6666666665, ans=0.1
2023-11-24 12:33:17,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2832886.6666666665, ans=0.2
2023-11-24 12:33:26,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2832953.3333333335, ans=0.0
2023-11-24 12:33:28,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 424950
2023-11-24 12:33:46,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2833086.6666666665, ans=0.125
2023-11-24 12:33:48,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=2833086.6666666665, ans=0.1
2023-11-24 12:33:54,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2833086.6666666665, ans=0.2
2023-11-24 12:33:59,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.42 vs. limit=22.5
2023-11-24 12:34:12,402 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4150, loss[loss=0.06141, simple_loss=0.08181, pruned_loss=0.01168, audio_tagging_loss=0.008823, over 14790.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09109, pruned_loss=0.01318, audio_tagging_loss=0.009035, over 3045993.46 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:34:14,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2833220.0, ans=0.1
2023-11-24 12:34:27,227 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.532e+01 9.065e+01 9.755e+01 1.245e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 12:34:30,565 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2833286.6666666665, ans=0.125
2023-11-24 12:34:31,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425000
2023-11-24 12:34:57,038 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:34:57,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2833420.0, ans=0.125
2023-11-24 12:35:14,959 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4200, loss[loss=0.09015, simple_loss=0.1302, pruned_loss=0.01958, audio_tagging_loss=0.005466, over 16503.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09228, pruned_loss=0.01337, audio_tagging_loss=0.008838, over 3043848.47 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:35:26,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2833553.3333333335, ans=0.125
2023-11-24 12:35:28,594 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:35:34,468 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425050
2023-11-24 12:35:35,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2833620.0, ans=0.125
2023-11-24 12:36:04,857 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.28 vs. limit=15.0
2023-11-24 12:36:18,763 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4250, loss[loss=0.03982, simple_loss=0.04563, pruned_loss=0.006352, audio_tagging_loss=0.01065, over 14768.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.0922, pruned_loss=0.01335, audio_tagging_loss=0.008803, over 3050380.51 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:36:31,958 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2833953.3333333335, ans=0.125
2023-11-24 12:36:32,903 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.335e+01 8.749e+01 9.313e+01 9.960e+01 2.008e+02, threshold=1.863e+02, percent-clipped=1.0
2023-11-24 12:36:37,219 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425100
2023-11-24 12:36:45,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2834020.0, ans=0.2
2023-11-24 12:36:59,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2834086.6666666665, ans=0.0
2023-11-24 12:37:08,127 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:37:20,822 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4300, loss[loss=0.06919, simple_loss=0.09901, pruned_loss=0.01255, audio_tagging_loss=0.007138, over 14878.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09269, pruned_loss=0.01342, audio_tagging_loss=0.008735, over 3052707.87 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:37:35,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2834286.6666666665, ans=0.125
2023-11-24 12:37:40,022 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425150
2023-11-24 12:37:58,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2834420.0, ans=0.0
2023-11-24 12:38:00,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2834420.0, ans=0.125
2023-11-24 12:38:04,413 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0
2023-11-24 12:38:08,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2834420.0, ans=0.125
2023-11-24 12:38:13,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2834486.6666666665, ans=0.07
2023-11-24 12:38:24,199 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4350, loss[loss=0.09043, simple_loss=0.121, pruned_loss=0.02149, audio_tagging_loss=0.008445, over 15439.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09326, pruned_loss=0.01341, audio_tagging_loss=0.008671, over 3049867.13 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:38:24,909 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.12 vs. limit=15.0
2023-11-24 12:38:35,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.53 vs. limit=22.5
2023-11-24 12:38:39,491 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.655e+01 9.322e+01 1.008e+02 1.169e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 12:38:43,135 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425200
2023-11-24 12:38:58,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2834686.6666666665, ans=0.0
2023-11-24 12:39:01,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.32 vs. limit=15.0
2023-11-24 12:39:10,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2834753.3333333335, ans=0.125
2023-11-24 12:39:21,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2834820.0, ans=0.125
2023-11-24 12:39:21,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2834820.0, ans=0.125
2023-11-24 12:39:27,402 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4400, loss[loss=0.05537, simple_loss=0.07045, pruned_loss=0.009865, audio_tagging_loss=0.01028, over 15507.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09245, pruned_loss=0.01333, audio_tagging_loss=0.008697, over 3048737.47 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:39:30,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.15 vs. limit=15.0
2023-11-24 12:39:45,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425250
2023-11-24 12:40:09,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2835086.6666666665, ans=0.0
2023-11-24 12:40:14,125 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.37 vs. limit=12.0
2023-11-24 12:40:29,272 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4450, loss[loss=0.05244, simple_loss=0.06135, pruned_loss=0.008756, audio_tagging_loss=0.01301, over 14345.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09333, pruned_loss=0.0135, audio_tagging_loss=0.008712, over 3050915.60 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:40:36,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2835220.0, ans=0.2
2023-11-24 12:40:37,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2835220.0, ans=0.125
2023-11-24 12:40:44,625 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.474e+01 8.586e+01 9.357e+01 9.969e+01 1.625e+02, threshold=1.871e+02, percent-clipped=0.0
2023-11-24 12:40:48,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425300
2023-11-24 12:41:03,902 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.51 vs. limit=15.0
2023-11-24 12:41:12,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2835420.0, ans=0.09899494936611666
2023-11-24 12:41:12,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2835420.0, ans=0.2
2023-11-24 12:41:32,477 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4500, loss[loss=0.05962, simple_loss=0.07979, pruned_loss=0.01077, audio_tagging_loss=0.00895, over 13750.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09316, pruned_loss=0.01342, audio_tagging_loss=0.008697, over 3047065.33 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:41:35,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.15 vs. limit=15.0
2023-11-24 12:41:51,053 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425350
2023-11-24 12:42:21,264 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.70 vs. limit=12.0
2023-11-24 12:42:28,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2835820.0, ans=0.125
2023-11-24 12:42:31,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2835820.0, ans=0.125
2023-11-24 12:42:35,578 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4550, loss[loss=0.06752, simple_loss=0.08512, pruned_loss=0.01369, audio_tagging_loss=0.01127, over 16265.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09219, pruned_loss=0.01317, audio_tagging_loss=0.008783, over 3050222.26 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:42:48,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2835953.3333333335, ans=0.0
2023-11-24 12:42:50,418 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.548e+01 8.325e+01 9.085e+01 9.707e+01 1.236e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 12:42:50,735 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2835953.3333333335, ans=0.125
2023-11-24 12:42:51,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2835953.3333333335, ans=0.125
2023-11-24 12:42:54,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425400
2023-11-24 12:42:55,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2835953.3333333335, ans=0.125
2023-11-24 12:43:10,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2836020.0, ans=0.125
2023-11-24 12:43:23,429 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:43:26,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2836153.3333333335, ans=0.0
2023-11-24 12:43:34,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2836153.3333333335, ans=0.0
2023-11-24 12:43:38,253 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4600, loss[loss=0.06465, simple_loss=0.08426, pruned_loss=0.01534, audio_tagging_loss=0.00718, over 14516.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09165, pruned_loss=0.01316, audio_tagging_loss=0.008838, over 3050311.29 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:43:45,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2836220.0, ans=0.125
2023-11-24 12:43:51,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2836286.6666666665, ans=0.0
2023-11-24 12:43:57,353 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425450
2023-11-24 12:44:17,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2836420.0, ans=0.0
2023-11-24 12:44:32,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2836486.6666666665, ans=0.125
2023-11-24 12:44:40,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2836553.3333333335, ans=0.125
2023-11-24 12:44:41,251 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4650, loss[loss=0.07088, simple_loss=0.08713, pruned_loss=0.01558, audio_tagging_loss=0.01173, over 14596.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09191, pruned_loss=0.01334, audio_tagging_loss=0.008962, over 3052843.36 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:44:55,931 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.381e+01 8.503e+01 9.255e+01 1.001e+02 1.285e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 12:44:59,684 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425500
2023-11-24 12:44:59,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2836620.0, ans=0.125
2023-11-24 12:45:14,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2836686.6666666665, ans=0.2
2023-11-24 12:45:18,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2836753.3333333335, ans=0.1
2023-11-24 12:45:36,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2836820.0, ans=0.125
2023-11-24 12:45:43,870 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4700, loss[loss=0.07028, simple_loss=0.09658, pruned_loss=0.01297, audio_tagging_loss=0.009024, over 15344.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.0922, pruned_loss=0.01328, audio_tagging_loss=0.008987, over 3041555.83 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:45:45,286 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:46:02,026 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425550
2023-11-24 12:46:12,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2837020.0, ans=0.125
2023-11-24 12:46:15,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2837020.0, ans=0.2
2023-11-24 12:46:22,054 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.42 vs. limit=15.0
2023-11-24 12:46:26,246 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0
2023-11-24 12:46:31,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2837086.6666666665, ans=0.0
2023-11-24 12:46:32,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2837153.3333333335, ans=0.125
2023-11-24 12:46:44,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2837220.0, ans=0.95
2023-11-24 12:46:45,791 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4750, loss[loss=0.05849, simple_loss=0.08429, pruned_loss=0.00796, audio_tagging_loss=0.008385, over 14387.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09207, pruned_loss=0.01323, audio_tagging_loss=0.009077, over 3048641.91 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:46:48,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.38 vs. limit=12.0
2023-11-24 12:46:54,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2837220.0, ans=0.125
2023-11-24 12:47:01,273 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.647e+01 9.446e+01 1.032e+02 1.298e+02, threshold=1.889e+02, percent-clipped=0.0
2023-11-24 12:47:05,667 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425600
2023-11-24 12:47:19,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.43 vs. limit=15.0
2023-11-24 12:47:40,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2837486.6666666665, ans=0.125
2023-11-24 12:47:49,891 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4800, loss[loss=0.07418, simple_loss=0.1078, pruned_loss=0.01345, audio_tagging_loss=0.006832, over 15989.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09146, pruned_loss=0.01319, audio_tagging_loss=0.009119, over 3055675.82 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:48:06,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2837620.0, ans=0.2
2023-11-24 12:48:08,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2837620.0, ans=0.0
2023-11-24 12:48:09,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425650
2023-11-24 12:48:49,671 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.24 vs. limit=15.0
2023-11-24 12:48:54,409 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4850, loss[loss=0.05004, simple_loss=0.06068, pruned_loss=0.007253, audio_tagging_loss=0.01244, over 15791.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.0914, pruned_loss=0.01317, audio_tagging_loss=0.00916, over 3054519.15 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:49:00,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2837886.6666666665, ans=0.125
2023-11-24 12:49:03,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2837886.6666666665, ans=0.2
2023-11-24 12:49:04,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2837886.6666666665, ans=0.0
2023-11-24 12:49:08,623 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.265e+01 8.570e+01 9.281e+01 9.821e+01 1.175e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 12:49:12,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425700
2023-11-24 12:49:25,265 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2838020.0, ans=0.125
2023-11-24 12:49:34,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2838086.6666666665, ans=0.0
2023-11-24 12:49:43,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.54 vs. limit=6.0
2023-11-24 12:49:55,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2838220.0, ans=0.0
2023-11-24 12:49:56,097 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4900, loss[loss=0.05578, simple_loss=0.07, pruned_loss=0.008842, audio_tagging_loss=0.01194, over 14167.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09125, pruned_loss=0.01313, audio_tagging_loss=0.009173, over 3048783.13 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:50:14,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2838286.6666666665, ans=0.125
2023-11-24 12:50:14,165 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2838286.6666666665, ans=0.05
2023-11-24 12:50:15,787 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425750
2023-11-24 12:50:21,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2838353.3333333335, ans=0.125
2023-11-24 12:50:35,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2838420.0, ans=0.07
2023-11-24 12:50:37,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2838420.0, ans=0.125
2023-11-24 12:50:39,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2838420.0, ans=0.125
2023-11-24 12:50:57,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2838553.3333333335, ans=0.0
2023-11-24 12:50:58,323 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 4950, loss[loss=0.06983, simple_loss=0.1007, pruned_loss=0.01211, audio_tagging_loss=0.007385, over 14819.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09173, pruned_loss=0.01305, audio_tagging_loss=0.008976, over 3045901.64 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:51:14,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 8.587e+01 9.264e+01 9.833e+01 1.255e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 12:51:17,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425800
2023-11-24 12:51:42,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2838753.3333333335, ans=0.0
2023-11-24 12:52:02,005 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5000, loss[loss=0.06939, simple_loss=0.1013, pruned_loss=0.01147, audio_tagging_loss=0.007264, over 16553.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.0912, pruned_loss=0.01299, audio_tagging_loss=0.00896, over 3047108.62 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:52:16,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2838953.3333333335, ans=0.125
2023-11-24 12:52:19,881 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425850
2023-11-24 12:52:22,942 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.99 vs. limit=12.0
2023-11-24 12:52:49,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2839086.6666666665, ans=0.035
2023-11-24 12:52:49,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.47 vs. limit=12.0
2023-11-24 12:52:53,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.16 vs. limit=15.0
2023-11-24 12:53:03,496 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5050, loss[loss=0.04523, simple_loss=0.05368, pruned_loss=0.005955, audio_tagging_loss=0.01244, over 14859.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.09018, pruned_loss=0.0128, audio_tagging_loss=0.008814, over 3042400.44 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:53:17,861 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.090e+01 8.266e+01 8.922e+01 9.681e+01 1.367e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 12:53:22,170 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425900
2023-11-24 12:53:25,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=2839286.6666666665, ans=0.02
2023-11-24 12:53:32,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2839353.3333333335, ans=0.125
2023-11-24 12:54:06,723 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5100, loss[loss=0.07214, simple_loss=0.09721, pruned_loss=0.01264, audio_tagging_loss=0.0109, over 15239.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.08946, pruned_loss=0.01296, audio_tagging_loss=0.008868, over 3036792.92 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:54:12,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2839553.3333333335, ans=0.1
2023-11-24 12:54:26,916 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 425950
2023-11-24 12:54:51,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2839753.3333333335, ans=0.0
2023-11-24 12:54:54,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.35 vs. limit=15.0
2023-11-24 12:55:10,655 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.60 vs. limit=22.5
2023-11-24 12:55:11,160 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5150, loss[loss=0.07428, simple_loss=0.1028, pruned_loss=0.01526, audio_tagging_loss=0.007635, over 15631.00 frames. ], tot_loss[loss=0.0659, simple_loss=0.08852, pruned_loss=0.01274, audio_tagging_loss=0.008899, over 3027070.24 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:55:26,159 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.369e+01 9.000e+01 9.837e+01 1.217e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 12:55:29,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426000
2023-11-24 12:55:29,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2839953.3333333335, ans=0.0
2023-11-24 12:55:50,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2840086.6666666665, ans=0.2
2023-11-24 12:56:03,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2840153.3333333335, ans=0.125
2023-11-24 12:56:07,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2840153.3333333335, ans=0.0
2023-11-24 12:56:14,385 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5200, loss[loss=0.06899, simple_loss=0.09328, pruned_loss=0.01318, audio_tagging_loss=0.009177, over 14109.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.09039, pruned_loss=0.01303, audio_tagging_loss=0.008726, over 3031045.82 frames. ], batch size: 52, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:56:20,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2840220.0, ans=0.125
2023-11-24 12:56:26,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2840286.6666666665, ans=0.125
2023-11-24 12:56:26,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2840286.6666666665, ans=0.125
2023-11-24 12:56:29,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2840286.6666666665, ans=0.1
2023-11-24 12:56:32,691 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426050
2023-11-24 12:56:50,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2840420.0, ans=0.125
2023-11-24 12:56:54,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2840420.0, ans=0.0
2023-11-24 12:56:56,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2840420.0, ans=0.2
2023-11-24 12:56:58,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2840420.0, ans=0.1
2023-11-24 12:56:58,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2840420.0, ans=0.125
2023-11-24 12:57:15,466 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5250, loss[loss=0.06713, simple_loss=0.08273, pruned_loss=0.01513, audio_tagging_loss=0.01064, over 15090.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.09039, pruned_loss=0.01298, audio_tagging_loss=0.008683, over 3042673.95 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:57:19,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2840553.3333333335, ans=0.125
2023-11-24 12:57:28,356 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.43 vs. limit=15.0
2023-11-24 12:57:32,384 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.236e+01 8.528e+01 9.139e+01 1.004e+02 1.210e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 12:57:35,507 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426100
2023-11-24 12:58:00,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2840753.3333333335, ans=0.2
2023-11-24 12:58:15,827 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2840820.0, ans=0.0
2023-11-24 12:58:19,033 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5300, loss[loss=0.06908, simple_loss=0.09402, pruned_loss=0.01438, audio_tagging_loss=0.007688, over 14994.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09123, pruned_loss=0.01317, audio_tagging_loss=0.008707, over 3044838.77 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:58:28,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2840886.6666666665, ans=0.125
2023-11-24 12:58:33,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2840953.3333333335, ans=0.125
2023-11-24 12:58:38,030 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426150
2023-11-24 12:58:39,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2840953.3333333335, ans=0.125
2023-11-24 12:58:47,034 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.99 vs. limit=15.0
2023-11-24 12:58:56,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2841086.6666666665, ans=0.0
2023-11-24 12:58:58,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2841086.6666666665, ans=0.125
2023-11-24 12:58:59,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2841086.6666666665, ans=0.5
2023-11-24 12:59:05,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2841086.6666666665, ans=0.0
2023-11-24 12:59:15,000 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.03 vs. limit=15.0
2023-11-24 12:59:22,039 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5350, loss[loss=0.07129, simple_loss=0.09236, pruned_loss=0.01541, audio_tagging_loss=0.009693, over 14785.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09161, pruned_loss=0.01325, audio_tagging_loss=0.008669, over 3045909.31 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:59:29,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2841220.0, ans=0.125
2023-11-24 12:59:30,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2841220.0, ans=0.125
2023-11-24 12:59:37,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.311e+01 8.528e+01 9.137e+01 9.887e+01 1.205e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 12:59:40,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426200
2023-11-24 13:00:24,517 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5400, loss[loss=0.08839, simple_loss=0.1191, pruned_loss=0.01822, audio_tagging_loss=0.01061, over 14870.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09199, pruned_loss=0.01334, audio_tagging_loss=0.008761, over 3042388.31 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:00:43,681 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426250
2023-11-24 13:00:43,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2841620.0, ans=0.05
2023-11-24 13:00:55,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2841686.6666666665, ans=0.125
2023-11-24 13:01:04,142 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:01:07,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2841753.3333333335, ans=0.125
2023-11-24 13:01:17,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2841820.0, ans=0.0
2023-11-24 13:01:27,628 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5450, loss[loss=0.05077, simple_loss=0.06096, pruned_loss=0.00944, audio_tagging_loss=0.01084, over 13490.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09133, pruned_loss=0.01316, audio_tagging_loss=0.008835, over 3043173.66 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:01:35,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2841886.6666666665, ans=0.0
2023-11-24 13:01:43,555 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.576e+01 9.179e+01 9.817e+01 1.405e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 13:01:46,726 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426300
2023-11-24 13:01:51,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2842020.0, ans=0.125
2023-11-24 13:02:00,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2842020.0, ans=0.2
2023-11-24 13:02:07,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2842086.6666666665, ans=0.125
2023-11-24 13:02:30,298 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5500, loss[loss=0.06932, simple_loss=0.09431, pruned_loss=0.0141, audio_tagging_loss=0.008063, over 15465.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09106, pruned_loss=0.01322, audio_tagging_loss=0.008828, over 3044007.81 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:02:41,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2842286.6666666665, ans=0.0
2023-11-24 13:02:47,108 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0
2023-11-24 13:02:48,921 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426350
2023-11-24 13:03:06,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2842420.0, ans=0.0
2023-11-24 13:03:06,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2842420.0, ans=0.125
2023-11-24 13:03:32,893 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5550, loss[loss=0.06953, simple_loss=0.09686, pruned_loss=0.01188, audio_tagging_loss=0.009213, over 14005.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09101, pruned_loss=0.01318, audio_tagging_loss=0.008978, over 3050163.48 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 13:03:50,770 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.724e+01 9.337e+01 1.024e+02 1.345e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-24 13:03:52,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426400
2023-11-24 13:03:54,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2842620.0, ans=0.09899494936611666
2023-11-24 13:04:24,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2842820.0, ans=0.125
2023-11-24 13:04:36,532 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5600, loss[loss=0.06074, simple_loss=0.07652, pruned_loss=0.009102, audio_tagging_loss=0.01338, over 14622.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09167, pruned_loss=0.01321, audio_tagging_loss=0.008991, over 3052424.24 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:04:54,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2842953.3333333335, ans=0.0
2023-11-24 13:04:55,187 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426450
2023-11-24 13:04:55,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2842953.3333333335, ans=22.5
2023-11-24 13:05:01,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2843020.0, ans=15.0
2023-11-24 13:05:04,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2843020.0, ans=0.2
2023-11-24 13:05:20,375 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:05:39,334 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5650, loss[loss=0.06105, simple_loss=0.08052, pruned_loss=0.0124, audio_tagging_loss=0.008394, over 15731.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.0909, pruned_loss=0.01309, audio_tagging_loss=0.009136, over 3057982.76 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:05:49,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2843220.0, ans=0.2
2023-11-24 13:05:55,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2843286.6666666665, ans=0.2
2023-11-24 13:05:56,552 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.507e+01 8.997e+01 9.661e+01 1.393e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 13:05:57,874 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426500
2023-11-24 13:05:59,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2843286.6666666665, ans=0.125
2023-11-24 13:06:07,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2843353.3333333335, ans=0.125
2023-11-24 13:06:31,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2843486.6666666665, ans=0.125
2023-11-24 13:06:34,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2843486.6666666665, ans=0.0
2023-11-24 13:06:34,602 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.61 vs. limit=15.0
2023-11-24 13:06:41,761 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5700, loss[loss=0.09546, simple_loss=0.1382, pruned_loss=0.01837, audio_tagging_loss=0.00799, over 14993.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.08973, pruned_loss=0.01277, audio_tagging_loss=0.009223, over 3049500.71 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:06:45,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2843553.3333333335, ans=0.125
2023-11-24 13:06:57,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2843620.0, ans=0.2
2023-11-24 13:07:01,124 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426550
2023-11-24 13:07:04,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2843620.0, ans=0.2
2023-11-24 13:07:16,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2843686.6666666665, ans=0.0
2023-11-24 13:07:25,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2843753.3333333335, ans=0.125
2023-11-24 13:07:37,191 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:07:42,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2843820.0, ans=0.2
2023-11-24 13:07:43,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2843886.6666666665, ans=0.125
2023-11-24 13:07:44,430 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5750, loss[loss=0.0529, simple_loss=0.07248, pruned_loss=0.008419, audio_tagging_loss=0.008242, over 14961.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08932, pruned_loss=0.01269, audio_tagging_loss=0.009096, over 3054724.97 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:07:47,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2843886.6666666665, ans=0.0
2023-11-24 13:08:02,009 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.384e+01 8.873e+01 9.530e+01 1.243e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 13:08:03,321 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426600
2023-11-24 13:08:18,623 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:08:43,366 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.70 vs. limit=15.0
2023-11-24 13:08:48,106 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5800, loss[loss=0.08096, simple_loss=0.1088, pruned_loss=0.01941, audio_tagging_loss=0.007129, over 14692.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.08982, pruned_loss=0.01265, audio_tagging_loss=0.008941, over 3050425.31 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:08:49,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2844220.0, ans=0.2
2023-11-24 13:09:05,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426650
2023-11-24 13:09:16,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2844353.3333333335, ans=0.125
2023-11-24 13:09:28,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2844420.0, ans=0.125
2023-11-24 13:09:29,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.08 vs. limit=15.0
2023-11-24 13:09:30,008 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2844420.0, ans=0.015
2023-11-24 13:09:30,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2844420.0, ans=0.125
2023-11-24 13:09:31,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2844420.0, ans=0.0
2023-11-24 13:09:32,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2844420.0, ans=0.125
2023-11-24 13:09:34,862 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.11 vs. limit=15.0
2023-11-24 13:09:39,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2844486.6666666665, ans=0.0
2023-11-24 13:09:40,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2844486.6666666665, ans=0.0
2023-11-24 13:09:41,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2844486.6666666665, ans=0.1
2023-11-24 13:09:49,426 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5850, loss[loss=0.08415, simple_loss=0.113, pruned_loss=0.01911, audio_tagging_loss=0.008548, over 14901.00 frames. ], tot_loss[loss=0.0663, simple_loss=0.08938, pruned_loss=0.01273, audio_tagging_loss=0.008879, over 3049705.95 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:09:52,048 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:09:57,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2844553.3333333335, ans=0.0
2023-11-24 13:10:07,023 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.013e+01 8.489e+01 9.123e+01 9.809e+01 1.343e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 13:10:08,930 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426700
2023-11-24 13:10:35,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2844753.3333333335, ans=0.125
2023-11-24 13:10:40,877 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.86 vs. limit=15.0
2023-11-24 13:10:43,837 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:10:44,132 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2844820.0, ans=0.04949747468305833
2023-11-24 13:10:48,911 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.04 vs. limit=15.0
2023-11-24 13:10:52,089 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5900, loss[loss=0.04747, simple_loss=0.05797, pruned_loss=0.00789, audio_tagging_loss=0.01059, over 15270.00 frames. ], tot_loss[loss=0.0663, simple_loss=0.08898, pruned_loss=0.01287, audio_tagging_loss=0.008939, over 3043171.48 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:11:11,055 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426750
2023-11-24 13:11:14,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2844953.3333333335, ans=0.125
2023-11-24 13:11:37,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2845086.6666666665, ans=0.0
2023-11-24 13:11:44,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2845153.3333333335, ans=0.0
2023-11-24 13:11:53,911 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 5950, loss[loss=0.05615, simple_loss=0.06696, pruned_loss=0.01147, audio_tagging_loss=0.01121, over 15007.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09034, pruned_loss=0.01296, audio_tagging_loss=0.008858, over 3046768.81 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:11:55,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2845220.0, ans=0.0
2023-11-24 13:11:58,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2845220.0, ans=0.125
2023-11-24 13:12:04,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2845220.0, ans=0.1
2023-11-24 13:12:08,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2845286.6666666665, ans=0.125
2023-11-24 13:12:11,010 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.386e+01 9.064e+01 9.655e+01 1.412e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 13:12:12,312 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426800
2023-11-24 13:12:13,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2845286.6666666665, ans=0.2
2023-11-24 13:12:34,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2845420.0, ans=0.1
2023-11-24 13:12:40,492 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2845420.0, ans=0.125
2023-11-24 13:12:51,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2845486.6666666665, ans=0.09899494936611666
2023-11-24 13:12:55,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2845553.3333333335, ans=0.1
2023-11-24 13:12:56,163 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6000, loss[loss=0.07072, simple_loss=0.09887, pruned_loss=0.01394, audio_tagging_loss=0.007339, over 14834.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09063, pruned_loss=0.01304, audio_tagging_loss=0.008787, over 3044006.10 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:12:56,163 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 13:13:17,811 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2953, 4.2969, 4.4940, 4.4834], device='cuda:2')
2023-11-24 13:13:36,420 INFO [train_asr.py:1253] (2/4) Epoch 36, validation: loss=0.05813, simple_loss=0.0509, pruned_loss=0.005269, audio_tagging_loss=0.02741, over 4681554.00 frames. 
2023-11-24 13:13:36,421 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 13:13:44,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2845553.3333333335, ans=0.2
2023-11-24 13:13:52,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2845620.0, ans=0.1
2023-11-24 13:13:53,639 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:13:54,686 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426850
2023-11-24 13:14:03,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2845686.6666666665, ans=0.125
2023-11-24 13:14:20,378 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:14:38,877 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6050, loss[loss=0.07255, simple_loss=0.09398, pruned_loss=0.01553, audio_tagging_loss=0.01003, over 15344.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09086, pruned_loss=0.01299, audio_tagging_loss=0.008814, over 3047911.89 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:14:46,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2845886.6666666665, ans=0.1
2023-11-24 13:14:51,562 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.54 vs. limit=6.0
2023-11-24 13:14:53,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2845953.3333333335, ans=0.125
2023-11-24 13:14:53,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.41 vs. limit=12.0
2023-11-24 13:14:54,495 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2845953.3333333335, ans=0.125
2023-11-24 13:14:55,410 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.206e+01 8.477e+01 9.229e+01 1.011e+02 1.265e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 13:14:56,709 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426900
2023-11-24 13:15:01,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2846020.0, ans=0.0
2023-11-24 13:15:02,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2846020.0, ans=0.1
2023-11-24 13:15:23,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2846086.6666666665, ans=0.2
2023-11-24 13:15:39,756 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6100, loss[loss=0.0614, simple_loss=0.07363, pruned_loss=0.01704, audio_tagging_loss=0.007548, over 15119.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09116, pruned_loss=0.01309, audio_tagging_loss=0.008695, over 3046068.14 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:15:55,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.47 vs. limit=12.0
2023-11-24 13:15:56,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2846286.6666666665, ans=0.125
2023-11-24 13:15:59,130 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 426950
2023-11-24 13:16:07,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2846353.3333333335, ans=10.0
2023-11-24 13:16:37,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2846486.6666666665, ans=0.07
2023-11-24 13:16:42,229 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6150, loss[loss=0.07958, simple_loss=0.1091, pruned_loss=0.01702, audio_tagging_loss=0.00803, over 14281.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09052, pruned_loss=0.01302, audio_tagging_loss=0.008771, over 3046549.14 frames. ], batch size: 52, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:17:00,447 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.380e+01 9.080e+01 9.869e+01 1.269e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 13:17:01,753 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427000
2023-11-24 13:17:13,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2846686.6666666665, ans=0.1
2023-11-24 13:17:45,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2846886.6666666665, ans=0.04949747468305833
2023-11-24 13:17:46,192 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6200, loss[loss=0.06591, simple_loss=0.09607, pruned_loss=0.008846, audio_tagging_loss=0.009032, over 15611.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.08983, pruned_loss=0.01297, audio_tagging_loss=0.008937, over 3043548.09 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:17:57,074 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2846953.3333333335, ans=0.1
2023-11-24 13:18:00,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2846953.3333333335, ans=0.07
2023-11-24 13:18:04,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427050
2023-11-24 13:18:13,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2847020.0, ans=0.125
2023-11-24 13:18:22,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2847086.6666666665, ans=0.0
2023-11-24 13:18:22,869 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.94 vs. limit=15.0
2023-11-24 13:18:38,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2847153.3333333335, ans=0.0
2023-11-24 13:18:46,641 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:18:48,609 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6250, loss[loss=0.08339, simple_loss=0.1248, pruned_loss=0.01386, audio_tagging_loss=0.007147, over 15647.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.08971, pruned_loss=0.013, audio_tagging_loss=0.009036, over 3039337.65 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:18:48,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2847220.0, ans=0.1
2023-11-24 13:18:56,259 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.10 vs. limit=15.0
2023-11-24 13:18:56,435 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.94 vs. limit=22.5
2023-11-24 13:19:07,100 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.748e+01 9.375e+01 1.005e+02 1.259e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 13:19:07,232 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427100
2023-11-24 13:19:13,179 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.07 vs. limit=15.0
2023-11-24 13:19:15,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2847353.3333333335, ans=0.125
2023-11-24 13:19:25,416 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.67 vs. limit=22.5
2023-11-24 13:19:28,805 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.23 vs. limit=15.0
2023-11-24 13:19:51,307 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6300, loss[loss=0.07551, simple_loss=0.1059, pruned_loss=0.01595, audio_tagging_loss=0.006636, over 15078.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09099, pruned_loss=0.01326, audio_tagging_loss=0.009077, over 3050873.76 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:19:52,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2847553.3333333335, ans=0.125
2023-11-24 13:20:11,579 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427150
2023-11-24 13:20:14,445 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=12.0
2023-11-24 13:20:21,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2847686.6666666665, ans=0.125
2023-11-24 13:20:39,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2847753.3333333335, ans=0.125
2023-11-24 13:20:47,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2847820.0, ans=0.0
2023-11-24 13:20:47,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2847820.0, ans=0.0
2023-11-24 13:20:55,123 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6350, loss[loss=0.07719, simple_loss=0.1063, pruned_loss=0.0149, audio_tagging_loss=0.009111, over 15421.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09132, pruned_loss=0.01324, audio_tagging_loss=0.009194, over 3054485.46 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:21:13,483 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.568e+01 9.156e+01 9.751e+01 1.252e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 13:21:13,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427200
2023-11-24 13:21:21,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2848020.0, ans=0.0
2023-11-24 13:21:22,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2848020.0, ans=0.125
2023-11-24 13:21:45,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2848153.3333333335, ans=0.0
2023-11-24 13:21:50,915 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2848153.3333333335, ans=0.125
2023-11-24 13:21:57,652 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6400, loss[loss=0.07963, simple_loss=0.1109, pruned_loss=0.01359, audio_tagging_loss=0.0106, over 14861.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09061, pruned_loss=0.01319, audio_tagging_loss=0.009267, over 3048243.66 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:22:05,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2848220.0, ans=0.0
2023-11-24 13:22:15,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427250
2023-11-24 13:22:19,401 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.31 vs. limit=12.0
2023-11-24 13:22:35,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2848420.0, ans=0.04949747468305833
2023-11-24 13:22:45,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2848420.0, ans=0.125
2023-11-24 13:22:59,745 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6450, loss[loss=0.0578, simple_loss=0.0821, pruned_loss=0.007547, audio_tagging_loss=0.009205, over 15946.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09086, pruned_loss=0.0131, audio_tagging_loss=0.009263, over 3042184.56 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:23:00,163 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2848553.3333333335, ans=0.2
2023-11-24 13:23:02,525 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2848553.3333333335, ans=0.125
2023-11-24 13:23:17,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2848620.0, ans=0.125
2023-11-24 13:23:18,989 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.428e+01 8.325e+01 9.052e+01 9.909e+01 1.282e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 13:23:19,156 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427300
2023-11-24 13:23:37,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2848753.3333333335, ans=0.125
2023-11-24 13:24:03,302 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6500, loss[loss=0.07807, simple_loss=0.1011, pruned_loss=0.01633, audio_tagging_loss=0.01118, over 15233.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09052, pruned_loss=0.01301, audio_tagging_loss=0.009229, over 3043046.49 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:24:07,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2848886.6666666665, ans=0.0
2023-11-24 13:24:10,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2848886.6666666665, ans=0.07
2023-11-24 13:24:22,745 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427350
2023-11-24 13:24:30,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2849020.0, ans=0.0
2023-11-24 13:24:31,722 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.84 vs. limit=15.0
2023-11-24 13:24:32,429 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2849020.0, ans=0.0
2023-11-24 13:24:39,574 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:25:06,995 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6550, loss[loss=0.05986, simple_loss=0.08091, pruned_loss=0.009737, audio_tagging_loss=0.009669, over 15613.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09044, pruned_loss=0.01305, audio_tagging_loss=0.009114, over 3043491.70 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:25:15,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2849220.0, ans=0.125
2023-11-24 13:25:18,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2849286.6666666665, ans=0.1
2023-11-24 13:25:25,368 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427400
2023-11-24 13:25:26,383 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.536e+01 9.192e+01 9.836e+01 1.277e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-24 13:25:26,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2849286.6666666665, ans=0.0
2023-11-24 13:25:31,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2849353.3333333335, ans=0.0
2023-11-24 13:25:36,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2849353.3333333335, ans=0.0
2023-11-24 13:25:49,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=12.68 vs. limit=22.5
2023-11-24 13:26:06,298 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2849486.6666666665, ans=0.125
2023-11-24 13:26:09,476 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6600, loss[loss=0.09157, simple_loss=0.1313, pruned_loss=0.01989, audio_tagging_loss=0.006016, over 16546.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09095, pruned_loss=0.01313, audio_tagging_loss=0.008949, over 3042337.77 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:26:13,843 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.64 vs. limit=15.0
2023-11-24 13:26:14,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.05 vs. limit=15.0
2023-11-24 13:26:26,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2849620.0, ans=0.2
2023-11-24 13:26:28,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427450
2023-11-24 13:26:31,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2849620.0, ans=0.0
2023-11-24 13:26:40,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2849686.6666666665, ans=0.5
2023-11-24 13:26:43,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2849686.6666666665, ans=0.125
2023-11-24 13:26:52,466 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.40 vs. limit=15.0
2023-11-24 13:27:13,528 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6650, loss[loss=0.06661, simple_loss=0.09637, pruned_loss=0.01226, audio_tagging_loss=0.00617, over 14484.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09106, pruned_loss=0.01317, audio_tagging_loss=0.00884, over 3037131.92 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:27:14,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2849886.6666666665, ans=0.125
2023-11-24 13:27:31,963 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427500
2023-11-24 13:27:33,579 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.517e+01 9.117e+01 9.877e+01 1.434e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 13:27:33,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2849953.3333333335, ans=0.05
2023-11-24 13:27:49,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2850086.6666666665, ans=0.125
2023-11-24 13:28:05,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2850153.3333333335, ans=0.125
2023-11-24 13:28:16,072 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6700, loss[loss=0.05603, simple_loss=0.08123, pruned_loss=0.008805, audio_tagging_loss=0.006608, over 16399.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09109, pruned_loss=0.01333, audio_tagging_loss=0.008862, over 3037821.15 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:28:24,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2850220.0, ans=0.125
2023-11-24 13:28:34,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427550
2023-11-24 13:28:48,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2850353.3333333335, ans=0.0
2023-11-24 13:29:01,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2850420.0, ans=0.07
2023-11-24 13:29:19,045 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6750, loss[loss=0.04194, simple_loss=0.05293, pruned_loss=0.00561, audio_tagging_loss=0.009864, over 14455.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09091, pruned_loss=0.01324, audio_tagging_loss=0.008778, over 3026601.82 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:29:26,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2850553.3333333335, ans=0.09899494936611666
2023-11-24 13:29:38,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427600
2023-11-24 13:29:39,144 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.434e+01 8.966e+01 9.987e+01 1.535e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 13:29:44,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2850686.6666666665, ans=0.0
2023-11-24 13:30:19,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2850820.0, ans=0.125
2023-11-24 13:30:22,662 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6800, loss[loss=0.06847, simple_loss=0.09233, pruned_loss=0.01369, audio_tagging_loss=0.008615, over 15418.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09072, pruned_loss=0.01319, audio_tagging_loss=0.008793, over 3032202.00 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:30:35,429 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:30:35,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2850953.3333333335, ans=0.2
2023-11-24 13:30:41,071 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427650
2023-11-24 13:30:44,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2850953.3333333335, ans=0.0
2023-11-24 13:31:12,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2851153.3333333335, ans=0.125
2023-11-24 13:31:14,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2851153.3333333335, ans=0.2
2023-11-24 13:31:24,681 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6850, loss[loss=0.05971, simple_loss=0.08305, pruned_loss=0.008404, audio_tagging_loss=0.009785, over 15767.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09063, pruned_loss=0.01308, audio_tagging_loss=0.008745, over 3030202.30 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:31:33,631 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.65 vs. limit=15.0
2023-11-24 13:31:36,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2851286.6666666665, ans=0.0
2023-11-24 13:31:42,079 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2851286.6666666665, ans=0.125
2023-11-24 13:31:43,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427700
2023-11-24 13:31:44,249 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.309e+01 9.090e+01 9.871e+01 1.187e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 13:31:44,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2851286.6666666665, ans=0.0
2023-11-24 13:31:49,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2851353.3333333335, ans=0.0
2023-11-24 13:31:57,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2851353.3333333335, ans=0.09899494936611666
2023-11-24 13:31:59,241 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.38 vs. limit=15.0
2023-11-24 13:32:14,830 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2851486.6666666665, ans=0.125
2023-11-24 13:32:26,253 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6900, loss[loss=0.06433, simple_loss=0.08663, pruned_loss=0.01436, audio_tagging_loss=0.006653, over 14991.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09038, pruned_loss=0.01297, audio_tagging_loss=0.008793, over 3026745.52 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:32:35,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2851553.3333333335, ans=0.1
2023-11-24 13:32:45,897 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427750
2023-11-24 13:32:59,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2851686.6666666665, ans=0.125
2023-11-24 13:32:59,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.03 vs. limit=12.0
2023-11-24 13:33:03,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2851753.3333333335, ans=0.125
2023-11-24 13:33:07,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2851753.3333333335, ans=0.0
2023-11-24 13:33:14,570 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:33:28,830 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 6950, loss[loss=0.07198, simple_loss=0.09239, pruned_loss=0.01533, audio_tagging_loss=0.01046, over 14713.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09079, pruned_loss=0.01312, audio_tagging_loss=0.008723, over 3028587.35 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:33:33,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2851886.6666666665, ans=0.2
2023-11-24 13:33:38,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2851886.6666666665, ans=0.2
2023-11-24 13:33:47,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427800
2023-11-24 13:33:50,032 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.545e+01 9.109e+01 9.801e+01 1.234e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 13:33:54,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2852020.0, ans=0.125
2023-11-24 13:34:10,445 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.26 vs. limit=22.5
2023-11-24 13:34:31,859 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7000, loss[loss=0.06848, simple_loss=0.07874, pruned_loss=0.01899, audio_tagging_loss=0.01012, over 14330.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.08998, pruned_loss=0.01295, audio_tagging_loss=0.008837, over 3031361.33 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:34:49,879 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427850
2023-11-24 13:34:52,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2852286.6666666665, ans=0.2
2023-11-24 13:35:02,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2852353.3333333335, ans=0.0
2023-11-24 13:35:04,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.29 vs. limit=15.0
2023-11-24 13:35:06,460 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.83 vs. limit=6.0
2023-11-24 13:35:17,846 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.95 vs. limit=15.0
2023-11-24 13:35:29,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2852486.6666666665, ans=0.125
2023-11-24 13:35:33,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2852553.3333333335, ans=0.1
2023-11-24 13:35:34,280 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7050, loss[loss=0.07274, simple_loss=0.09391, pruned_loss=0.01503, audio_tagging_loss=0.01076, over 15928.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.08998, pruned_loss=0.0129, audio_tagging_loss=0.008926, over 3027552.75 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:35:53,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427900
2023-11-24 13:35:53,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2852620.0, ans=0.1
2023-11-24 13:35:57,375 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.121e+01 8.577e+01 9.267e+01 9.852e+01 1.175e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 13:36:10,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2852686.6666666665, ans=0.125
2023-11-24 13:36:27,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2852820.0, ans=0.0
2023-11-24 13:36:37,831 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7100, loss[loss=0.04996, simple_loss=0.0684, pruned_loss=0.007578, audio_tagging_loss=0.008184, over 16342.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.08995, pruned_loss=0.01284, audio_tagging_loss=0.008962, over 3036628.71 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 13:36:42,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2852886.6666666665, ans=0.0
2023-11-24 13:36:48,587 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=15.0
2023-11-24 13:36:56,906 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 427950
2023-11-24 13:37:11,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2853020.0, ans=0.125
2023-11-24 13:37:15,063 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2853086.6666666665, ans=0.125
2023-11-24 13:37:33,746 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2853153.3333333335, ans=0.125
2023-11-24 13:37:40,821 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7150, loss[loss=0.08442, simple_loss=0.1074, pruned_loss=0.02173, audio_tagging_loss=0.008982, over 15374.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09158, pruned_loss=0.0132, audio_tagging_loss=0.008926, over 3045612.04 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 13:37:46,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2853220.0, ans=0.1
2023-11-24 13:37:48,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2853220.0, ans=0.0
2023-11-24 13:37:57,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2853286.6666666665, ans=0.2
2023-11-24 13:37:59,463 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428000
2023-11-24 13:38:05,902 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.644e+01 9.347e+01 1.029e+02 1.240e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 13:38:11,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2853353.3333333335, ans=0.125
2023-11-24 13:38:21,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2853420.0, ans=0.125
2023-11-24 13:38:29,910 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.03 vs. limit=15.0
2023-11-24 13:38:31,838 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2853420.0, ans=0.125
2023-11-24 13:38:32,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2853486.6666666665, ans=0.125
2023-11-24 13:38:41,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2853486.6666666665, ans=0.125
2023-11-24 13:38:44,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2853486.6666666665, ans=0.04949747468305833
2023-11-24 13:38:46,526 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7200, loss[loss=0.06572, simple_loss=0.09369, pruned_loss=0.0116, audio_tagging_loss=0.007272, over 15337.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09092, pruned_loss=0.01314, audio_tagging_loss=0.008989, over 3039191.25 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:38:58,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2853620.0, ans=6.0
2023-11-24 13:39:04,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428050
2023-11-24 13:39:23,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2853753.3333333335, ans=0.125
2023-11-24 13:39:34,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.20 vs. limit=12.0
2023-11-24 13:39:48,326 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7250, loss[loss=0.06194, simple_loss=0.07813, pruned_loss=0.01324, audio_tagging_loss=0.009637, over 14510.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.0907, pruned_loss=0.01298, audio_tagging_loss=0.009066, over 3041034.27 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:39:51,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2853886.6666666665, ans=0.125
2023-11-24 13:39:52,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.60 vs. limit=22.5
2023-11-24 13:40:08,060 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428100
2023-11-24 13:40:12,047 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.442e+01 8.374e+01 8.915e+01 9.867e+01 1.264e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-24 13:40:19,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2854020.0, ans=0.1
2023-11-24 13:40:52,017 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7300, loss[loss=0.07551, simple_loss=0.1048, pruned_loss=0.01412, audio_tagging_loss=0.009001, over 15184.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09126, pruned_loss=0.01296, audio_tagging_loss=0.009046, over 3041191.04 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:41:01,347 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2854220.0, ans=0.125
2023-11-24 13:41:10,667 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428150
2023-11-24 13:41:10,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2854286.6666666665, ans=0.1
2023-11-24 13:41:30,458 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:41:30,460 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2854420.0, ans=0.1
2023-11-24 13:41:35,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2854420.0, ans=0.125
2023-11-24 13:41:36,188 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.09 vs. limit=6.0
2023-11-24 13:41:53,636 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7350, loss[loss=0.04822, simple_loss=0.06468, pruned_loss=0.008327, audio_tagging_loss=0.007549, over 14868.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.0911, pruned_loss=0.01303, audio_tagging_loss=0.008856, over 3040977.04 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:41:53,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2854553.3333333335, ans=0.125
2023-11-24 13:41:54,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.22 vs. limit=15.0
2023-11-24 13:41:57,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2854553.3333333335, ans=0.125
2023-11-24 13:41:59,821 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2854553.3333333335, ans=0.125
2023-11-24 13:42:08,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2854620.0, ans=0.125
2023-11-24 13:42:12,256 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428200
2023-11-24 13:42:13,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2854620.0, ans=0.0
2023-11-24 13:42:15,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2854620.0, ans=10.0
2023-11-24 13:42:15,966 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.426e+01 8.548e+01 9.171e+01 1.029e+02 1.460e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 13:42:17,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2854686.6666666665, ans=0.5
2023-11-24 13:42:20,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2854686.6666666665, ans=0.125
2023-11-24 13:42:23,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2854686.6666666665, ans=0.125
2023-11-24 13:42:23,381 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2854686.6666666665, ans=0.125
2023-11-24 13:42:41,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2854753.3333333335, ans=0.1
2023-11-24 13:42:48,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2854820.0, ans=0.125
2023-11-24 13:42:50,127 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.50 vs. limit=22.5
2023-11-24 13:42:53,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2854820.0, ans=0.0
2023-11-24 13:42:55,283 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7400, loss[loss=0.07392, simple_loss=0.1051, pruned_loss=0.01502, audio_tagging_loss=0.006348, over 14334.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09071, pruned_loss=0.01291, audio_tagging_loss=0.008836, over 3040208.04 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:43:01,904 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2854886.6666666665, ans=0.125
2023-11-24 13:43:04,178 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2854886.6666666665, ans=0.125
2023-11-24 13:43:08,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2854953.3333333335, ans=0.05
2023-11-24 13:43:14,789 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428250
2023-11-24 13:43:23,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2855020.0, ans=0.125
2023-11-24 13:43:47,130 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:43:51,340 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.21 vs. limit=15.0
2023-11-24 13:43:58,316 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7450, loss[loss=0.08403, simple_loss=0.1249, pruned_loss=0.01589, audio_tagging_loss=0.005692, over 15106.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09073, pruned_loss=0.01287, audio_tagging_loss=0.008827, over 3039140.50 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:44:14,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2855286.6666666665, ans=0.125
2023-11-24 13:44:17,767 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428300
2023-11-24 13:44:21,195 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.449e+01 8.702e+01 9.284e+01 1.003e+02 1.240e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 13:45:01,221 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7500, loss[loss=0.06802, simple_loss=0.08608, pruned_loss=0.01563, audio_tagging_loss=0.009344, over 15360.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09001, pruned_loss=0.01302, audio_tagging_loss=0.008841, over 3058784.54 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:45:19,262 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428350
2023-11-24 13:45:33,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2855686.6666666665, ans=0.125
2023-11-24 13:45:53,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2855820.0, ans=0.2
2023-11-24 13:46:02,120 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.80 vs. limit=15.0
2023-11-24 13:46:02,700 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7550, loss[loss=0.08192, simple_loss=0.1051, pruned_loss=0.01976, audio_tagging_loss=0.00963, over 14560.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09012, pruned_loss=0.01301, audio_tagging_loss=0.008823, over 3059426.95 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:46:10,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2855886.6666666665, ans=0.0
2023-11-24 13:46:21,619 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428400
2023-11-24 13:46:26,024 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.421e+01 8.555e+01 9.258e+01 9.894e+01 1.443e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 13:46:32,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2856020.0, ans=0.125
2023-11-24 13:46:39,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2856086.6666666665, ans=0.125
2023-11-24 13:47:05,642 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7600, loss[loss=0.05877, simple_loss=0.0818, pruned_loss=0.01307, audio_tagging_loss=0.004794, over 15163.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08967, pruned_loss=0.01285, audio_tagging_loss=0.008781, over 3054108.52 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:47:05,880 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2856220.0, ans=0.07
2023-11-24 13:47:07,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2856220.0, ans=0.2
2023-11-24 13:47:13,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2856220.0, ans=0.07
2023-11-24 13:47:24,813 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428450
2023-11-24 13:47:25,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2856286.6666666665, ans=0.0
2023-11-24 13:47:47,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2856420.0, ans=0.125
2023-11-24 13:47:56,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2856486.6666666665, ans=0.125
2023-11-24 13:48:01,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.40 vs. limit=22.5
2023-11-24 13:48:05,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2856486.6666666665, ans=0.2
2023-11-24 13:48:09,000 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7650, loss[loss=0.07079, simple_loss=0.1087, pruned_loss=0.01005, audio_tagging_loss=0.006384, over 15931.00 frames. ], tot_loss[loss=0.06627, simple_loss=0.08933, pruned_loss=0.01278, audio_tagging_loss=0.008831, over 3054023.56 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:48:14,422 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-24 13:48:27,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428500
2023-11-24 13:48:29,286 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.11 vs. limit=22.5
2023-11-24 13:48:30,891 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.493e+01 9.050e+01 9.587e+01 1.815e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 13:48:43,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2856686.6666666665, ans=0.125
2023-11-24 13:48:50,163 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0
2023-11-24 13:48:53,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2856753.3333333335, ans=0.0
2023-11-24 13:48:58,099 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2856753.3333333335, ans=0.2
2023-11-24 13:48:59,564 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.79 vs. limit=10.0
2023-11-24 13:49:06,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2856820.0, ans=0.125
2023-11-24 13:49:10,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2856820.0, ans=0.0
2023-11-24 13:49:12,507 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7700, loss[loss=0.05282, simple_loss=0.07427, pruned_loss=0.008842, audio_tagging_loss=0.00685, over 13648.00 frames. ], tot_loss[loss=0.06627, simple_loss=0.08969, pruned_loss=0.01258, audio_tagging_loss=0.008848, over 3053420.13 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:49:16,438 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2856886.6666666665, ans=0.2
2023-11-24 13:49:28,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2856953.3333333335, ans=0.125
2023-11-24 13:49:29,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2856953.3333333335, ans=0.125
2023-11-24 13:49:31,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428550
2023-11-24 13:49:34,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2856953.3333333335, ans=0.07
2023-11-24 13:49:42,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2857020.0, ans=0.09899494936611666
2023-11-24 13:49:46,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2857020.0, ans=0.125
2023-11-24 13:49:48,125 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:50:04,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2857153.3333333335, ans=0.125
2023-11-24 13:50:15,592 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7750, loss[loss=0.07668, simple_loss=0.1102, pruned_loss=0.01193, audio_tagging_loss=0.009629, over 15124.00 frames. ], tot_loss[loss=0.06651, simple_loss=0.08983, pruned_loss=0.01269, audio_tagging_loss=0.008907, over 3055952.31 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:50:34,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428600
2023-11-24 13:50:38,541 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.354e+01 9.227e+01 9.861e+01 1.458e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 13:50:41,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2857353.3333333335, ans=0.1
2023-11-24 13:51:18,253 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7800, loss[loss=0.08142, simple_loss=0.1096, pruned_loss=0.01587, audio_tagging_loss=0.01075, over 15357.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.09013, pruned_loss=0.01264, audio_tagging_loss=0.008867, over 3050369.82 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:51:36,755 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428650
2023-11-24 13:51:37,153 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=12.0
2023-11-24 13:51:40,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2857620.0, ans=0.0
2023-11-24 13:51:45,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2857686.6666666665, ans=0.1
2023-11-24 13:51:46,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2857686.6666666665, ans=0.1
2023-11-24 13:52:09,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.61 vs. limit=10.0
2023-11-24 13:52:10,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2857820.0, ans=0.125
2023-11-24 13:52:20,575 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7850, loss[loss=0.07576, simple_loss=0.1051, pruned_loss=0.01321, audio_tagging_loss=0.009976, over 15733.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.08987, pruned_loss=0.01271, audio_tagging_loss=0.008961, over 3054649.40 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:52:29,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2857886.6666666665, ans=0.0
2023-11-24 13:52:35,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2857953.3333333335, ans=0.025
2023-11-24 13:52:38,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2857953.3333333335, ans=0.2
2023-11-24 13:52:39,516 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428700
2023-11-24 13:52:43,082 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.656e+01 8.512e+01 9.149e+01 9.887e+01 1.408e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 13:52:48,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2858020.0, ans=0.125
2023-11-24 13:52:57,503 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.12 vs. limit=15.0
2023-11-24 13:53:14,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2858153.3333333335, ans=0.0
2023-11-24 13:53:23,078 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7900, loss[loss=0.07697, simple_loss=0.1009, pruned_loss=0.01814, audio_tagging_loss=0.008375, over 15123.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09035, pruned_loss=0.01277, audio_tagging_loss=0.009003, over 3054366.36 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:53:42,179 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428750
2023-11-24 13:53:43,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2858286.6666666665, ans=0.125
2023-11-24 13:53:47,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2858353.3333333335, ans=0.125
2023-11-24 13:53:53,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2858353.3333333335, ans=0.0
2023-11-24 13:53:59,240 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.46 vs. limit=15.0
2023-11-24 13:54:16,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2858486.6666666665, ans=0.1
2023-11-24 13:54:20,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2858486.6666666665, ans=0.0
2023-11-24 13:54:21,429 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.92 vs. limit=15.0
2023-11-24 13:54:26,264 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 7950, loss[loss=0.07491, simple_loss=0.1001, pruned_loss=0.01484, audio_tagging_loss=0.01003, over 14944.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09023, pruned_loss=0.01278, audio_tagging_loss=0.009096, over 3053598.46 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:54:32,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2858553.3333333335, ans=0.1
2023-11-24 13:54:40,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2858620.0, ans=0.0
2023-11-24 13:54:41,136 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:54:44,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428800
2023-11-24 13:54:49,824 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.877e+01 9.547e+01 1.019e+02 1.290e+02, threshold=1.909e+02, percent-clipped=0.0
2023-11-24 13:55:00,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2858686.6666666665, ans=0.125
2023-11-24 13:55:06,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2858753.3333333335, ans=0.125
2023-11-24 13:55:08,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2858753.3333333335, ans=0.125
2023-11-24 13:55:19,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2858820.0, ans=0.025
2023-11-24 13:55:27,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2858886.6666666665, ans=0.2
2023-11-24 13:55:28,599 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8000, loss[loss=0.05885, simple_loss=0.09005, pruned_loss=0.006097, audio_tagging_loss=0.007729, over 15671.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.08951, pruned_loss=0.01276, audio_tagging_loss=0.009234, over 3052082.79 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:55:46,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428850
2023-11-24 13:56:07,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2859086.6666666665, ans=0.125
2023-11-24 13:56:09,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2859086.6666666665, ans=0.125
2023-11-24 13:56:13,053 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2859086.6666666665, ans=0.09899494936611666
2023-11-24 13:56:15,570 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2859086.6666666665, ans=0.0
2023-11-24 13:56:29,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2859220.0, ans=0.0
2023-11-24 13:56:30,599 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8050, loss[loss=0.05612, simple_loss=0.05817, pruned_loss=0.01377, audio_tagging_loss=0.01327, over 15210.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.08917, pruned_loss=0.01276, audio_tagging_loss=0.00938, over 3054498.07 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:56:40,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2859220.0, ans=0.0
2023-11-24 13:56:44,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2859286.6666666665, ans=0.125
2023-11-24 13:56:49,532 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428900
2023-11-24 13:56:53,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2859286.6666666665, ans=0.2
2023-11-24 13:56:53,489 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.99 vs. limit=15.0
2023-11-24 13:56:55,199 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.620e+01 9.250e+01 9.827e+01 1.123e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 13:57:04,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2859353.3333333335, ans=0.125
2023-11-24 13:57:32,540 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8100, loss[loss=0.05762, simple_loss=0.07278, pruned_loss=0.01225, audio_tagging_loss=0.008971, over 13944.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.08939, pruned_loss=0.01275, audio_tagging_loss=0.009298, over 3055785.35 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:57:44,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2859620.0, ans=0.0
2023-11-24 13:57:51,003 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 428950
2023-11-24 13:58:17,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2859753.3333333335, ans=0.09899494936611666
2023-11-24 13:58:34,215 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8150, loss[loss=0.06724, simple_loss=0.08078, pruned_loss=0.01687, audio_tagging_loss=0.009987, over 14264.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09013, pruned_loss=0.01295, audio_tagging_loss=0.009065, over 3050415.34 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:58:36,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.35 vs. limit=10.0
2023-11-24 13:58:42,284 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.24 vs. limit=15.0
2023-11-24 13:58:53,437 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429000
2023-11-24 13:59:00,121 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.313e+01 8.661e+01 9.369e+01 1.006e+02 1.730e+02, threshold=1.874e+02, percent-clipped=0.0
2023-11-24 13:59:00,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2860020.0, ans=0.0
2023-11-24 13:59:00,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2860020.0, ans=0.125
2023-11-24 13:59:12,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2860086.6666666665, ans=0.125
2023-11-24 13:59:33,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2860153.3333333335, ans=0.1
2023-11-24 13:59:36,943 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8200, loss[loss=0.05375, simple_loss=0.06969, pruned_loss=0.007303, audio_tagging_loss=0.01161, over 15855.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09037, pruned_loss=0.01297, audio_tagging_loss=0.008931, over 3050801.21 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:59:37,004 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:59:45,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2860220.0, ans=0.125
2023-11-24 13:59:55,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429050
2023-11-24 14:00:03,815 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.24 vs. limit=15.0
2023-11-24 14:00:05,093 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.85 vs. limit=15.0
2023-11-24 14:00:39,589 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8250, loss[loss=0.06457, simple_loss=0.08833, pruned_loss=0.01195, audio_tagging_loss=0.008452, over 16300.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09058, pruned_loss=0.01306, audio_tagging_loss=0.008904, over 3048298.71 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:00:50,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2860620.0, ans=0.125
2023-11-24 14:00:57,983 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429100
2023-11-24 14:00:58,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2860620.0, ans=0.04949747468305833
2023-11-24 14:01:03,752 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.495e+01 9.085e+01 9.878e+01 1.172e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 14:01:30,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.59 vs. limit=22.5
2023-11-24 14:01:41,636 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8300, loss[loss=0.06859, simple_loss=0.09775, pruned_loss=0.0122, audio_tagging_loss=0.007524, over 16128.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.08996, pruned_loss=0.01291, audio_tagging_loss=0.008925, over 3054225.98 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 14:01:43,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2860886.6666666665, ans=0.125
2023-11-24 14:01:59,618 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429150
2023-11-24 14:02:23,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2861086.6666666665, ans=0.1
2023-11-24 14:02:23,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2861086.6666666665, ans=0.0
2023-11-24 14:02:31,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2861153.3333333335, ans=0.0
2023-11-24 14:02:32,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2861153.3333333335, ans=0.1
2023-11-24 14:02:42,970 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8350, loss[loss=0.06284, simple_loss=0.08432, pruned_loss=0.008674, audio_tagging_loss=0.01201, over 14407.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.08969, pruned_loss=0.01299, audio_tagging_loss=0.008951, over 3055733.22 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 14:02:43,205 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:02:59,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2861286.6666666665, ans=0.125
2023-11-24 14:03:02,471 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429200
2023-11-24 14:03:05,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2861286.6666666665, ans=0.0
2023-11-24 14:03:09,858 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2861353.3333333335, ans=0.1
2023-11-24 14:03:10,595 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.776e+01 9.274e+01 1.012e+02 1.289e+02, threshold=1.855e+02, percent-clipped=0.0
2023-11-24 14:03:43,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2861486.6666666665, ans=0.2
2023-11-24 14:03:44,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2861486.6666666665, ans=0.0
2023-11-24 14:03:46,362 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8400, loss[loss=0.05099, simple_loss=0.07457, pruned_loss=0.006227, audio_tagging_loss=0.007479, over 14653.00 frames. ], tot_loss[loss=0.0661, simple_loss=0.08863, pruned_loss=0.01279, audio_tagging_loss=0.008997, over 3049073.38 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:03:56,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2861553.3333333335, ans=0.125
2023-11-24 14:04:03,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.80 vs. limit=22.5
2023-11-24 14:04:04,678 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429250
2023-11-24 14:04:08,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2861620.0, ans=0.0
2023-11-24 14:04:08,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.41 vs. limit=15.0
2023-11-24 14:04:32,212 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2861753.3333333335, ans=0.0
2023-11-24 14:04:33,232 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2861753.3333333335, ans=0.0
2023-11-24 14:04:47,754 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8450, loss[loss=0.06621, simple_loss=0.08023, pruned_loss=0.01368, audio_tagging_loss=0.01241, over 15683.00 frames. ], tot_loss[loss=0.06588, simple_loss=0.08833, pruned_loss=0.01279, audio_tagging_loss=0.008934, over 3046510.63 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:04:53,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2861886.6666666665, ans=0.0
2023-11-24 14:04:57,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2861886.6666666665, ans=0.125
2023-11-24 14:04:58,643 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:05:05,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429300
2023-11-24 14:05:13,167 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.551e+01 9.201e+01 9.862e+01 3.144e+02, threshold=1.840e+02, percent-clipped=1.0
2023-11-24 14:05:17,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2862020.0, ans=0.125
2023-11-24 14:05:37,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2862153.3333333335, ans=0.2
2023-11-24 14:05:46,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2862153.3333333335, ans=0.125
2023-11-24 14:05:48,430 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8500, loss[loss=0.09496, simple_loss=0.1353, pruned_loss=0.02094, audio_tagging_loss=0.006356, over 14672.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.08963, pruned_loss=0.01302, audio_tagging_loss=0.008773, over 3043067.18 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:05:57,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2862220.0, ans=0.2
2023-11-24 14:06:07,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2862286.6666666665, ans=0.1
2023-11-24 14:06:08,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429350
2023-11-24 14:06:23,064 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2862353.3333333335, ans=0.125
2023-11-24 14:06:24,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2862353.3333333335, ans=0.125
2023-11-24 14:06:51,107 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8550, loss[loss=0.06311, simple_loss=0.09023, pruned_loss=0.00884, audio_tagging_loss=0.009158, over 15786.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09132, pruned_loss=0.01313, audio_tagging_loss=0.008701, over 3044398.65 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:07:03,671 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.87 vs. limit=15.0
2023-11-24 14:07:06,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2862620.0, ans=0.1
2023-11-24 14:07:10,170 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429400
2023-11-24 14:07:16,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2862686.6666666665, ans=0.125
2023-11-24 14:07:17,339 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.305e+01 8.644e+01 9.213e+01 9.720e+01 1.244e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 14:07:18,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2862686.6666666665, ans=0.125
2023-11-24 14:07:20,258 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.67 vs. limit=12.0
2023-11-24 14:07:25,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2862686.6666666665, ans=0.125
2023-11-24 14:07:29,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2862753.3333333335, ans=0.125
2023-11-24 14:07:45,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2862820.0, ans=0.1
2023-11-24 14:07:46,371 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2862820.0, ans=0.2
2023-11-24 14:07:53,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2862886.6666666665, ans=0.0
2023-11-24 14:07:54,012 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8600, loss[loss=0.1114, simple_loss=0.1699, pruned_loss=0.02273, audio_tagging_loss=0.003704, over 16426.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09215, pruned_loss=0.01346, audio_tagging_loss=0.008788, over 3047303.16 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:07:58,372 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.96 vs. limit=22.5
2023-11-24 14:08:06,821 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.65 vs. limit=22.5
2023-11-24 14:08:12,050 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429450
2023-11-24 14:08:25,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2863020.0, ans=0.0
2023-11-24 14:08:26,715 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.65 vs. limit=15.0
2023-11-24 14:08:27,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2863020.0, ans=0.1
2023-11-24 14:08:55,753 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8650, loss[loss=0.05163, simple_loss=0.07359, pruned_loss=0.006258, audio_tagging_loss=0.008571, over 16271.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09219, pruned_loss=0.01332, audio_tagging_loss=0.00884, over 3046228.09 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:09:10,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2863286.6666666665, ans=0.95
2023-11-24 14:09:14,501 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429500
2023-11-24 14:09:21,999 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.258e+01 8.469e+01 9.080e+01 9.893e+01 1.324e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 14:09:57,748 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8700, loss[loss=0.07178, simple_loss=0.1035, pruned_loss=0.01129, audio_tagging_loss=0.008732, over 16109.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09195, pruned_loss=0.01329, audio_tagging_loss=0.008872, over 3049830.20 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:10:06,812 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2863553.3333333335, ans=0.125
2023-11-24 14:10:15,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2863620.0, ans=0.125
2023-11-24 14:10:17,271 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429550
2023-11-24 14:10:19,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2863620.0, ans=0.0
2023-11-24 14:10:55,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2863820.0, ans=0.1
2023-11-24 14:10:58,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2863820.0, ans=0.1
2023-11-24 14:11:00,210 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8750, loss[loss=0.04673, simple_loss=0.06233, pruned_loss=0.007154, audio_tagging_loss=0.008407, over 14142.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09273, pruned_loss=0.01341, audio_tagging_loss=0.008982, over 3056082.75 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:11:13,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2863953.3333333335, ans=0.125
2023-11-24 14:11:17,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429600
2023-11-24 14:11:18,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2863953.3333333335, ans=0.125
2023-11-24 14:11:25,142 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.456e+01 8.680e+01 9.303e+01 1.028e+02 1.677e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 14:11:39,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2864086.6666666665, ans=0.125
2023-11-24 14:12:02,032 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8800, loss[loss=0.03609, simple_loss=0.03997, pruned_loss=0.004924, audio_tagging_loss=0.01118, over 14154.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09224, pruned_loss=0.01319, audio_tagging_loss=0.009127, over 3056468.82 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:12:07,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2864220.0, ans=0.0
2023-11-24 14:12:21,452 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429650
2023-11-24 14:12:59,118 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:13:04,082 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8850, loss[loss=0.07503, simple_loss=0.09319, pruned_loss=0.01884, audio_tagging_loss=0.009589, over 14251.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09263, pruned_loss=0.01322, audio_tagging_loss=0.00915, over 3058666.27 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:13:14,422 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2864553.3333333335, ans=0.125
2023-11-24 14:13:16,439 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:13:23,692 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429700
2023-11-24 14:13:30,785 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.004e+01 8.562e+01 9.103e+01 9.836e+01 1.222e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 14:13:53,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2864820.0, ans=0.1
2023-11-24 14:14:07,368 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8900, loss[loss=0.07052, simple_loss=0.09457, pruned_loss=0.01351, audio_tagging_loss=0.009723, over 16250.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09226, pruned_loss=0.01314, audio_tagging_loss=0.009013, over 3060215.23 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:14:19,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2864953.3333333335, ans=0.125
2023-11-24 14:14:19,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2864953.3333333335, ans=0.125
2023-11-24 14:14:21,406 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2864953.3333333335, ans=0.125
2023-11-24 14:14:25,770 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429750
2023-11-24 14:14:29,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2864953.3333333335, ans=0.1
2023-11-24 14:14:32,042 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.88 vs. limit=15.0
2023-11-24 14:14:48,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2865086.6666666665, ans=0.0
2023-11-24 14:14:49,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2865086.6666666665, ans=0.0
2023-11-24 14:14:53,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2865086.6666666665, ans=0.2
2023-11-24 14:14:53,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2865086.6666666665, ans=0.2
2023-11-24 14:15:09,575 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 8950, loss[loss=0.06947, simple_loss=0.09837, pruned_loss=0.01222, audio_tagging_loss=0.008072, over 15332.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09165, pruned_loss=0.01304, audio_tagging_loss=0.008929, over 3056047.50 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:15:16,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.69 vs. limit=15.0
2023-11-24 14:15:18,275 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0
2023-11-24 14:15:27,793 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429800
2023-11-24 14:15:35,387 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.550e+01 9.254e+01 9.924e+01 1.359e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 14:16:04,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2865486.6666666665, ans=0.125
2023-11-24 14:16:11,692 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9000, loss[loss=0.07746, simple_loss=0.1062, pruned_loss=0.01529, audio_tagging_loss=0.009074, over 14818.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09083, pruned_loss=0.0131, audio_tagging_loss=0.008924, over 3058356.73 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:16:11,693 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 14:16:42,207 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.1556, 2.4784, 5.0116, 3.0270], device='cuda:2')
2023-11-24 14:16:50,254 INFO [train_asr.py:1253] (2/4) Epoch 36, validation: loss=0.05864, simple_loss=0.05081, pruned_loss=0.005226, audio_tagging_loss=0.02801, over 4681554.00 frames. 
2023-11-24 14:16:50,255 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 14:17:00,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2865553.3333333335, ans=0.1
2023-11-24 14:17:03,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2865620.0, ans=0.125
2023-11-24 14:17:08,433 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429850
2023-11-24 14:17:41,292 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2865820.0, ans=0.125
2023-11-24 14:17:50,611 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.79 vs. limit=22.5
2023-11-24 14:17:52,338 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9050, loss[loss=0.115, simple_loss=0.1677, pruned_loss=0.02558, audio_tagging_loss=0.005526, over 16886.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09176, pruned_loss=0.01336, audio_tagging_loss=0.008878, over 3062056.17 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:17:57,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2865886.6666666665, ans=0.125
2023-11-24 14:18:03,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2865953.3333333335, ans=0.0
2023-11-24 14:18:05,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2865953.3333333335, ans=0.125
2023-11-24 14:18:10,808 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429900
2023-11-24 14:18:12,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2865953.3333333335, ans=0.125
2023-11-24 14:18:19,340 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.179e+01 8.711e+01 9.316e+01 9.957e+01 1.805e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 14:18:23,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2866020.0, ans=0.025
2023-11-24 14:18:35,224 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2866086.6666666665, ans=0.0
2023-11-24 14:18:53,965 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9100, loss[loss=0.07522, simple_loss=0.1048, pruned_loss=0.0151, audio_tagging_loss=0.007711, over 15295.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09132, pruned_loss=0.01322, audio_tagging_loss=0.008829, over 3058138.44 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:18:57,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2866220.0, ans=0.1
2023-11-24 14:19:00,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.95 vs. limit=10.0
2023-11-24 14:19:06,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2866286.6666666665, ans=0.125
2023-11-24 14:19:13,293 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 429950
2023-11-24 14:19:26,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2866353.3333333335, ans=0.1
2023-11-24 14:19:41,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2866420.0, ans=0.0
2023-11-24 14:19:43,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2866486.6666666665, ans=0.2
2023-11-24 14:19:56,592 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9150, loss[loss=0.08514, simple_loss=0.1159, pruned_loss=0.02152, audio_tagging_loss=0.005679, over 14624.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.0914, pruned_loss=0.01334, audio_tagging_loss=0.008822, over 3048019.20 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:20:15,362 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430000
2023-11-24 14:20:15,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2866620.0, ans=0.125
2023-11-24 14:20:23,785 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.464e+01 8.685e+01 9.383e+01 1.043e+02 1.576e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-24 14:20:25,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2866686.6666666665, ans=0.125
2023-11-24 14:20:39,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2866753.3333333335, ans=0.2
2023-11-24 14:20:43,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2866753.3333333335, ans=0.04949747468305833
2023-11-24 14:20:55,132 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.90 vs. limit=12.0
2023-11-24 14:20:59,762 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9200, loss[loss=0.06583, simple_loss=0.08248, pruned_loss=0.01581, audio_tagging_loss=0.008781, over 16556.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09043, pruned_loss=0.01306, audio_tagging_loss=0.008745, over 3049891.63 frames. ], batch size: 64, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:21:08,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2866886.6666666665, ans=0.0
2023-11-24 14:21:18,590 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430050
2023-11-24 14:21:19,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2866953.3333333335, ans=0.125
2023-11-24 14:21:29,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2867020.0, ans=0.125
2023-11-24 14:21:45,831 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2867086.6666666665, ans=0.125
2023-11-24 14:21:53,550 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2867153.3333333335, ans=0.2
2023-11-24 14:22:02,251 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9250, loss[loss=0.07411, simple_loss=0.1024, pruned_loss=0.01726, audio_tagging_loss=0.005671, over 15624.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09051, pruned_loss=0.01311, audio_tagging_loss=0.008849, over 3055062.44 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:22:14,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2867286.6666666665, ans=0.0
2023-11-24 14:22:21,279 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430100
2023-11-24 14:22:28,744 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2867353.3333333335, ans=0.0
2023-11-24 14:22:30,820 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.050e+01 8.538e+01 9.195e+01 9.828e+01 1.249e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 14:23:04,572 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9300, loss[loss=0.07097, simple_loss=0.0959, pruned_loss=0.01372, audio_tagging_loss=0.009301, over 14993.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09072, pruned_loss=0.01297, audio_tagging_loss=0.008859, over 3054498.12 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:23:08,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2867553.3333333335, ans=0.0
2023-11-24 14:23:22,954 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430150
2023-11-24 14:23:49,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2867753.3333333335, ans=0.125
2023-11-24 14:23:51,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2867753.3333333335, ans=0.0
2023-11-24 14:23:55,782 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2867820.0, ans=0.125
2023-11-24 14:23:59,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2867820.0, ans=0.2
2023-11-24 14:24:06,047 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9350, loss[loss=0.06065, simple_loss=0.08914, pruned_loss=0.008682, audio_tagging_loss=0.007397, over 14827.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.09005, pruned_loss=0.01276, audio_tagging_loss=0.008855, over 3051111.03 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:24:25,384 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430200
2023-11-24 14:24:35,737 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.521e+01 8.556e+01 8.988e+01 9.800e+01 1.410e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 14:25:00,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2868153.3333333335, ans=0.07
2023-11-24 14:25:07,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2868220.0, ans=0.125
2023-11-24 14:25:08,882 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9400, loss[loss=0.08687, simple_loss=0.1165, pruned_loss=0.02094, audio_tagging_loss=0.007701, over 14635.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.0907, pruned_loss=0.0129, audio_tagging_loss=0.008853, over 3050163.31 frames. ], batch size: 52, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:25:27,781 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430250
2023-11-24 14:25:29,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2868286.6666666665, ans=0.125
2023-11-24 14:25:31,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2868286.6666666665, ans=0.2
2023-11-24 14:25:32,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2868286.6666666665, ans=0.125
2023-11-24 14:25:41,795 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.88 vs. limit=15.0
2023-11-24 14:25:49,137 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.01 vs. limit=22.5
2023-11-24 14:26:04,643 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0
2023-11-24 14:26:10,706 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:26:11,877 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9450, loss[loss=0.05553, simple_loss=0.08197, pruned_loss=0.005649, audio_tagging_loss=0.008893, over 15161.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09026, pruned_loss=0.01295, audio_tagging_loss=0.008996, over 3049168.87 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:26:18,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2868553.3333333335, ans=0.0
2023-11-24 14:26:30,250 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430300
2023-11-24 14:26:39,331 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.666e+01 9.464e+01 1.022e+02 1.388e+02, threshold=1.893e+02, percent-clipped=0.0
2023-11-24 14:26:55,268 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2868753.3333333335, ans=0.1
2023-11-24 14:27:02,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2868820.0, ans=0.0
2023-11-24 14:27:10,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2868820.0, ans=0.1
2023-11-24 14:27:13,060 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9500, loss[loss=0.07908, simple_loss=0.1032, pruned_loss=0.01658, audio_tagging_loss=0.0109, over 14706.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09034, pruned_loss=0.01298, audio_tagging_loss=0.009122, over 3043600.38 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:27:29,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2868953.3333333335, ans=0.0
2023-11-24 14:27:31,245 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430350
2023-11-24 14:28:10,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2869153.3333333335, ans=0.125
2023-11-24 14:28:13,182 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:28:14,245 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9550, loss[loss=0.06992, simple_loss=0.09307, pruned_loss=0.01479, audio_tagging_loss=0.008591, over 14650.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09101, pruned_loss=0.0131, audio_tagging_loss=0.009178, over 3045127.25 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:28:23,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2869220.0, ans=0.125
2023-11-24 14:28:33,369 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430400
2023-11-24 14:28:43,139 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.513e+01 9.117e+01 9.826e+01 1.260e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 14:29:07,007 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=15.0
2023-11-24 14:29:15,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2869553.3333333335, ans=0.125
2023-11-24 14:29:16,891 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9600, loss[loss=0.05088, simple_loss=0.05688, pruned_loss=0.01096, audio_tagging_loss=0.01148, over 13965.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09215, pruned_loss=0.01331, audio_tagging_loss=0.009043, over 3048074.62 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:29:27,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2869553.3333333335, ans=0.125
2023-11-24 14:29:34,330 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2869620.0, ans=0.125
2023-11-24 14:29:35,222 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430450
2023-11-24 14:29:42,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2869686.6666666665, ans=0.1
2023-11-24 14:30:06,372 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.60 vs. limit=15.0
2023-11-24 14:30:12,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2869820.0, ans=0.125
2023-11-24 14:30:19,688 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9650, loss[loss=0.0698, simple_loss=0.08456, pruned_loss=0.01729, audio_tagging_loss=0.01023, over 15514.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09221, pruned_loss=0.01322, audio_tagging_loss=0.008976, over 3049979.00 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:30:37,835 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430500
2023-11-24 14:30:49,802 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.279e+01 8.447e+01 9.267e+01 9.901e+01 1.317e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 14:30:59,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2870086.6666666665, ans=0.125
2023-11-24 14:30:59,592 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.92 vs. limit=15.0
2023-11-24 14:31:16,932 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2870153.3333333335, ans=0.0
2023-11-24 14:31:22,787 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9700, loss[loss=0.1015, simple_loss=0.1369, pruned_loss=0.02418, audio_tagging_loss=0.008893, over 16323.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09279, pruned_loss=0.01346, audio_tagging_loss=0.008844, over 3049394.58 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:31:24,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2870220.0, ans=0.2
2023-11-24 14:31:29,322 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2870220.0, ans=0.125
2023-11-24 14:31:36,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2870286.6666666665, ans=0.125
2023-11-24 14:31:42,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430550
2023-11-24 14:32:04,954 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2870420.0, ans=0.95
2023-11-24 14:32:20,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2870486.6666666665, ans=0.0
2023-11-24 14:32:25,253 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.21 vs. limit=15.0
2023-11-24 14:32:25,809 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9750, loss[loss=0.07196, simple_loss=0.09726, pruned_loss=0.01465, audio_tagging_loss=0.008673, over 13910.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09217, pruned_loss=0.01345, audio_tagging_loss=0.008796, over 3046428.92 frames. ], batch size: 52, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:32:30,068 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0
2023-11-24 14:32:34,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2870553.3333333335, ans=15.0
2023-11-24 14:32:37,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2870553.3333333335, ans=0.2
2023-11-24 14:32:39,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.96 vs. limit=15.0
2023-11-24 14:32:44,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2870620.0, ans=0.125
2023-11-24 14:32:46,554 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430600
2023-11-24 14:32:57,818 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.545e+01 8.594e+01 9.328e+01 1.010e+02 1.236e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-24 14:33:05,746 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.94 vs. limit=15.0
2023-11-24 14:33:13,427 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2870753.3333333335, ans=0.125
2023-11-24 14:33:16,561 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.65 vs. limit=22.5
2023-11-24 14:33:31,843 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9800, loss[loss=0.05922, simple_loss=0.07461, pruned_loss=0.01036, audio_tagging_loss=0.01155, over 15844.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09138, pruned_loss=0.01329, audio_tagging_loss=0.008856, over 3040556.09 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:33:32,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2870886.6666666665, ans=0.1
2023-11-24 14:33:49,736 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.30 vs. limit=10.0
2023-11-24 14:33:50,283 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430650
2023-11-24 14:33:57,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2871020.0, ans=0.1
2023-11-24 14:34:06,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2871020.0, ans=0.125
2023-11-24 14:34:06,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2871020.0, ans=0.07
2023-11-24 14:34:28,053 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:34:28,425 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2871153.3333333335, ans=0.1
2023-11-24 14:34:34,251 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9850, loss[loss=0.04693, simple_loss=0.05782, pruned_loss=0.009017, audio_tagging_loss=0.009008, over 13899.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09187, pruned_loss=0.01321, audio_tagging_loss=0.008816, over 3038979.69 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:34:34,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2871220.0, ans=0.0
2023-11-24 14:34:43,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.11 vs. limit=22.5
2023-11-24 14:34:53,789 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430700
2023-11-24 14:35:05,555 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.366e+01 9.070e+01 9.726e+01 1.220e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 14:35:09,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2871353.3333333335, ans=0.1
2023-11-24 14:35:15,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2871420.0, ans=0.0
2023-11-24 14:35:37,196 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9900, loss[loss=0.07896, simple_loss=0.1156, pruned_loss=0.01453, audio_tagging_loss=0.006649, over 16152.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09179, pruned_loss=0.0131, audio_tagging_loss=0.00872, over 3035625.07 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:35:37,444 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2871553.3333333335, ans=0.2
2023-11-24 14:35:45,592 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.03 vs. limit=12.0
2023-11-24 14:35:48,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2871553.3333333335, ans=0.125
2023-11-24 14:35:56,903 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430750
2023-11-24 14:36:04,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2871686.6666666665, ans=0.125
2023-11-24 14:36:15,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2871753.3333333335, ans=0.125
2023-11-24 14:36:16,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2871753.3333333335, ans=0.0
2023-11-24 14:36:17,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2871753.3333333335, ans=0.125
2023-11-24 14:36:18,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2871753.3333333335, ans=0.125
2023-11-24 14:36:24,516 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.76 vs. limit=15.0
2023-11-24 14:36:41,000 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 9950, loss[loss=0.05569, simple_loss=0.07831, pruned_loss=0.01095, audio_tagging_loss=0.005587, over 14275.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09208, pruned_loss=0.01308, audio_tagging_loss=0.008757, over 3036404.70 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:36:44,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2871886.6666666665, ans=0.1
2023-11-24 14:36:49,704 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2871886.6666666665, ans=0.125
2023-11-24 14:36:52,741 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.50 vs. limit=6.0
2023-11-24 14:36:54,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2871953.3333333335, ans=0.035
2023-11-24 14:36:54,708 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2871953.3333333335, ans=0.125
2023-11-24 14:36:55,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2871953.3333333335, ans=0.125
2023-11-24 14:36:58,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2871953.3333333335, ans=0.0
2023-11-24 14:36:59,224 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430800
2023-11-24 14:37:05,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2872020.0, ans=0.125
2023-11-24 14:37:11,007 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.477e+01 9.166e+01 9.822e+01 1.194e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 14:37:37,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2872153.3333333335, ans=0.1
2023-11-24 14:37:38,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2872153.3333333335, ans=0.1
2023-11-24 14:37:44,592 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10000, loss[loss=0.04705, simple_loss=0.05975, pruned_loss=0.006119, audio_tagging_loss=0.01106, over 14633.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09103, pruned_loss=0.01277, audio_tagging_loss=0.008661, over 3035791.26 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:37:45,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2872220.0, ans=0.1
2023-11-24 14:37:56,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2872286.6666666665, ans=0.125
2023-11-24 14:38:04,133 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430850
2023-11-24 14:38:34,207 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.33 vs. limit=15.0
2023-11-24 14:38:36,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2872486.6666666665, ans=0.125
2023-11-24 14:38:43,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2872486.6666666665, ans=0.1
2023-11-24 14:38:49,905 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.12 vs. limit=15.0
2023-11-24 14:38:50,170 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10050, loss[loss=0.09291, simple_loss=0.1309, pruned_loss=0.01934, audio_tagging_loss=0.008117, over 14952.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09184, pruned_loss=0.01299, audio_tagging_loss=0.008721, over 3037946.73 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:38:56,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.94 vs. limit=12.0
2023-11-24 14:38:58,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2872553.3333333335, ans=0.125
2023-11-24 14:38:59,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0
2023-11-24 14:39:00,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2872553.3333333335, ans=0.1
2023-11-24 14:39:09,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430900
2023-11-24 14:39:12,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2872620.0, ans=0.2
2023-11-24 14:39:15,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2872686.6666666665, ans=0.1
2023-11-24 14:39:20,965 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.464e+01 8.558e+01 9.038e+01 9.635e+01 1.199e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 14:39:25,153 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.43 vs. limit=15.0
2023-11-24 14:39:26,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2872686.6666666665, ans=0.0
2023-11-24 14:39:36,400 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2872753.3333333335, ans=0.1
2023-11-24 14:39:38,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2872753.3333333335, ans=0.0
2023-11-24 14:39:52,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2872886.6666666665, ans=0.125
2023-11-24 14:39:54,110 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10100, loss[loss=0.05952, simple_loss=0.07501, pruned_loss=0.01132, audio_tagging_loss=0.0107, over 15376.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.0915, pruned_loss=0.0131, audio_tagging_loss=0.008765, over 3041488.58 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:39:55,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2872886.6666666665, ans=0.1
2023-11-24 14:40:05,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2872886.6666666665, ans=0.2
2023-11-24 14:40:09,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2872953.3333333335, ans=0.125
2023-11-24 14:40:13,318 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 430950
2023-11-24 14:40:20,655 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2873020.0, ans=0.1
2023-11-24 14:40:39,371 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.28 vs. limit=22.5
2023-11-24 14:40:41,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2873086.6666666665, ans=0.1
2023-11-24 14:40:46,841 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:40:49,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2873153.3333333335, ans=0.1
2023-11-24 14:40:51,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2873153.3333333335, ans=0.125
2023-11-24 14:40:58,562 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10150, loss[loss=0.06332, simple_loss=0.08143, pruned_loss=0.01382, audio_tagging_loss=0.008783, over 16063.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09208, pruned_loss=0.01323, audio_tagging_loss=0.00878, over 3044778.89 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:41:02,875 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.43 vs. limit=22.5
2023-11-24 14:41:14,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.96 vs. limit=15.0
2023-11-24 14:41:17,701 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431000
2023-11-24 14:41:20,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2873286.6666666665, ans=0.0
2023-11-24 14:41:29,398 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.382e+01 8.738e+01 9.315e+01 1.037e+02 1.393e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 14:41:29,482 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:41:34,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2873353.3333333335, ans=0.125
2023-11-24 14:41:42,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2873420.0, ans=0.125
2023-11-24 14:42:03,129 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10200, loss[loss=0.05923, simple_loss=0.07534, pruned_loss=0.01124, audio_tagging_loss=0.01032, over 15647.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.0916, pruned_loss=0.01319, audio_tagging_loss=0.008928, over 3054979.36 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:42:11,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2873553.3333333335, ans=0.1
2023-11-24 14:42:22,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431050
2023-11-24 14:42:27,531 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:42:41,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2873753.3333333335, ans=0.125
2023-11-24 14:43:06,379 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10250, loss[loss=0.0524, simple_loss=0.06709, pruned_loss=0.01092, audio_tagging_loss=0.007938, over 14717.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09244, pruned_loss=0.01351, audio_tagging_loss=0.008936, over 3055062.71 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 14:43:21,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2873953.3333333335, ans=0.125
2023-11-24 14:43:25,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431100
2023-11-24 14:43:25,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2873953.3333333335, ans=0.125
2023-11-24 14:43:37,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2874020.0, ans=0.0
2023-11-24 14:43:39,456 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.593e+01 9.288e+01 9.901e+01 1.156e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-24 14:44:10,534 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10300, loss[loss=0.07159, simple_loss=0.09697, pruned_loss=0.01037, audio_tagging_loss=0.01273, over 16137.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.0914, pruned_loss=0.01325, audio_tagging_loss=0.009076, over 3056992.38 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 14:44:17,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2874220.0, ans=0.2
2023-11-24 14:44:28,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2874286.6666666665, ans=0.0
2023-11-24 14:44:29,047 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431150
2023-11-24 14:44:29,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2874286.6666666665, ans=0.2
2023-11-24 14:44:30,578 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:44:48,891 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.03 vs. limit=15.0
2023-11-24 14:44:53,737 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.84 vs. limit=6.0
2023-11-24 14:45:12,593 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10350, loss[loss=0.07908, simple_loss=0.1084, pruned_loss=0.01416, audio_tagging_loss=0.01074, over 15935.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09123, pruned_loss=0.01332, audio_tagging_loss=0.009234, over 3055758.34 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 14:45:32,522 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431200
2023-11-24 14:45:35,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2874620.0, ans=0.1
2023-11-24 14:45:46,003 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.769e+01 8.642e+01 9.293e+01 9.950e+01 1.145e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 14:45:50,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2874753.3333333335, ans=0.0
2023-11-24 14:46:17,030 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10400, loss[loss=0.06822, simple_loss=0.08874, pruned_loss=0.01302, audio_tagging_loss=0.01083, over 15461.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09068, pruned_loss=0.01324, audio_tagging_loss=0.009292, over 3052210.93 frames. ], batch size: 63, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:46:21,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2874886.6666666665, ans=0.125
2023-11-24 14:46:36,159 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431250
2023-11-24 14:46:47,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2875020.0, ans=0.1
2023-11-24 14:46:56,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2875086.6666666665, ans=0.0
2023-11-24 14:47:00,527 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2875086.6666666665, ans=0.125
2023-11-24 14:47:08,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2875153.3333333335, ans=0.125
2023-11-24 14:47:21,119 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10450, loss[loss=0.07841, simple_loss=0.09758, pruned_loss=0.01747, audio_tagging_loss=0.01215, over 15534.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09021, pruned_loss=0.01307, audio_tagging_loss=0.009198, over 3051177.35 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:47:40,001 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431300
2023-11-24 14:47:54,048 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.463e+01 9.315e+01 9.862e+01 1.314e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 14:48:05,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2875420.0, ans=0.0
2023-11-24 14:48:24,019 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10500, loss[loss=0.0706, simple_loss=0.1042, pruned_loss=0.0103, audio_tagging_loss=0.008195, over 15454.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09072, pruned_loss=0.01311, audio_tagging_loss=0.00906, over 3049854.67 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:48:36,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2875620.0, ans=0.2
2023-11-24 14:48:42,977 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431350
2023-11-24 14:48:49,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2875686.6666666665, ans=0.1
2023-11-24 14:49:02,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2875753.3333333335, ans=0.04949747468305833
2023-11-24 14:49:07,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2875753.3333333335, ans=0.0
2023-11-24 14:49:09,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2875753.3333333335, ans=0.125
2023-11-24 14:49:21,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2875820.0, ans=0.0
2023-11-24 14:49:26,978 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10550, loss[loss=0.07195, simple_loss=0.1009, pruned_loss=0.01403, audio_tagging_loss=0.007457, over 15260.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09026, pruned_loss=0.01304, audio_tagging_loss=0.008884, over 3047141.16 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:49:45,324 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431400
2023-11-24 14:49:58,958 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.301e+01 8.652e+01 9.234e+01 9.954e+01 1.298e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 14:50:01,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2876020.0, ans=0.125
2023-11-24 14:50:02,887 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2876086.6666666665, ans=0.0
2023-11-24 14:50:27,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2876153.3333333335, ans=0.1
2023-11-24 14:50:29,160 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10600, loss[loss=0.06502, simple_loss=0.0915, pruned_loss=0.01097, audio_tagging_loss=0.008292, over 14274.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09064, pruned_loss=0.013, audio_tagging_loss=0.008821, over 3047296.93 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:50:34,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2876220.0, ans=0.1
2023-11-24 14:50:48,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431450
2023-11-24 14:50:49,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2876286.6666666665, ans=0.125
2023-11-24 14:50:57,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2876353.3333333335, ans=0.04949747468305833
2023-11-24 14:51:31,305 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10650, loss[loss=0.06729, simple_loss=0.09379, pruned_loss=0.009593, audio_tagging_loss=0.0108, over 15595.00 frames. ], tot_loss[loss=0.06648, simple_loss=0.08977, pruned_loss=0.0128, audio_tagging_loss=0.008798, over 3045787.28 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:51:31,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2876553.3333333335, ans=0.125
2023-11-24 14:51:51,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431500
2023-11-24 14:52:04,978 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.032e+01 8.499e+01 9.165e+01 9.904e+01 1.171e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 14:52:07,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2876686.6666666665, ans=0.0
2023-11-24 14:52:28,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2876820.0, ans=0.025
2023-11-24 14:52:36,255 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10700, loss[loss=0.06263, simple_loss=0.08677, pruned_loss=0.01244, audio_tagging_loss=0.006803, over 16071.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.08995, pruned_loss=0.01288, audio_tagging_loss=0.008756, over 3039179.70 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:52:48,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2876953.3333333335, ans=0.125
2023-11-24 14:52:55,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431550
2023-11-24 14:53:00,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2877020.0, ans=0.125
2023-11-24 14:53:03,603 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-24 14:53:31,116 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.93 vs. limit=15.0
2023-11-24 14:53:40,855 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10750, loss[loss=0.05427, simple_loss=0.07224, pruned_loss=0.008686, audio_tagging_loss=0.009465, over 15179.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.08997, pruned_loss=0.01293, audio_tagging_loss=0.008786, over 3037128.46 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:53:41,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2877220.0, ans=0.1
2023-11-24 14:53:59,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431600
2023-11-24 14:54:14,493 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.628e+01 9.351e+01 9.677e+01 1.250e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-24 14:54:43,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.07 vs. limit=10.0
2023-11-24 14:54:43,926 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10800, loss[loss=0.06689, simple_loss=0.07836, pruned_loss=0.01604, audio_tagging_loss=0.01168, over 15900.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09031, pruned_loss=0.01297, audio_tagging_loss=0.008796, over 3040929.52 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:54:44,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2877553.3333333335, ans=0.0
2023-11-24 14:54:51,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2877553.3333333335, ans=0.125
2023-11-24 14:54:54,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2877620.0, ans=0.125
2023-11-24 14:55:02,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431650
2023-11-24 14:55:03,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2877620.0, ans=0.09899494936611666
2023-11-24 14:55:03,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2877620.0, ans=0.2
2023-11-24 14:55:07,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2877686.6666666665, ans=0.2
2023-11-24 14:55:19,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2877686.6666666665, ans=0.0
2023-11-24 14:55:28,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2877753.3333333335, ans=0.125
2023-11-24 14:55:39,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2877820.0, ans=0.0
2023-11-24 14:55:46,755 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10850, loss[loss=0.0639, simple_loss=0.0858, pruned_loss=0.01229, audio_tagging_loss=0.008705, over 14775.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09013, pruned_loss=0.01302, audio_tagging_loss=0.008805, over 3034078.78 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:55:47,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2877886.6666666665, ans=0.1
2023-11-24 14:55:53,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.95 vs. limit=22.5
2023-11-24 14:55:56,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2877886.6666666665, ans=0.125
2023-11-24 14:56:05,717 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431700
2023-11-24 14:56:18,665 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.559e+01 9.138e+01 9.969e+01 1.288e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 14:56:25,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2878086.6666666665, ans=0.125
2023-11-24 14:56:45,762 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:56:47,965 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.23 vs. limit=10.0
2023-11-24 14:56:49,789 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10900, loss[loss=0.06511, simple_loss=0.08853, pruned_loss=0.01252, audio_tagging_loss=0.008333, over 15677.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09024, pruned_loss=0.01303, audio_tagging_loss=0.008802, over 3041306.53 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:56:56,508 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.99 vs. limit=6.0
2023-11-24 14:57:07,883 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431750
2023-11-24 14:57:10,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-24 14:57:16,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2878353.3333333335, ans=0.07
2023-11-24 14:57:44,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2878486.6666666665, ans=0.125
2023-11-24 14:57:51,579 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 10950, loss[loss=0.05117, simple_loss=0.07126, pruned_loss=0.00832, audio_tagging_loss=0.007218, over 16250.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09019, pruned_loss=0.01304, audio_tagging_loss=0.008873, over 3038940.67 frames. ], batch size: 62, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:57:57,721 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2878553.3333333335, ans=0.95
2023-11-24 14:58:01,332 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2878553.3333333335, ans=0.125
2023-11-24 14:58:04,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2878620.0, ans=0.1
2023-11-24 14:58:09,995 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431800
2023-11-24 14:58:18,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2878686.6666666665, ans=0.0
2023-11-24 14:58:24,303 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.538e+01 9.236e+01 1.007e+02 1.326e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 14:58:28,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2878753.3333333335, ans=0.0
2023-11-24 14:58:33,402 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.71 vs. limit=15.0
2023-11-24 14:58:35,984 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.64 vs. limit=22.5
2023-11-24 14:58:47,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2878820.0, ans=0.125
2023-11-24 14:58:52,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2878886.6666666665, ans=0.2
2023-11-24 14:58:53,761 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11000, loss[loss=0.09433, simple_loss=0.118, pruned_loss=0.02607, audio_tagging_loss=0.009256, over 16887.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09022, pruned_loss=0.01292, audio_tagging_loss=0.008939, over 3044531.94 frames. ], batch size: 63, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:59:01,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2878886.6666666665, ans=0.1
2023-11-24 14:59:03,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2878886.6666666665, ans=0.05
2023-11-24 14:59:05,652 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:59:13,462 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431850
2023-11-24 14:59:38,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2879086.6666666665, ans=0.0
2023-11-24 14:59:51,549 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.17 vs. limit=15.0
2023-11-24 14:59:52,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2879153.3333333335, ans=0.2
2023-11-24 14:59:54,595 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2879153.3333333335, ans=0.125
2023-11-24 14:59:56,739 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11050, loss[loss=0.08056, simple_loss=0.1103, pruned_loss=0.01679, audio_tagging_loss=0.008616, over 15619.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09062, pruned_loss=0.01296, audio_tagging_loss=0.009007, over 3042709.81 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:00:01,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2879220.0, ans=0.125
2023-11-24 15:00:15,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431900
2023-11-24 15:00:27,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2879353.3333333335, ans=0.1
2023-11-24 15:00:27,737 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.67 vs. limit=22.5
2023-11-24 15:00:28,280 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.634e+01 9.324e+01 1.017e+02 1.244e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 15:00:35,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2879420.0, ans=0.0
2023-11-24 15:00:38,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2879420.0, ans=0.0
2023-11-24 15:00:54,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2879486.6666666665, ans=0.125
2023-11-24 15:00:58,516 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.61 vs. limit=15.0
2023-11-24 15:00:59,015 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11100, loss[loss=0.06694, simple_loss=0.08757, pruned_loss=0.01284, audio_tagging_loss=0.01031, over 15542.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09054, pruned_loss=0.01296, audio_tagging_loss=0.009078, over 3050022.62 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:01:05,754 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.87 vs. limit=15.0
2023-11-24 15:01:06,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2879553.3333333335, ans=0.2
2023-11-24 15:01:17,722 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 431950
2023-11-24 15:01:19,488 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.08 vs. limit=22.5
2023-11-24 15:01:28,759 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.34 vs. limit=15.0
2023-11-24 15:01:43,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2879753.3333333335, ans=0.125
2023-11-24 15:02:00,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2879886.6666666665, ans=0.1
2023-11-24 15:02:00,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.13 vs. limit=22.5
2023-11-24 15:02:00,875 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11150, loss[loss=0.05173, simple_loss=0.05841, pruned_loss=0.01116, audio_tagging_loss=0.01136, over 15725.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09044, pruned_loss=0.01306, audio_tagging_loss=0.009131, over 3044796.01 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:02:05,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2879886.6666666665, ans=0.0
2023-11-24 15:02:21,344 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432000
2023-11-24 15:02:29,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2880020.0, ans=0.0
2023-11-24 15:02:30,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2880020.0, ans=0.125
2023-11-24 15:02:35,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2880020.0, ans=0.0
2023-11-24 15:02:37,215 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.446e+01 9.218e+01 9.993e+01 1.214e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 15:02:37,797 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.08 vs. limit=6.0
2023-11-24 15:03:03,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2880153.3333333335, ans=0.2
2023-11-24 15:03:07,425 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11200, loss[loss=0.05603, simple_loss=0.07285, pruned_loss=0.01026, audio_tagging_loss=0.009344, over 14639.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08924, pruned_loss=0.01272, audio_tagging_loss=0.00925, over 3044362.69 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:03:19,445 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.23 vs. limit=15.0
2023-11-24 15:03:25,987 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432050
2023-11-24 15:03:50,274 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.02 vs. limit=15.0
2023-11-24 15:03:56,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2880486.6666666665, ans=0.0
2023-11-24 15:04:09,705 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11250, loss[loss=0.0797, simple_loss=0.1061, pruned_loss=0.0176, audio_tagging_loss=0.009042, over 15870.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.08942, pruned_loss=0.01274, audio_tagging_loss=0.009248, over 3048462.46 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:04:11,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2880553.3333333335, ans=0.125
2023-11-24 15:04:27,466 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432100
2023-11-24 15:04:30,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2880620.0, ans=0.1
2023-11-24 15:04:38,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2880686.6666666665, ans=0.0
2023-11-24 15:04:41,558 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.423e+01 8.402e+01 9.085e+01 9.896e+01 1.491e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 15:05:00,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2880820.0, ans=0.125
2023-11-24 15:05:03,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2880820.0, ans=0.0
2023-11-24 15:05:07,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.08 vs. limit=15.0
2023-11-24 15:05:10,588 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11300, loss[loss=0.08783, simple_loss=0.1187, pruned_loss=0.02147, audio_tagging_loss=0.007009, over 15916.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.08981, pruned_loss=0.0128, audio_tagging_loss=0.009102, over 3043387.24 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:05:14,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2880886.6666666665, ans=0.1
2023-11-24 15:05:15,601 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2880886.6666666665, ans=0.05
2023-11-24 15:05:29,594 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432150
2023-11-24 15:05:32,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2880953.3333333335, ans=0.125
2023-11-24 15:05:38,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.63 vs. limit=22.5
2023-11-24 15:05:40,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2881020.0, ans=0.125
2023-11-24 15:05:40,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2881020.0, ans=0.2
2023-11-24 15:05:48,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2881086.6666666665, ans=0.125
2023-11-24 15:06:02,317 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.32 vs. limit=15.0
2023-11-24 15:06:10,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2881153.3333333335, ans=0.125
2023-11-24 15:06:13,223 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11350, loss[loss=0.06728, simple_loss=0.0824, pruned_loss=0.01288, audio_tagging_loss=0.0132, over 15804.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.08953, pruned_loss=0.01275, audio_tagging_loss=0.009043, over 3044446.26 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:06:25,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2881286.6666666665, ans=0.125
2023-11-24 15:06:31,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.43 vs. limit=22.5
2023-11-24 15:06:32,158 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432200
2023-11-24 15:06:45,065 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.23 vs. limit=12.0
2023-11-24 15:06:45,420 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.608e+01 9.253e+01 9.930e+01 1.315e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 15:06:57,809 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.37 vs. limit=15.0
2023-11-24 15:07:16,360 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11400, loss[loss=0.07583, simple_loss=0.1028, pruned_loss=0.01789, audio_tagging_loss=0.006524, over 15971.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.08991, pruned_loss=0.01289, audio_tagging_loss=0.008946, over 3046666.21 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:07:19,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2881553.3333333335, ans=0.125
2023-11-24 15:07:33,213 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2881620.0, ans=0.0
2023-11-24 15:07:34,324 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432250
2023-11-24 15:07:45,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2881686.6666666665, ans=0.2
2023-11-24 15:08:10,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2881820.0, ans=0.0
2023-11-24 15:08:12,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2881820.0, ans=0.125
2023-11-24 15:08:18,279 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11450, loss[loss=0.08366, simple_loss=0.1208, pruned_loss=0.01804, audio_tagging_loss=0.005205, over 16094.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09053, pruned_loss=0.01305, audio_tagging_loss=0.008905, over 3050358.93 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:08:35,060 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2881953.3333333335, ans=0.1
2023-11-24 15:08:37,389 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432300
2023-11-24 15:08:42,742 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2882020.0, ans=0.2
2023-11-24 15:08:44,448 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.57 vs. limit=10.0
2023-11-24 15:08:51,396 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.204e+01 8.606e+01 9.263e+01 9.916e+01 1.290e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 15:09:02,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2882086.6666666665, ans=0.2
2023-11-24 15:09:03,190 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:09:16,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2882153.3333333335, ans=0.125
2023-11-24 15:09:20,631 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11500, loss[loss=0.06691, simple_loss=0.09199, pruned_loss=0.01039, audio_tagging_loss=0.01053, over 14273.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09045, pruned_loss=0.0131, audio_tagging_loss=0.008901, over 3046988.53 frames. ], batch size: 53, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:09:27,419 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2882220.0, ans=0.125
2023-11-24 15:09:39,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432350
2023-11-24 15:09:44,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2882353.3333333335, ans=0.125
2023-11-24 15:09:57,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2882420.0, ans=0.125
2023-11-24 15:10:12,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2882486.6666666665, ans=0.125
2023-11-24 15:10:21,965 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11550, loss[loss=0.0702, simple_loss=0.09766, pruned_loss=0.01425, audio_tagging_loss=0.007122, over 16365.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09029, pruned_loss=0.01298, audio_tagging_loss=0.008906, over 3052562.31 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:10:31,343 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.04 vs. limit=15.0
2023-11-24 15:10:40,360 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432400
2023-11-24 15:10:55,502 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.62 vs. limit=10.0
2023-11-24 15:10:55,832 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.536e+01 9.084e+01 9.886e+01 1.170e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 15:11:01,146 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:11:07,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2882753.3333333335, ans=0.125
2023-11-24 15:11:24,098 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11600, loss[loss=0.06856, simple_loss=0.09498, pruned_loss=0.01132, audio_tagging_loss=0.009749, over 14750.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.09014, pruned_loss=0.01298, audio_tagging_loss=0.00891, over 3050831.66 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:11:30,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2882886.6666666665, ans=10.0
2023-11-24 15:11:43,006 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432450
2023-11-24 15:12:14,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2883153.3333333335, ans=0.125
2023-11-24 15:12:21,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2883153.3333333335, ans=0.125
2023-11-24 15:12:26,854 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11650, loss[loss=0.08461, simple_loss=0.1117, pruned_loss=0.02216, audio_tagging_loss=0.006583, over 14161.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09088, pruned_loss=0.01311, audio_tagging_loss=0.008868, over 3046899.57 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:12:40,845 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2883286.6666666665, ans=0.1
2023-11-24 15:12:45,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432500
2023-11-24 15:12:57,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2883353.3333333335, ans=0.125
2023-11-24 15:13:01,373 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.376e+01 8.463e+01 9.086e+01 9.673e+01 1.142e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 15:13:12,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2883420.0, ans=0.125
2023-11-24 15:13:24,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2883486.6666666665, ans=0.125
2023-11-24 15:13:27,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2883553.3333333335, ans=0.04949747468305833
2023-11-24 15:13:28,772 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11700, loss[loss=0.06929, simple_loss=0.09572, pruned_loss=0.01456, audio_tagging_loss=0.006869, over 14996.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.08996, pruned_loss=0.01299, audio_tagging_loss=0.008951, over 3044800.72 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:13:35,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2883553.3333333335, ans=0.125
2023-11-24 15:13:47,036 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432550
2023-11-24 15:13:57,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2883686.6666666665, ans=0.09899494936611666
2023-11-24 15:14:07,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2883753.3333333335, ans=0.1
2023-11-24 15:14:16,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2883753.3333333335, ans=0.125
2023-11-24 15:14:23,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2883820.0, ans=0.07
2023-11-24 15:14:31,234 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11750, loss[loss=0.07978, simple_loss=0.1018, pruned_loss=0.01923, audio_tagging_loss=0.009677, over 16136.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.08956, pruned_loss=0.01294, audio_tagging_loss=0.008911, over 3048967.07 frames. ], batch size: 62, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:14:50,164 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432600
2023-11-24 15:14:56,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2884020.0, ans=0.125
2023-11-24 15:15:04,210 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2884020.0, ans=0.0
2023-11-24 15:15:06,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2884020.0, ans=0.0
2023-11-24 15:15:07,580 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.388e+01 9.250e+01 9.905e+01 1.592e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 15:15:08,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2884086.6666666665, ans=0.125
2023-11-24 15:15:34,415 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11800, loss[loss=0.08535, simple_loss=0.1069, pruned_loss=0.02372, audio_tagging_loss=0.008202, over 14981.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08888, pruned_loss=0.01301, audio_tagging_loss=0.008993, over 3038429.32 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:15:52,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432650
2023-11-24 15:15:54,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2884286.6666666665, ans=0.2
2023-11-24 15:15:58,070 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=22.5
2023-11-24 15:16:03,040 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.30 vs. limit=15.0
2023-11-24 15:16:10,318 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2884420.0, ans=0.0
2023-11-24 15:16:23,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2884486.6666666665, ans=0.2
2023-11-24 15:16:36,943 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11850, loss[loss=0.06491, simple_loss=0.09232, pruned_loss=0.009368, audio_tagging_loss=0.009387, over 15488.00 frames. ], tot_loss[loss=0.06632, simple_loss=0.089, pruned_loss=0.01282, audio_tagging_loss=0.008998, over 3037384.07 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:16:47,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=2884620.0, ans=0.02
2023-11-24 15:16:53,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2884620.0, ans=0.125
2023-11-24 15:16:54,803 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432700
2023-11-24 15:17:00,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2884686.6666666665, ans=0.125
2023-11-24 15:17:13,151 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.609e+01 8.479e+01 9.210e+01 9.675e+01 1.207e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 15:17:13,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2884753.3333333335, ans=0.0
2023-11-24 15:17:27,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2884820.0, ans=0.035
2023-11-24 15:17:38,791 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11900, loss[loss=0.08243, simple_loss=0.1172, pruned_loss=0.01469, audio_tagging_loss=0.00914, over 16763.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09018, pruned_loss=0.01292, audio_tagging_loss=0.009046, over 3036174.71 frames. ], batch size: 62, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:17:51,855 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=15.0
2023-11-24 15:17:58,161 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432750
2023-11-24 15:18:16,927 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.73 vs. limit=10.0
2023-11-24 15:18:40,572 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 11950, loss[loss=0.08299, simple_loss=0.1193, pruned_loss=0.01578, audio_tagging_loss=0.007553, over 15617.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08965, pruned_loss=0.01271, audio_tagging_loss=0.009144, over 3026523.66 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:18:56,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2885286.6666666665, ans=0.0
2023-11-24 15:19:00,168 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432800
2023-11-24 15:19:02,113 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.30 vs. limit=12.0
2023-11-24 15:19:12,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2885353.3333333335, ans=0.125
2023-11-24 15:19:16,917 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.459e+01 8.985e+01 9.590e+01 1.349e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 15:19:17,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2885420.0, ans=0.125
2023-11-24 15:19:22,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.98 vs. limit=15.0
2023-11-24 15:19:30,294 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.05 vs. limit=15.0
2023-11-24 15:19:37,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.45 vs. limit=15.0
2023-11-24 15:19:41,561 INFO [train_asr.py:1221] (2/4) Epoch 36, batch 12000, loss[loss=0.06851, simple_loss=0.0917, pruned_loss=0.0118, audio_tagging_loss=0.01086, over 15379.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.08967, pruned_loss=0.0128, audio_tagging_loss=0.009216, over 3029390.11 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:19:41,562 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 15:20:12,954 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4980, 3.7996, 2.9185, 3.6839], device='cuda:2')
2023-11-24 15:20:23,127 INFO [train_asr.py:1253] (2/4) Epoch 36, validation: loss=0.05822, simple_loss=0.05085, pruned_loss=0.005219, audio_tagging_loss=0.02757, over 4681554.00 frames. 
2023-11-24 15:20:23,128 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 15:20:25,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2885553.3333333335, ans=0.125
2023-11-24 15:20:38,354 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.47 vs. limit=10.0
2023-11-24 15:20:39,985 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432850
2023-11-24 15:20:43,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.60 vs. limit=12.0
2023-11-24 15:20:44,659 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2885686.6666666665, ans=0.0
2023-11-24 15:21:26,950 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 0, loss[loss=0.08437, simple_loss=0.1033, pruned_loss=0.01255, audio_tagging_loss=0.02017, over 14409.00 frames. ], tot_loss[loss=0.08437, simple_loss=0.1033, pruned_loss=0.01255, audio_tagging_loss=0.02017, over 14409.00 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:21:26,951 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 15:21:41,035 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.0874, 4.7397, 4.4106, 4.4844], device='cuda:2')
2023-11-24 15:21:50,169 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.8843, 1.4740, 3.5085, 2.9994, 2.8391, 3.0836, 3.1199, 3.2515],
       device='cuda:2')
2023-11-24 15:22:03,084 INFO [train_asr.py:1253] (2/4) Epoch 37, validation: loss=0.05797, simple_loss=0.05085, pruned_loss=0.005252, audio_tagging_loss=0.02729, over 4681554.00 frames. 
2023-11-24 15:22:03,085 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 15:22:09,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2885720.0, ans=0.125
2023-11-24 15:22:19,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2885786.6666666665, ans=0.0
2023-11-24 15:22:25,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2885786.6666666665, ans=0.125
2023-11-24 15:22:53,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432900
2023-11-24 15:22:55,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.60 vs. limit=15.0
2023-11-24 15:23:00,263 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:23:06,059 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 50, loss[loss=0.07439, simple_loss=0.09506, pruned_loss=0.01045, audio_tagging_loss=0.01641, over 15246.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09089, pruned_loss=0.01298, audio_tagging_loss=0.01706, over 681464.96 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:23:06,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2886053.3333333335, ans=0.125
2023-11-24 15:23:06,416 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2886053.3333333335, ans=0.125
2023-11-24 15:23:07,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2886053.3333333335, ans=0.125
2023-11-24 15:23:10,788 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.891e+01 9.190e+01 9.778e+01 1.072e+02 1.495e+02, threshold=1.956e+02, percent-clipped=0.0
2023-11-24 15:23:28,212 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:23:39,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2886186.6666666665, ans=0.125
2023-11-24 15:23:47,662 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2886253.3333333335, ans=0.2
2023-11-24 15:23:55,641 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 432950
2023-11-24 15:23:57,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2886320.0, ans=0.1
2023-11-24 15:24:07,951 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 100, loss[loss=0.07203, simple_loss=0.09425, pruned_loss=0.01184, audio_tagging_loss=0.01306, over 15530.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09265, pruned_loss=0.01299, audio_tagging_loss=0.01585, over 1201772.06 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:24:16,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2886386.6666666665, ans=0.1
2023-11-24 15:24:54,483 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2886586.6666666665, ans=0.125
2023-11-24 15:24:57,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433000
2023-11-24 15:25:09,897 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 150, loss[loss=0.06309, simple_loss=0.0705, pruned_loss=0.01177, audio_tagging_loss=0.01606, over 14590.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09041, pruned_loss=0.01272, audio_tagging_loss=0.01459, over 1604706.45 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:25:16,637 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.469e+01 8.996e+01 9.560e+01 1.019e+02 1.193e+02, threshold=1.912e+02, percent-clipped=0.0
2023-11-24 15:25:30,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2886786.6666666665, ans=0.125
2023-11-24 15:25:35,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2886853.3333333335, ans=0.1
2023-11-24 15:25:46,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2886920.0, ans=0.0
2023-11-24 15:25:46,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2886920.0, ans=0.0
2023-11-24 15:25:59,125 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433050
2023-11-24 15:26:00,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2886986.6666666665, ans=0.0
2023-11-24 15:26:12,776 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 200, loss[loss=0.09199, simple_loss=0.1209, pruned_loss=0.01931, audio_tagging_loss=0.01225, over 15321.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09082, pruned_loss=0.01281, audio_tagging_loss=0.01286, over 1926895.20 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:26:38,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2887186.6666666665, ans=0.125
2023-11-24 15:26:46,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2887186.6666666665, ans=0.125
2023-11-24 15:27:02,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433100
2023-11-24 15:27:04,382 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:27:06,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2887320.0, ans=0.0
2023-11-24 15:27:15,139 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 250, loss[loss=0.05041, simple_loss=0.07025, pruned_loss=0.008186, audio_tagging_loss=0.007098, over 15906.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09055, pruned_loss=0.01304, audio_tagging_loss=0.01174, over 2171064.59 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:27:20,917 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.613e+01 8.812e+01 9.417e+01 1.032e+02 1.666e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-24 15:27:44,710 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.74 vs. limit=15.0
2023-11-24 15:27:47,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2887520.0, ans=0.0
2023-11-24 15:28:04,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433150
2023-11-24 15:28:16,219 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 300, loss[loss=0.07994, simple_loss=0.1133, pruned_loss=0.01632, audio_tagging_loss=0.006956, over 16107.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09239, pruned_loss=0.01346, audio_tagging_loss=0.01083, over 2365709.06 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:28:34,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2887786.6666666665, ans=0.125
2023-11-24 15:28:36,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2887786.6666666665, ans=0.125
2023-11-24 15:28:39,154 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.42 vs. limit=22.5
2023-11-24 15:28:57,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2887920.0, ans=0.125
2023-11-24 15:29:05,465 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433200
2023-11-24 15:29:15,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2887986.6666666665, ans=0.125
2023-11-24 15:29:18,706 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 350, loss[loss=0.06997, simple_loss=0.09383, pruned_loss=0.01171, audio_tagging_loss=0.01134, over 16179.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09196, pruned_loss=0.0132, audio_tagging_loss=0.01035, over 2521293.33 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:29:19,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-24 15:29:25,195 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.488e+01 8.531e+01 9.197e+01 9.925e+01 1.241e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 15:29:28,190 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.09 vs. limit=15.0
2023-11-24 15:29:45,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2888186.6666666665, ans=0.125
2023-11-24 15:30:09,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433250
2023-11-24 15:30:12,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2888320.0, ans=0.0
2023-11-24 15:30:17,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2888320.0, ans=0.0
2023-11-24 15:30:21,520 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 400, loss[loss=0.05551, simple_loss=0.07118, pruned_loss=0.01007, audio_tagging_loss=0.009859, over 14736.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09155, pruned_loss=0.01308, audio_tagging_loss=0.01006, over 2637562.24 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:30:27,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2888386.6666666665, ans=0.125
2023-11-24 15:30:52,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2888520.0, ans=0.125
2023-11-24 15:31:03,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2888586.6666666665, ans=0.125
2023-11-24 15:31:11,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433300
2023-11-24 15:31:12,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2888653.3333333335, ans=0.125
2023-11-24 15:31:23,318 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 450, loss[loss=0.08053, simple_loss=0.11, pruned_loss=0.01785, audio_tagging_loss=0.007707, over 15577.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09149, pruned_loss=0.01317, audio_tagging_loss=0.009722, over 2725461.21 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:31:30,278 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.409e+01 9.030e+01 9.754e+01 1.188e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 15:31:56,394 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.68 vs. limit=15.0
2023-11-24 15:32:08,515 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2888920.0, ans=0.05
2023-11-24 15:32:08,887 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.66 vs. limit=15.0
2023-11-24 15:32:11,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2888986.6666666665, ans=10.0
2023-11-24 15:32:12,959 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433350
2023-11-24 15:32:25,465 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 500, loss[loss=0.08659, simple_loss=0.1154, pruned_loss=0.01919, audio_tagging_loss=0.009689, over 14894.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09143, pruned_loss=0.01328, audio_tagging_loss=0.009507, over 2793835.41 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:32:34,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2889053.3333333335, ans=0.125
2023-11-24 15:33:09,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.31 vs. limit=15.0
2023-11-24 15:33:13,145 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2889253.3333333335, ans=0.1
2023-11-24 15:33:16,181 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433400
2023-11-24 15:33:28,899 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 550, loss[loss=0.06272, simple_loss=0.08656, pruned_loss=0.01061, audio_tagging_loss=0.008829, over 15211.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09081, pruned_loss=0.0131, audio_tagging_loss=0.009421, over 2851858.54 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:33:30,679 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.66 vs. limit=15.0
2023-11-24 15:33:36,623 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.403e+01 9.014e+01 9.757e+01 1.245e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 15:33:40,657 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.28 vs. limit=12.0
2023-11-24 15:33:46,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2889453.3333333335, ans=0.125
2023-11-24 15:34:12,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2889586.6666666665, ans=0.125
2023-11-24 15:34:17,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2889653.3333333335, ans=0.2
2023-11-24 15:34:18,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433450
2023-11-24 15:34:21,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2889653.3333333335, ans=0.07
2023-11-24 15:34:30,537 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 600, loss[loss=0.05635, simple_loss=0.0716, pruned_loss=0.01274, audio_tagging_loss=0.007813, over 15362.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09092, pruned_loss=0.01318, audio_tagging_loss=0.009308, over 2895000.76 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:34:40,721 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.03 vs. limit=22.5
2023-11-24 15:34:58,198 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.27 vs. limit=10.0
2023-11-24 15:35:09,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2889920.0, ans=0.1
2023-11-24 15:35:15,894 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2889920.0, ans=0.2
2023-11-24 15:35:20,539 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433500
2023-11-24 15:35:26,100 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.91 vs. limit=22.5
2023-11-24 15:35:33,123 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 650, loss[loss=0.046, simple_loss=0.05698, pruned_loss=0.007757, audio_tagging_loss=0.00975, over 14564.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09065, pruned_loss=0.01313, audio_tagging_loss=0.009245, over 2929180.09 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:35:40,756 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 8.381e+01 9.134e+01 9.904e+01 1.320e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 15:35:58,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2890186.6666666665, ans=0.1
2023-11-24 15:36:19,717 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.45 vs. limit=22.5
2023-11-24 15:36:23,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433550
2023-11-24 15:36:33,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2890320.0, ans=0.125
2023-11-24 15:36:35,536 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 700, loss[loss=0.08343, simple_loss=0.1047, pruned_loss=0.01791, audio_tagging_loss=0.01316, over 15269.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09064, pruned_loss=0.01325, audio_tagging_loss=0.009197, over 2965690.24 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:36:43,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2890386.6666666665, ans=0.125
2023-11-24 15:36:50,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2890453.3333333335, ans=0.2
2023-11-24 15:36:52,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2890453.3333333335, ans=0.125
2023-11-24 15:36:56,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2890453.3333333335, ans=0.2
2023-11-24 15:37:06,934 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2890520.0, ans=0.125
2023-11-24 15:37:16,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2890586.6666666665, ans=0.125
2023-11-24 15:37:25,574 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433600
2023-11-24 15:37:32,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2890653.3333333335, ans=0.0
2023-11-24 15:37:35,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2890653.3333333335, ans=0.2
2023-11-24 15:37:38,604 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 750, loss[loss=0.06215, simple_loss=0.08123, pruned_loss=0.01142, audio_tagging_loss=0.0101, over 15388.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09046, pruned_loss=0.01311, audio_tagging_loss=0.009237, over 2986522.91 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:37:45,697 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.386e+01 8.622e+01 9.264e+01 1.020e+02 2.352e+02, threshold=1.853e+02, percent-clipped=1.0
2023-11-24 15:37:49,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2890786.6666666665, ans=0.035
2023-11-24 15:38:07,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2890853.3333333335, ans=0.0
2023-11-24 15:38:09,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2890853.3333333335, ans=0.07
2023-11-24 15:38:28,485 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433650
2023-11-24 15:38:40,558 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 800, loss[loss=0.08934, simple_loss=0.1314, pruned_loss=0.01552, audio_tagging_loss=0.008106, over 14853.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09137, pruned_loss=0.01322, audio_tagging_loss=0.009197, over 3005489.66 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:38:47,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.49 vs. limit=15.0
2023-11-24 15:38:54,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2891120.0, ans=0.125
2023-11-24 15:39:21,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2891253.3333333335, ans=0.125
2023-11-24 15:39:28,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2891253.3333333335, ans=0.0
2023-11-24 15:39:31,214 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433700
2023-11-24 15:39:31,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2891320.0, ans=0.0
2023-11-24 15:39:40,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2891320.0, ans=0.0
2023-11-24 15:39:41,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2891320.0, ans=0.0
2023-11-24 15:39:43,387 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 850, loss[loss=0.06294, simple_loss=0.09383, pruned_loss=0.008035, audio_tagging_loss=0.007988, over 14864.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09105, pruned_loss=0.0131, audio_tagging_loss=0.009339, over 3016912.52 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:39:51,001 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.494e+01 9.223e+01 9.765e+01 1.207e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 15:40:07,411 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:40:28,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2891586.6666666665, ans=0.05
2023-11-24 15:40:29,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2891586.6666666665, ans=0.025
2023-11-24 15:40:33,376 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433750
2023-11-24 15:40:42,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2891653.3333333335, ans=0.2
2023-11-24 15:40:45,684 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 900, loss[loss=0.07414, simple_loss=0.1062, pruned_loss=0.01308, audio_tagging_loss=0.007963, over 15086.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09094, pruned_loss=0.01316, audio_tagging_loss=0.009349, over 3014846.06 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:40:57,272 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2891786.6666666665, ans=0.125
2023-11-24 15:40:59,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2891786.6666666665, ans=0.1
2023-11-24 15:41:20,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2891853.3333333335, ans=0.125
2023-11-24 15:41:24,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2891920.0, ans=0.125
2023-11-24 15:41:30,835 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.05 vs. limit=15.0
2023-11-24 15:41:32,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2891920.0, ans=0.0
2023-11-24 15:41:35,802 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433800
2023-11-24 15:41:48,025 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 950, loss[loss=0.05239, simple_loss=0.0719, pruned_loss=0.00706, audio_tagging_loss=0.009374, over 15358.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09048, pruned_loss=0.0131, audio_tagging_loss=0.00937, over 3009006.71 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:41:49,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2892053.3333333335, ans=10.0
2023-11-24 15:41:57,415 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.340e+01 8.595e+01 9.194e+01 9.952e+01 1.132e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 15:42:21,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2892186.6666666665, ans=0.125
2023-11-24 15:42:21,330 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.69 vs. limit=15.0
2023-11-24 15:42:38,346 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433850
2023-11-24 15:42:51,316 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1000, loss[loss=0.06833, simple_loss=0.09813, pruned_loss=0.01395, audio_tagging_loss=0.00532, over 14820.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09065, pruned_loss=0.01304, audio_tagging_loss=0.009081, over 3011966.03 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:43:09,177 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=17.72 vs. limit=22.5
2023-11-24 15:43:17,436 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:43:29,146 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.67 vs. limit=22.5
2023-11-24 15:43:40,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433900
2023-11-24 15:43:52,916 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1050, loss[loss=0.05281, simple_loss=0.06548, pruned_loss=0.009097, audio_tagging_loss=0.01097, over 14895.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09117, pruned_loss=0.01312, audio_tagging_loss=0.008944, over 3025082.76 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:43:59,504 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.37 vs. limit=15.0
2023-11-24 15:44:01,781 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.504e+01 9.256e+01 1.007e+02 1.365e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 15:44:21,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.10 vs. limit=15.0
2023-11-24 15:44:34,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2892920.0, ans=0.0
2023-11-24 15:44:34,645 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.31 vs. limit=15.0
2023-11-24 15:44:42,852 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 433950
2023-11-24 15:44:55,271 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1100, loss[loss=0.06, simple_loss=0.07739, pruned_loss=0.01332, audio_tagging_loss=0.007979, over 13640.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09029, pruned_loss=0.01285, audio_tagging_loss=0.008808, over 3023300.04 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:44:57,706 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:45:10,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2893120.0, ans=0.2
2023-11-24 15:45:24,650 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.07 vs. limit=22.5
2023-11-24 15:45:29,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2893186.6666666665, ans=0.1
2023-11-24 15:45:35,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2893253.3333333335, ans=0.0
2023-11-24 15:45:45,333 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434000
2023-11-24 15:45:52,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2893320.0, ans=0.125
2023-11-24 15:45:52,658 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.55 vs. limit=6.0
2023-11-24 15:45:58,258 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1150, loss[loss=0.07005, simple_loss=0.1012, pruned_loss=0.01063, audio_tagging_loss=0.008833, over 15891.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09021, pruned_loss=0.01296, audio_tagging_loss=0.008858, over 3021785.69 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:46:01,165 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-24 15:46:06,382 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.608e+01 9.158e+01 9.709e+01 1.539e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 15:46:24,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2893520.0, ans=0.125
2023-11-24 15:46:46,404 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.14 vs. limit=15.0
2023-11-24 15:46:47,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2893653.3333333335, ans=0.2
2023-11-24 15:46:48,195 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434050
2023-11-24 15:47:00,400 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1200, loss[loss=0.08004, simple_loss=0.1103, pruned_loss=0.01776, audio_tagging_loss=0.007125, over 14028.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08943, pruned_loss=0.01279, audio_tagging_loss=0.008927, over 3025384.17 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:47:00,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2893720.0, ans=0.0
2023-11-24 15:47:09,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.89 vs. limit=10.0
2023-11-24 15:47:50,130 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434100
2023-11-24 15:48:01,997 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1250, loss[loss=0.06096, simple_loss=0.08912, pruned_loss=0.009578, audio_tagging_loss=0.006825, over 15144.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09024, pruned_loss=0.01297, audio_tagging_loss=0.008929, over 3028579.76 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:48:02,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=15.0
2023-11-24 15:48:03,343 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:48:11,439 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.496e+01 8.998e+01 9.780e+01 2.107e+02, threshold=1.800e+02, percent-clipped=1.0
2023-11-24 15:48:11,802 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2894053.3333333335, ans=0.125
2023-11-24 15:48:16,492 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.85 vs. limit=15.0
2023-11-24 15:48:19,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=15.0
2023-11-24 15:48:51,831 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434150
2023-11-24 15:49:05,246 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1300, loss[loss=0.07451, simple_loss=0.1116, pruned_loss=0.01109, audio_tagging_loss=0.007618, over 14056.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.09009, pruned_loss=0.01281, audio_tagging_loss=0.008861, over 3032052.51 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:49:11,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2894386.6666666665, ans=0.125
2023-11-24 15:49:15,419 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.66 vs. limit=15.0
2023-11-24 15:49:33,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2894520.0, ans=0.05
2023-11-24 15:49:36,681 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2894520.0, ans=0.1
2023-11-24 15:49:54,807 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434200
2023-11-24 15:50:07,472 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1350, loss[loss=0.07574, simple_loss=0.1083, pruned_loss=0.01315, audio_tagging_loss=0.008457, over 16631.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09051, pruned_loss=0.01291, audio_tagging_loss=0.008806, over 3036305.95 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:50:14,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2894720.0, ans=0.1
2023-11-24 15:50:15,728 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.208e+01 8.839e+01 9.831e+01 1.172e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-24 15:50:43,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2894920.0, ans=0.125
2023-11-24 15:50:51,208 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:50:52,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2894920.0, ans=0.125
2023-11-24 15:50:56,490 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434250
2023-11-24 15:50:56,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2894986.6666666665, ans=0.0
2023-11-24 15:51:00,311 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2894986.6666666665, ans=0.1
2023-11-24 15:51:08,229 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1400, loss[loss=0.08642, simple_loss=0.1171, pruned_loss=0.01975, audio_tagging_loss=0.008141, over 16004.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.0904, pruned_loss=0.01281, audio_tagging_loss=0.008835, over 3040296.65 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:51:40,670 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:51:45,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2895253.3333333335, ans=0.0
2023-11-24 15:51:47,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2895253.3333333335, ans=0.0
2023-11-24 15:51:57,608 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434300
2023-11-24 15:52:01,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2895320.0, ans=0.0
2023-11-24 15:52:10,537 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1450, loss[loss=0.06927, simple_loss=0.09436, pruned_loss=0.01337, audio_tagging_loss=0.00872, over 15278.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09054, pruned_loss=0.01295, audio_tagging_loss=0.008944, over 3033942.67 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:52:20,299 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.122e+01 8.571e+01 9.242e+01 1.000e+02 1.352e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 15:52:46,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2895586.6666666665, ans=0.2
2023-11-24 15:53:00,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434350
2023-11-24 15:53:09,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2895653.3333333335, ans=0.0
2023-11-24 15:53:10,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2895720.0, ans=0.125
2023-11-24 15:53:12,006 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1500, loss[loss=0.0719, simple_loss=0.1081, pruned_loss=0.01027, audio_tagging_loss=0.007573, over 14890.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.091, pruned_loss=0.01294, audio_tagging_loss=0.00893, over 3035004.88 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:53:19,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2895720.0, ans=0.125
2023-11-24 15:53:40,645 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2895853.3333333335, ans=0.1
2023-11-24 15:53:51,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2895920.0, ans=0.0
2023-11-24 15:53:51,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2895920.0, ans=0.0
2023-11-24 15:53:54,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2895920.0, ans=0.125
2023-11-24 15:54:01,267 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434400
2023-11-24 15:54:03,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2895986.6666666665, ans=0.1
2023-11-24 15:54:04,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2895986.6666666665, ans=0.0
2023-11-24 15:54:04,674 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:54:14,014 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1550, loss[loss=0.0784, simple_loss=0.1043, pruned_loss=0.01484, audio_tagging_loss=0.01139, over 14387.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09091, pruned_loss=0.01289, audio_tagging_loss=0.008939, over 3033161.21 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:54:17,666 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2896053.3333333335, ans=0.0
2023-11-24 15:54:22,723 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:54:23,505 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.499e+01 8.776e+01 9.350e+01 1.010e+02 1.250e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-24 15:55:02,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2896320.0, ans=0.2
2023-11-24 15:55:03,748 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434450
2023-11-24 15:55:16,191 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1600, loss[loss=0.06914, simple_loss=0.09637, pruned_loss=0.01343, audio_tagging_loss=0.007531, over 14534.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09055, pruned_loss=0.01291, audio_tagging_loss=0.009078, over 3033907.30 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:55:21,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2896386.6666666665, ans=0.125
2023-11-24 15:55:26,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2896386.6666666665, ans=0.125
2023-11-24 15:55:31,167 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.38 vs. limit=15.0
2023-11-24 15:55:43,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2896520.0, ans=0.125
2023-11-24 15:55:59,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2896586.6666666665, ans=0.0
2023-11-24 15:56:00,484 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2896586.6666666665, ans=0.2
2023-11-24 15:56:03,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.51 vs. limit=22.5
2023-11-24 15:56:06,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434500
2023-11-24 15:56:06,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2896653.3333333335, ans=0.04949747468305833
2023-11-24 15:56:07,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2896653.3333333335, ans=0.125
2023-11-24 15:56:14,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2896653.3333333335, ans=0.125
2023-11-24 15:56:18,327 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1650, loss[loss=0.07822, simple_loss=0.1166, pruned_loss=0.01133, audio_tagging_loss=0.008599, over 16080.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09059, pruned_loss=0.013, audio_tagging_loss=0.009126, over 3038874.96 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:56:28,155 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.503e+01 8.575e+01 9.140e+01 1.003e+02 1.202e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 15:56:36,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2896786.6666666665, ans=0.125
2023-11-24 15:56:43,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.33 vs. limit=22.5
2023-11-24 15:56:51,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2896853.3333333335, ans=0.2
2023-11-24 15:56:54,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2896920.0, ans=0.125
2023-11-24 15:57:08,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434550
2023-11-24 15:57:09,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2896986.6666666665, ans=0.2
2023-11-24 15:57:17,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.76 vs. limit=15.0
2023-11-24 15:57:20,455 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1700, loss[loss=0.06042, simple_loss=0.08598, pruned_loss=0.00941, audio_tagging_loss=0.008025, over 16839.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09133, pruned_loss=0.01292, audio_tagging_loss=0.00913, over 3043966.60 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:57:22,170 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.92 vs. limit=12.0
2023-11-24 15:57:24,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2897053.3333333335, ans=0.125
2023-11-24 15:57:33,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2897120.0, ans=0.125
2023-11-24 15:57:33,806 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-24 15:57:35,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2897120.0, ans=0.125
2023-11-24 15:58:06,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2897253.3333333335, ans=0.0
2023-11-24 15:58:07,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2897253.3333333335, ans=0.125
2023-11-24 15:58:10,025 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434600
2023-11-24 15:58:22,746 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1750, loss[loss=0.06521, simple_loss=0.08796, pruned_loss=0.01175, audio_tagging_loss=0.009483, over 13842.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09039, pruned_loss=0.01286, audio_tagging_loss=0.009104, over 3041918.00 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:58:30,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2897386.6666666665, ans=0.125
2023-11-24 15:58:33,876 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 8.384e+01 9.014e+01 9.814e+01 1.863e+02, threshold=1.803e+02, percent-clipped=1.0
2023-11-24 15:58:44,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.57 vs. limit=22.5
2023-11-24 15:58:55,391 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:58:56,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2897520.0, ans=0.07
2023-11-24 15:59:03,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2897586.6666666665, ans=0.2
2023-11-24 15:59:09,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2897586.6666666665, ans=0.125
2023-11-24 15:59:12,505 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434650
2023-11-24 15:59:24,847 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1800, loss[loss=0.07064, simple_loss=0.1036, pruned_loss=0.01142, audio_tagging_loss=0.007407, over 14882.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09048, pruned_loss=0.01299, audio_tagging_loss=0.008939, over 3037442.50 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:59:31,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2897720.0, ans=0.125
2023-11-24 15:59:38,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2897786.6666666665, ans=0.04949747468305833
2023-11-24 15:59:48,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2897853.3333333335, ans=0.125
2023-11-24 16:00:06,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2897920.0, ans=0.0
2023-11-24 16:00:11,196 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2897920.0, ans=0.0
2023-11-24 16:00:15,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434700
2023-11-24 16:00:19,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2897986.6666666665, ans=0.125
2023-11-24 16:00:23,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2897986.6666666665, ans=0.95
2023-11-24 16:00:27,536 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1850, loss[loss=0.07429, simple_loss=0.09952, pruned_loss=0.01671, audio_tagging_loss=0.007813, over 14903.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09041, pruned_loss=0.01304, audio_tagging_loss=0.008886, over 3043226.62 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:00:38,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.639e+01 8.698e+01 9.278e+01 9.936e+01 1.415e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 16:00:52,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2898186.6666666665, ans=0.0
2023-11-24 16:01:11,331 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.60 vs. limit=15.0
2023-11-24 16:01:17,899 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434750
2023-11-24 16:01:29,998 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1900, loss[loss=0.06924, simple_loss=0.09255, pruned_loss=0.01321, audio_tagging_loss=0.009763, over 15222.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09168, pruned_loss=0.01318, audio_tagging_loss=0.008751, over 3048685.87 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:01:35,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2898386.6666666665, ans=0.125
2023-11-24 16:02:14,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2898586.6666666665, ans=0.125
2023-11-24 16:02:20,222 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434800
2023-11-24 16:02:23,108 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:02:32,970 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 1950, loss[loss=0.05148, simple_loss=0.06398, pruned_loss=0.008897, audio_tagging_loss=0.01059, over 14471.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09049, pruned_loss=0.013, audio_tagging_loss=0.00875, over 3043199.52 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:02:39,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2898720.0, ans=0.0
2023-11-24 16:02:43,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2898720.0, ans=0.125
2023-11-24 16:02:44,687 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.590e+01 9.514e+01 1.026e+02 1.248e+02, threshold=1.903e+02, percent-clipped=0.0
2023-11-24 16:03:02,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.41 vs. limit=5.0
2023-11-24 16:03:22,715 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434850
2023-11-24 16:03:28,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2898986.6666666665, ans=0.125
2023-11-24 16:03:34,753 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2899053.3333333335, ans=0.0
2023-11-24 16:03:35,530 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2000, loss[loss=0.07753, simple_loss=0.1064, pruned_loss=0.01457, audio_tagging_loss=0.009755, over 15409.00 frames. ], tot_loss[loss=0.06648, simple_loss=0.0896, pruned_loss=0.01287, audio_tagging_loss=0.008801, over 3035844.55 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:04:05,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2899186.6666666665, ans=0.1
2023-11-24 16:04:05,535 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.05 vs. limit=15.0
2023-11-24 16:04:25,319 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434900
2023-11-24 16:04:36,758 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2050, loss[loss=0.06312, simple_loss=0.08906, pruned_loss=0.0102, audio_tagging_loss=0.00839, over 15402.00 frames. ], tot_loss[loss=0.06633, simple_loss=0.08955, pruned_loss=0.01283, audio_tagging_loss=0.008732, over 3034755.40 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:04:44,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2899386.6666666665, ans=0.2
2023-11-24 16:04:49,653 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.636e+01 9.159e+01 9.953e+01 1.239e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 16:04:52,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2899453.3333333335, ans=0.125
2023-11-24 16:05:00,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2899520.0, ans=0.125
2023-11-24 16:05:02,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2899520.0, ans=0.125
2023-11-24 16:05:06,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2899520.0, ans=0.125
2023-11-24 16:05:11,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2899520.0, ans=0.0
2023-11-24 16:05:12,202 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.35 vs. limit=22.5
2023-11-24 16:05:27,046 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 434950
2023-11-24 16:05:32,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2899653.3333333335, ans=0.0
2023-11-24 16:05:34,921 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2899653.3333333335, ans=0.2
2023-11-24 16:05:39,836 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2100, loss[loss=0.0725, simple_loss=0.1032, pruned_loss=0.01372, audio_tagging_loss=0.007168, over 15117.00 frames. ], tot_loss[loss=0.06653, simple_loss=0.08995, pruned_loss=0.01285, audio_tagging_loss=0.008704, over 3038799.91 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:06:03,877 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2899853.3333333335, ans=0.125
2023-11-24 16:06:05,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2899853.3333333335, ans=0.0
2023-11-24 16:06:07,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2899853.3333333335, ans=0.0
2023-11-24 16:06:29,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435000
2023-11-24 16:06:42,436 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2150, loss[loss=0.05169, simple_loss=0.06508, pruned_loss=0.008353, audio_tagging_loss=0.0108, over 14743.00 frames. ], tot_loss[loss=0.0662, simple_loss=0.08932, pruned_loss=0.01275, audio_tagging_loss=0.008787, over 3035593.61 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:06:52,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2900053.3333333335, ans=0.125
2023-11-24 16:06:54,771 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.565e+01 9.008e+01 9.643e+01 1.315e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 16:06:58,677 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2900120.0, ans=0.0
2023-11-24 16:07:01,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2900120.0, ans=15.0
2023-11-24 16:07:07,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2900186.6666666665, ans=0.035
2023-11-24 16:07:19,828 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:07:30,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2900253.3333333335, ans=0.125
2023-11-24 16:07:32,286 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435050
2023-11-24 16:07:44,817 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2200, loss[loss=0.06834, simple_loss=0.08949, pruned_loss=0.01498, audio_tagging_loss=0.008615, over 15453.00 frames. ], tot_loss[loss=0.06649, simple_loss=0.08971, pruned_loss=0.01278, audio_tagging_loss=0.008852, over 3037516.27 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:08:15,039 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.48 vs. limit=15.0
2023-11-24 16:08:24,060 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.24 vs. limit=22.5
2023-11-24 16:08:26,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2900586.6666666665, ans=0.125
2023-11-24 16:08:34,453 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435100
2023-11-24 16:08:47,279 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2250, loss[loss=0.07725, simple_loss=0.1149, pruned_loss=0.01302, audio_tagging_loss=0.00677, over 14563.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09037, pruned_loss=0.01301, audio_tagging_loss=0.008724, over 3043265.37 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:08:47,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2900720.0, ans=0.2
2023-11-24 16:08:59,269 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.499e+01 9.314e+01 1.008e+02 1.259e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 16:08:59,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2900786.6666666665, ans=0.2
2023-11-24 16:09:11,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2900853.3333333335, ans=0.125
2023-11-24 16:09:36,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.27 vs. limit=12.0
2023-11-24 16:09:37,553 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435150
2023-11-24 16:09:39,528 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.90 vs. limit=15.0
2023-11-24 16:09:41,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2900986.6666666665, ans=0.125
2023-11-24 16:09:41,654 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.58 vs. limit=22.5
2023-11-24 16:09:44,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2900986.6666666665, ans=0.125
2023-11-24 16:09:49,108 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2300, loss[loss=0.08095, simple_loss=0.1155, pruned_loss=0.01607, audio_tagging_loss=0.007126, over 15772.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.08999, pruned_loss=0.01301, audio_tagging_loss=0.008771, over 3049762.79 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:10:18,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=2901186.6666666665, ans=0.02
2023-11-24 16:10:39,084 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435200
2023-11-24 16:10:44,189 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:10:45,995 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.33 vs. limit=15.0
2023-11-24 16:10:51,726 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2350, loss[loss=0.08863, simple_loss=0.114, pruned_loss=0.02212, audio_tagging_loss=0.009491, over 16318.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08993, pruned_loss=0.01288, audio_tagging_loss=0.008844, over 3047422.01 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:11:04,100 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.491e+01 9.098e+01 9.735e+01 1.165e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 16:11:12,378 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.10 vs. limit=22.5
2023-11-24 16:11:14,849 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.47 vs. limit=15.0
2023-11-24 16:11:20,989 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.37 vs. limit=22.5
2023-11-24 16:11:25,946 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2901520.0, ans=15.0
2023-11-24 16:11:27,939 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2901586.6666666665, ans=0.125
2023-11-24 16:11:41,201 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435250
2023-11-24 16:11:41,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2901653.3333333335, ans=0.1
2023-11-24 16:11:53,575 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2400, loss[loss=0.06652, simple_loss=0.08012, pruned_loss=0.01241, audio_tagging_loss=0.01405, over 15231.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09037, pruned_loss=0.01296, audio_tagging_loss=0.00897, over 3051083.60 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:11:59,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2901720.0, ans=0.125
2023-11-24 16:12:06,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2901786.6666666665, ans=0.125
2023-11-24 16:12:27,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2901853.3333333335, ans=0.125
2023-11-24 16:12:30,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2901920.0, ans=0.0
2023-11-24 16:12:43,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2901986.6666666665, ans=0.125
2023-11-24 16:12:44,387 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435300
2023-11-24 16:12:55,579 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.91 vs. limit=15.0
2023-11-24 16:12:56,199 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2450, loss[loss=0.06827, simple_loss=0.08532, pruned_loss=0.01444, audio_tagging_loss=0.01117, over 15795.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09168, pruned_loss=0.01317, audio_tagging_loss=0.008956, over 3060586.67 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:13:09,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.445e+01 8.651e+01 9.072e+01 9.865e+01 1.248e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 16:13:33,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2902253.3333333335, ans=0.125
2023-11-24 16:13:36,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2902253.3333333335, ans=0.125
2023-11-24 16:13:46,169 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435350
2023-11-24 16:13:58,419 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2500, loss[loss=0.06377, simple_loss=0.08432, pruned_loss=0.01245, audio_tagging_loss=0.009157, over 15998.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09091, pruned_loss=0.01305, audio_tagging_loss=0.009115, over 3052398.15 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:14:01,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2902386.6666666665, ans=0.125
2023-11-24 16:14:12,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2902453.3333333335, ans=0.0
2023-11-24 16:14:24,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2902520.0, ans=0.2
2023-11-24 16:14:26,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2902520.0, ans=0.0
2023-11-24 16:14:31,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2902520.0, ans=0.125
2023-11-24 16:14:34,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2902520.0, ans=0.2
2023-11-24 16:14:38,030 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2902586.6666666665, ans=0.0
2023-11-24 16:14:48,502 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435400
2023-11-24 16:14:50,140 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.97 vs. limit=12.0
2023-11-24 16:15:01,951 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2550, loss[loss=0.05746, simple_loss=0.07207, pruned_loss=0.01015, audio_tagging_loss=0.01127, over 14307.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09136, pruned_loss=0.01299, audio_tagging_loss=0.009038, over 3048237.66 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:15:15,543 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.436e+01 8.712e+01 9.254e+01 9.868e+01 2.546e+02, threshold=1.851e+02, percent-clipped=1.0
2023-11-24 16:15:18,157 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2902786.6666666665, ans=0.125
2023-11-24 16:15:30,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2902853.3333333335, ans=0.1
2023-11-24 16:15:36,391 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:15:42,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2902920.0, ans=0.125
2023-11-24 16:15:50,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2902986.6666666665, ans=0.1
2023-11-24 16:15:51,492 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435450
2023-11-24 16:15:56,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2902986.6666666665, ans=0.0
2023-11-24 16:16:03,752 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2600, loss[loss=0.06473, simple_loss=0.08423, pruned_loss=0.01501, audio_tagging_loss=0.007608, over 13494.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09101, pruned_loss=0.01297, audio_tagging_loss=0.008877, over 3051263.44 frames. ], batch size: 52, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:16:06,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2903053.3333333335, ans=0.125
2023-11-24 16:16:21,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2903120.0, ans=0.125
2023-11-24 16:16:23,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2903120.0, ans=0.125
2023-11-24 16:16:49,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2903253.3333333335, ans=0.125
2023-11-24 16:16:53,220 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435500
2023-11-24 16:17:05,539 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2650, loss[loss=0.06323, simple_loss=0.09188, pruned_loss=0.01001, audio_tagging_loss=0.007282, over 14202.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09102, pruned_loss=0.01301, audio_tagging_loss=0.00876, over 3044088.64 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:17:18,506 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.439e+01 8.953e+01 1.003e+02 3.059e+02, threshold=1.791e+02, percent-clipped=1.0
2023-11-24 16:17:29,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2903520.0, ans=0.125
2023-11-24 16:17:30,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.47 vs. limit=15.0
2023-11-24 16:17:54,381 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435550
2023-11-24 16:17:55,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.14 vs. limit=15.0
2023-11-24 16:18:05,545 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.18 vs. limit=15.0
2023-11-24 16:18:06,100 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2700, loss[loss=0.06461, simple_loss=0.08154, pruned_loss=0.01686, audio_tagging_loss=0.006977, over 15424.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09103, pruned_loss=0.01317, audio_tagging_loss=0.008652, over 3040892.03 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:18:06,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2903720.0, ans=0.1
2023-11-24 16:18:09,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2903720.0, ans=0.0
2023-11-24 16:18:22,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2903786.6666666665, ans=0.2
2023-11-24 16:18:23,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2903786.6666666665, ans=0.0
2023-11-24 16:18:56,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435600
2023-11-24 16:19:09,897 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2750, loss[loss=0.05865, simple_loss=0.07958, pruned_loss=0.008866, audio_tagging_loss=0.009988, over 14671.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.09004, pruned_loss=0.0129, audio_tagging_loss=0.008701, over 3035782.12 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 8.0
2023-11-24 16:19:24,581 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.485e+01 8.630e+01 9.216e+01 9.892e+01 1.188e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 16:19:44,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2904186.6666666665, ans=0.1
2023-11-24 16:19:59,751 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435650
2023-11-24 16:20:02,138 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:20:08,058 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.54 vs. limit=15.0
2023-11-24 16:20:11,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2904386.6666666665, ans=0.1
2023-11-24 16:20:12,158 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2800, loss[loss=0.04581, simple_loss=0.05931, pruned_loss=0.007202, audio_tagging_loss=0.008952, over 13659.00 frames. ], tot_loss[loss=0.06632, simple_loss=0.08972, pruned_loss=0.01273, audio_tagging_loss=0.008737, over 3035911.35 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:20:13,834 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.00 vs. limit=15.0
2023-11-24 16:20:14,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2904386.6666666665, ans=0.125
2023-11-24 16:20:27,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2904453.3333333335, ans=0.025
2023-11-24 16:20:36,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2904520.0, ans=0.0
2023-11-24 16:20:47,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2904520.0, ans=0.2
2023-11-24 16:21:01,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435700
2023-11-24 16:21:12,827 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.37 vs. limit=15.0
2023-11-24 16:21:13,369 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2850, loss[loss=0.06042, simple_loss=0.08362, pruned_loss=0.01248, audio_tagging_loss=0.00613, over 14530.00 frames. ], tot_loss[loss=0.06654, simple_loss=0.09003, pruned_loss=0.01282, audio_tagging_loss=0.008702, over 3039799.92 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:21:29,331 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.333e+01 9.111e+01 9.785e+01 2.308e+02, threshold=1.822e+02, percent-clipped=1.0
2023-11-24 16:21:46,779 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2904853.3333333335, ans=0.025
2023-11-24 16:22:01,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2904920.0, ans=0.0
2023-11-24 16:22:03,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435750
2023-11-24 16:22:05,468 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2904986.6666666665, ans=0.0
2023-11-24 16:22:07,738 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2904986.6666666665, ans=0.1
2023-11-24 16:22:16,935 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2900, loss[loss=0.06758, simple_loss=0.09048, pruned_loss=0.01638, audio_tagging_loss=0.005955, over 15290.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09067, pruned_loss=0.0129, audio_tagging_loss=0.008654, over 3043482.21 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:22:45,126 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2905186.6666666665, ans=0.1
2023-11-24 16:23:06,459 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435800
2023-11-24 16:23:12,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2905320.0, ans=0.125
2023-11-24 16:23:16,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2905320.0, ans=0.125
2023-11-24 16:23:19,020 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 2950, loss[loss=0.08638, simple_loss=0.125, pruned_loss=0.01542, audio_tagging_loss=0.008447, over 14372.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.09059, pruned_loss=0.01287, audio_tagging_loss=0.008695, over 3041558.89 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:23:22,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2905386.6666666665, ans=0.2
2023-11-24 16:23:33,485 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.426e+01 9.166e+01 9.683e+01 1.219e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 16:23:41,566 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2905453.3333333335, ans=0.0
2023-11-24 16:23:49,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2905520.0, ans=0.2
2023-11-24 16:23:49,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2905520.0, ans=0.125
2023-11-24 16:23:53,222 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2905520.0, ans=0.1
2023-11-24 16:23:53,800 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.40 vs. limit=15.0
2023-11-24 16:23:54,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2905520.0, ans=0.1
2023-11-24 16:23:56,058 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.42 vs. limit=10.0
2023-11-24 16:23:59,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2905586.6666666665, ans=0.125
2023-11-24 16:24:09,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435850
2023-11-24 16:24:13,358 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.72 vs. limit=22.5
2023-11-24 16:24:15,295 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2905653.3333333335, ans=0.125
2023-11-24 16:24:20,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2905720.0, ans=0.125
2023-11-24 16:24:21,032 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3000, loss[loss=0.06889, simple_loss=0.08117, pruned_loss=0.01812, audio_tagging_loss=0.01017, over 14475.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.09007, pruned_loss=0.01287, audio_tagging_loss=0.008791, over 3039343.70 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:24:21,033 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 16:25:01,971 INFO [train_asr.py:1253] (2/4) Epoch 37, validation: loss=0.05757, simple_loss=0.05085, pruned_loss=0.005185, audio_tagging_loss=0.02697, over 4681554.00 frames. 
2023-11-24 16:25:01,972 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 16:25:02,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2905720.0, ans=0.0
2023-11-24 16:25:07,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.26 vs. limit=12.0
2023-11-24 16:25:08,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2905720.0, ans=0.125
2023-11-24 16:25:16,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2905786.6666666665, ans=0.1
2023-11-24 16:25:17,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2905786.6666666665, ans=0.0
2023-11-24 16:25:22,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2905786.6666666665, ans=0.0
2023-11-24 16:25:25,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2905853.3333333335, ans=0.125
2023-11-24 16:25:35,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2905853.3333333335, ans=0.125
2023-11-24 16:25:45,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2905920.0, ans=0.0
2023-11-24 16:25:51,886 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435900
2023-11-24 16:25:52,408 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.64 vs. limit=12.0
2023-11-24 16:26:04,081 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3050, loss[loss=0.06852, simple_loss=0.08417, pruned_loss=0.01658, audio_tagging_loss=0.009858, over 13853.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09133, pruned_loss=0.01309, audio_tagging_loss=0.008789, over 3040325.50 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:26:08,538 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.69 vs. limit=15.0
2023-11-24 16:26:17,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2906120.0, ans=0.07
2023-11-24 16:26:18,211 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.443e+01 9.096e+01 9.825e+01 1.321e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 16:26:24,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2906120.0, ans=0.2
2023-11-24 16:26:30,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2906186.6666666665, ans=0.05
2023-11-24 16:26:39,505 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.86 vs. limit=22.5
2023-11-24 16:26:40,226 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:26:40,870 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.87 vs. limit=22.5
2023-11-24 16:26:51,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2906253.3333333335, ans=0.0
2023-11-24 16:26:53,628 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.47 vs. limit=6.0
2023-11-24 16:26:54,061 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 435950
2023-11-24 16:27:05,798 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3100, loss[loss=0.09154, simple_loss=0.139, pruned_loss=0.01621, audio_tagging_loss=0.005855, over 16153.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09121, pruned_loss=0.01304, audio_tagging_loss=0.008817, over 3030147.51 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:27:09,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2906386.6666666665, ans=0.1
2023-11-24 16:27:13,110 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.40 vs. limit=12.0
2023-11-24 16:27:16,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2906386.6666666665, ans=0.1
2023-11-24 16:27:20,645 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.54 vs. limit=15.0
2023-11-24 16:27:55,674 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436000
2023-11-24 16:28:07,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2906653.3333333335, ans=0.125
2023-11-24 16:28:12,029 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3150, loss[loss=0.06343, simple_loss=0.08264, pruned_loss=0.01256, audio_tagging_loss=0.009559, over 14674.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09121, pruned_loss=0.01307, audio_tagging_loss=0.00891, over 3036242.76 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:28:13,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2906720.0, ans=0.1
2023-11-24 16:28:13,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2906720.0, ans=0.0
2023-11-24 16:28:27,186 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.617e+01 9.268e+01 9.907e+01 1.441e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 16:28:30,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2906786.6666666665, ans=0.05
2023-11-24 16:28:45,258 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2906853.3333333335, ans=0.125
2023-11-24 16:28:46,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2906853.3333333335, ans=0.0
2023-11-24 16:28:52,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2906920.0, ans=0.0
2023-11-24 16:29:01,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436050
2023-11-24 16:29:02,121 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=15.0
2023-11-24 16:29:02,989 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2906986.6666666665, ans=10.0
2023-11-24 16:29:14,527 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3200, loss[loss=0.06913, simple_loss=0.09643, pruned_loss=0.01274, audio_tagging_loss=0.008169, over 15198.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09157, pruned_loss=0.01313, audio_tagging_loss=0.009016, over 3038971.16 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:29:18,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2907053.3333333335, ans=0.1
2023-11-24 16:29:32,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2907120.0, ans=0.0
2023-11-24 16:29:43,284 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.90 vs. limit=10.0
2023-11-24 16:29:52,032 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.07 vs. limit=15.0
2023-11-24 16:30:03,526 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2907320.0, ans=0.1
2023-11-24 16:30:04,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436100
2023-11-24 16:30:08,283 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2907320.0, ans=0.1
2023-11-24 16:30:12,036 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.01 vs. limit=15.0
2023-11-24 16:30:16,188 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3250, loss[loss=0.06675, simple_loss=0.09368, pruned_loss=0.0113, audio_tagging_loss=0.008608, over 15570.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09058, pruned_loss=0.013, audio_tagging_loss=0.009035, over 3042579.63 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:30:22,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2907386.6666666665, ans=0.0
2023-11-24 16:30:24,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2907386.6666666665, ans=0.09899494936611666
2023-11-24 16:30:31,296 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.491e+01 9.014e+01 9.810e+01 1.302e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 16:30:34,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2907453.3333333335, ans=0.2
2023-11-24 16:31:05,842 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436150
2023-11-24 16:31:18,080 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3300, loss[loss=0.06742, simple_loss=0.08919, pruned_loss=0.01556, audio_tagging_loss=0.007267, over 16308.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09094, pruned_loss=0.01309, audio_tagging_loss=0.009055, over 3048409.58 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:31:46,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2907853.3333333335, ans=0.0
2023-11-24 16:32:07,861 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436200
2023-11-24 16:32:14,471 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2907986.6666666665, ans=0.05
2023-11-24 16:32:21,576 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3350, loss[loss=0.07691, simple_loss=0.1021, pruned_loss=0.01831, audio_tagging_loss=0.007554, over 15045.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09158, pruned_loss=0.0132, audio_tagging_loss=0.009054, over 3051881.95 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:32:35,789 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.766e+01 9.367e+01 1.008e+02 1.183e+02, threshold=1.873e+02, percent-clipped=0.0
2023-11-24 16:32:46,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2908186.6666666665, ans=0.2
2023-11-24 16:32:53,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2908186.6666666665, ans=0.0
2023-11-24 16:32:57,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2908253.3333333335, ans=0.125
2023-11-24 16:33:11,408 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436250
2023-11-24 16:33:19,040 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.58 vs. limit=15.0
2023-11-24 16:33:23,269 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3400, loss[loss=0.07274, simple_loss=0.0981, pruned_loss=0.01531, audio_tagging_loss=0.008374, over 15907.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09188, pruned_loss=0.01331, audio_tagging_loss=0.008962, over 3049515.62 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:33:37,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2908453.3333333335, ans=0.125
2023-11-24 16:33:40,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2908453.3333333335, ans=0.2
2023-11-24 16:33:45,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2908453.3333333335, ans=0.125
2023-11-24 16:33:53,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2908520.0, ans=0.125
2023-11-24 16:34:13,219 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436300
2023-11-24 16:34:17,375 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:34:26,315 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3450, loss[loss=0.07238, simple_loss=0.09431, pruned_loss=0.01387, audio_tagging_loss=0.01135, over 14925.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09157, pruned_loss=0.01321, audio_tagging_loss=0.008916, over 3045490.95 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:34:35,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2908720.0, ans=0.0
2023-11-24 16:34:42,908 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.653e+01 9.241e+01 9.942e+01 2.012e+02, threshold=1.848e+02, percent-clipped=1.0
2023-11-24 16:35:10,847 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2908920.0, ans=0.07
2023-11-24 16:35:11,992 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2908920.0, ans=0.0
2023-11-24 16:35:16,732 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436350
2023-11-24 16:35:29,771 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3500, loss[loss=0.06982, simple_loss=0.09525, pruned_loss=0.01673, audio_tagging_loss=0.005462, over 14516.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09052, pruned_loss=0.01306, audio_tagging_loss=0.008866, over 3045319.48 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:35:41,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2909120.0, ans=0.07
2023-11-24 16:35:42,841 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2909120.0, ans=0.125
2023-11-24 16:36:01,031 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:36:09,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2909253.3333333335, ans=0.125
2023-11-24 16:36:13,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2909253.3333333335, ans=0.125
2023-11-24 16:36:20,172 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436400
2023-11-24 16:36:33,044 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3550, loss[loss=0.0649, simple_loss=0.09062, pruned_loss=0.01176, audio_tagging_loss=0.007823, over 15403.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09029, pruned_loss=0.01297, audio_tagging_loss=0.008832, over 3044302.43 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:36:46,233 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.93 vs. limit=22.5
2023-11-24 16:36:49,075 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.431e+01 8.760e+01 9.470e+01 1.011e+02 1.264e+02, threshold=1.894e+02, percent-clipped=0.0
2023-11-24 16:36:55,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2909453.3333333335, ans=0.2
2023-11-24 16:36:58,495 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.67 vs. limit=15.0
2023-11-24 16:37:20,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2909586.6666666665, ans=0.125
2023-11-24 16:37:22,969 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436450
2023-11-24 16:37:35,202 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3600, loss[loss=0.08431, simple_loss=0.1173, pruned_loss=0.0192, audio_tagging_loss=0.006444, over 14570.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09102, pruned_loss=0.0131, audio_tagging_loss=0.0087, over 3044225.61 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:37:42,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2909720.0, ans=0.125
2023-11-24 16:37:42,051 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2909720.0, ans=0.1
2023-11-24 16:38:17,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2909920.0, ans=0.125
2023-11-24 16:38:25,040 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436500
2023-11-24 16:38:37,890 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3650, loss[loss=0.06214, simple_loss=0.0868, pruned_loss=0.00975, audio_tagging_loss=0.008993, over 14374.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09182, pruned_loss=0.0132, audio_tagging_loss=0.008765, over 3044629.47 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:38:54,061 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.295e+01 9.068e+01 9.649e+01 1.086e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 16:38:57,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2910120.0, ans=0.09899494936611666
2023-11-24 16:38:58,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2910120.0, ans=0.125
2023-11-24 16:39:11,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2910186.6666666665, ans=15.0
2023-11-24 16:39:27,851 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436550
2023-11-24 16:39:39,436 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3700, loss[loss=0.06034, simple_loss=0.07595, pruned_loss=0.0156, audio_tagging_loss=0.006775, over 14699.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09204, pruned_loss=0.01323, audio_tagging_loss=0.00866, over 3048528.66 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:40:18,180 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2910586.6666666665, ans=0.125
2023-11-24 16:40:20,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2910586.6666666665, ans=0.125
2023-11-24 16:40:29,924 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436600
2023-11-24 16:40:41,778 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2910720.0, ans=0.2
2023-11-24 16:40:43,384 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3750, loss[loss=0.05474, simple_loss=0.07402, pruned_loss=0.007493, audio_tagging_loss=0.01024, over 14907.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09172, pruned_loss=0.01312, audio_tagging_loss=0.008727, over 3047858.44 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:40:44,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2910720.0, ans=0.1
2023-11-24 16:40:58,421 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2910786.6666666665, ans=0.1
2023-11-24 16:41:01,145 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.745e+01 9.267e+01 9.947e+01 1.281e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 16:41:03,055 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.01 vs. limit=22.5
2023-11-24 16:41:11,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=2910853.3333333335, ans=0.02
2023-11-24 16:41:18,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2910853.3333333335, ans=0.125
2023-11-24 16:41:20,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2910920.0, ans=0.125
2023-11-24 16:41:25,466 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:41:25,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2910920.0, ans=0.1
2023-11-24 16:41:28,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2910920.0, ans=0.0
2023-11-24 16:41:30,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2910920.0, ans=0.2
2023-11-24 16:41:30,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2910920.0, ans=0.125
2023-11-24 16:41:33,340 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436650
2023-11-24 16:41:34,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2910986.6666666665, ans=0.125
2023-11-24 16:41:45,766 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3800, loss[loss=0.06467, simple_loss=0.08484, pruned_loss=0.0102, audio_tagging_loss=0.01204, over 15599.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09121, pruned_loss=0.01299, audio_tagging_loss=0.008853, over 3053720.42 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:41:54,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2911053.3333333335, ans=0.125
2023-11-24 16:42:20,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2911186.6666666665, ans=0.2
2023-11-24 16:42:29,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2911253.3333333335, ans=0.125
2023-11-24 16:42:36,213 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436700
2023-11-24 16:42:47,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2911386.6666666665, ans=0.125
2023-11-24 16:42:48,436 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3850, loss[loss=0.05978, simple_loss=0.08418, pruned_loss=0.008426, audio_tagging_loss=0.009261, over 15260.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09195, pruned_loss=0.013, audio_tagging_loss=0.008798, over 3051384.64 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:42:52,931 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0
2023-11-24 16:42:53,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2911386.6666666665, ans=0.2
2023-11-24 16:42:55,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2911386.6666666665, ans=0.125
2023-11-24 16:43:06,191 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.966e+01 8.511e+01 9.199e+01 9.867e+01 1.160e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 16:43:13,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2911520.0, ans=0.035
2023-11-24 16:43:38,384 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436750
2023-11-24 16:43:50,755 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3900, loss[loss=0.07532, simple_loss=0.09899, pruned_loss=0.01629, audio_tagging_loss=0.009544, over 14474.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09176, pruned_loss=0.01288, audio_tagging_loss=0.008788, over 3051091.13 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:44:41,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436800
2023-11-24 16:44:48,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2911986.6666666665, ans=0.0
2023-11-24 16:44:54,252 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 3950, loss[loss=0.06244, simple_loss=0.08712, pruned_loss=0.01041, audio_tagging_loss=0.008467, over 14899.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09088, pruned_loss=0.01268, audio_tagging_loss=0.008933, over 3047544.47 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:45:08,251 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0
2023-11-24 16:45:11,312 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.639e+01 9.043e+01 9.863e+01 1.208e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 16:45:44,216 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436850
2023-11-24 16:45:56,545 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4000, loss[loss=0.0732, simple_loss=0.1033, pruned_loss=0.01274, audio_tagging_loss=0.008815, over 15627.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09122, pruned_loss=0.01288, audio_tagging_loss=0.009068, over 3044435.38 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:45:57,966 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2912386.6666666665, ans=0.125
2023-11-24 16:46:05,500 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-24 16:46:13,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2912453.3333333335, ans=0.125
2023-11-24 16:46:15,511 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.23 vs. limit=10.0
2023-11-24 16:46:30,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.58 vs. limit=12.0
2023-11-24 16:46:33,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2912586.6666666665, ans=0.125
2023-11-24 16:46:40,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2912586.6666666665, ans=0.1
2023-11-24 16:46:45,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2912653.3333333335, ans=0.1
2023-11-24 16:46:46,484 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436900
2023-11-24 16:46:53,656 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2912653.3333333335, ans=0.1
2023-11-24 16:46:57,948 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.58 vs. limit=10.0
2023-11-24 16:46:58,167 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4050, loss[loss=0.05821, simple_loss=0.07731, pruned_loss=0.01017, audio_tagging_loss=0.009387, over 15157.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09111, pruned_loss=0.01287, audio_tagging_loss=0.009124, over 3044447.86 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:47:00,527 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:47:05,563 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0
2023-11-24 16:47:14,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2912786.6666666665, ans=0.1
2023-11-24 16:47:16,514 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.690e+01 9.276e+01 1.004e+02 1.184e+02, threshold=1.855e+02, percent-clipped=0.0
2023-11-24 16:47:19,020 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2912786.6666666665, ans=0.035
2023-11-24 16:47:38,633 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2912920.0, ans=0.0
2023-11-24 16:47:47,928 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 436950
2023-11-24 16:47:57,771 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.61 vs. limit=15.0
2023-11-24 16:48:01,367 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4100, loss[loss=0.05678, simple_loss=0.07061, pruned_loss=0.01058, audio_tagging_loss=0.0109, over 13592.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09108, pruned_loss=0.01291, audio_tagging_loss=0.009148, over 3038763.59 frames. ], batch size: 52, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:48:19,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2913120.0, ans=0.0
2023-11-24 16:48:35,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2913186.6666666665, ans=0.2
2023-11-24 16:48:51,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437000
2023-11-24 16:49:04,127 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4150, loss[loss=0.05718, simple_loss=0.07448, pruned_loss=0.009975, audio_tagging_loss=0.009959, over 15365.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09068, pruned_loss=0.01287, audio_tagging_loss=0.00904, over 3043729.37 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:49:15,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2913453.3333333335, ans=0.125
2023-11-24 16:49:22,041 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.740e+01 9.379e+01 9.988e+01 1.190e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 16:49:22,684 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.44 vs. limit=12.0
2023-11-24 16:49:47,791 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:49:48,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2913586.6666666665, ans=0.1
2023-11-24 16:49:53,878 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437050
2023-11-24 16:50:03,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2913653.3333333335, ans=0.125
2023-11-24 16:50:05,448 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.07 vs. limit=6.0
2023-11-24 16:50:05,824 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4200, loss[loss=0.07025, simple_loss=0.1035, pruned_loss=0.01244, audio_tagging_loss=0.006076, over 15312.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09122, pruned_loss=0.01296, audio_tagging_loss=0.008882, over 3049554.83 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:50:13,879 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2913720.0, ans=0.125
2023-11-24 16:50:15,061 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:50:15,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2913720.0, ans=0.125
2023-11-24 16:50:34,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2913853.3333333335, ans=0.125
2023-11-24 16:50:37,350 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2913853.3333333335, ans=0.0
2023-11-24 16:50:42,415 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.40 vs. limit=15.0
2023-11-24 16:50:43,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2913920.0, ans=0.2
2023-11-24 16:50:55,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2913986.6666666665, ans=0.125
2023-11-24 16:50:56,132 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437100
2023-11-24 16:51:02,134 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.96 vs. limit=22.5
2023-11-24 16:51:08,510 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4250, loss[loss=0.05727, simple_loss=0.0771, pruned_loss=0.009295, audio_tagging_loss=0.009431, over 14886.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09193, pruned_loss=0.01301, audio_tagging_loss=0.008789, over 3047491.33 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:51:27,405 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.332e+01 8.707e+01 9.618e+01 1.026e+02 1.343e+02, threshold=1.924e+02, percent-clipped=0.0
2023-11-24 16:51:27,684 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:51:32,823 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.88 vs. limit=22.5
2023-11-24 16:51:33,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2914186.6666666665, ans=0.025
2023-11-24 16:51:48,319 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:51:52,322 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.27 vs. limit=15.0
2023-11-24 16:51:56,618 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:51:58,252 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437150
2023-11-24 16:52:10,479 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4300, loss[loss=0.0568, simple_loss=0.08051, pruned_loss=0.01009, audio_tagging_loss=0.006449, over 17032.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09301, pruned_loss=0.01327, audio_tagging_loss=0.008681, over 3052015.50 frames. ], batch size: 64, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:52:22,731 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2914453.3333333335, ans=0.1
2023-11-24 16:52:26,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2914453.3333333335, ans=0.0
2023-11-24 16:52:41,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2914520.0, ans=0.2
2023-11-24 16:52:46,822 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.10 vs. limit=6.0
2023-11-24 16:52:52,083 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.39 vs. limit=12.0
2023-11-24 16:52:55,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2914586.6666666665, ans=0.125
2023-11-24 16:53:00,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437200
2023-11-24 16:53:00,590 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2914653.3333333335, ans=0.0
2023-11-24 16:53:12,543 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4350, loss[loss=0.0622, simple_loss=0.08239, pruned_loss=0.0104, audio_tagging_loss=0.01061, over 15664.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09337, pruned_loss=0.0133, audio_tagging_loss=0.008703, over 3050885.57 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:53:16,398 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2914720.0, ans=0.1
2023-11-24 16:53:32,387 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.693e+01 9.378e+01 1.028e+02 1.311e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 16:53:33,166 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=12.0
2023-11-24 16:53:38,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2914853.3333333335, ans=0.125
2023-11-24 16:53:47,086 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=3.81 vs. limit=6.0
2023-11-24 16:53:56,548 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.43 vs. limit=12.0
2023-11-24 16:53:59,805 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2914920.0, ans=0.125
2023-11-24 16:54:01,895 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437250
2023-11-24 16:54:13,956 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4400, loss[loss=0.07016, simple_loss=0.09094, pruned_loss=0.01346, audio_tagging_loss=0.01123, over 14600.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09248, pruned_loss=0.01322, audio_tagging_loss=0.008741, over 3053590.53 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:54:19,191 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2915053.3333333335, ans=0.125
2023-11-24 16:54:22,096 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.42 vs. limit=22.5
2023-11-24 16:54:23,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2915053.3333333335, ans=0.125
2023-11-24 16:54:23,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2915053.3333333335, ans=0.05
2023-11-24 16:54:36,557 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.68 vs. limit=8.0
2023-11-24 16:54:42,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2915186.6666666665, ans=0.1
2023-11-24 16:54:56,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2915253.3333333335, ans=0.0
2023-11-24 16:55:01,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2915253.3333333335, ans=0.125
2023-11-24 16:55:01,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2915253.3333333335, ans=0.125
2023-11-24 16:55:03,454 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437300
2023-11-24 16:55:10,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.09 vs. limit=15.0
2023-11-24 16:55:16,840 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4450, loss[loss=0.07487, simple_loss=0.09881, pruned_loss=0.01795, audio_tagging_loss=0.007522, over 15999.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09231, pruned_loss=0.01323, audio_tagging_loss=0.008628, over 3053009.85 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:55:19,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2915386.6666666665, ans=0.2
2023-11-24 16:55:20,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2915386.6666666665, ans=0.0
2023-11-24 16:55:36,771 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.463e+01 9.107e+01 9.644e+01 1.325e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 16:55:42,557 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2915520.0, ans=0.0
2023-11-24 16:56:06,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437350
2023-11-24 16:56:11,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2915653.3333333335, ans=0.0
2023-11-24 16:56:11,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2915653.3333333335, ans=0.125
2023-11-24 16:56:15,056 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2915653.3333333335, ans=0.0
2023-11-24 16:56:18,383 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4500, loss[loss=0.05567, simple_loss=0.08162, pruned_loss=0.008269, audio_tagging_loss=0.006591, over 15062.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09254, pruned_loss=0.0133, audio_tagging_loss=0.008542, over 3049802.34 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:56:28,262 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.03 vs. limit=15.0
2023-11-24 16:56:35,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2915786.6666666665, ans=0.1
2023-11-24 16:56:49,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2915853.3333333335, ans=0.1
2023-11-24 16:56:56,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2915920.0, ans=0.04949747468305833
2023-11-24 16:57:07,664 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437400
2023-11-24 16:57:20,275 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4550, loss[loss=0.07829, simple_loss=0.112, pruned_loss=0.01446, audio_tagging_loss=0.007814, over 15202.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09191, pruned_loss=0.01322, audio_tagging_loss=0.008515, over 3053193.53 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 4.0
2023-11-24 16:57:31,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2916120.0, ans=0.125
2023-11-24 16:57:32,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2916120.0, ans=0.0
2023-11-24 16:57:40,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2916120.0, ans=0.125
2023-11-24 16:57:43,343 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.438e+01 8.634e+01 9.216e+01 9.790e+01 1.251e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 16:57:52,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2916186.6666666665, ans=0.125
2023-11-24 16:58:06,738 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:58:09,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2916320.0, ans=0.0
2023-11-24 16:58:10,351 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437450
2023-11-24 16:58:20,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2916320.0, ans=0.125
2023-11-24 16:58:23,348 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4600, loss[loss=0.04295, simple_loss=0.04959, pruned_loss=0.007852, audio_tagging_loss=0.0103, over 14432.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09136, pruned_loss=0.01314, audio_tagging_loss=0.00871, over 3049429.38 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:59:12,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437500
2023-11-24 16:59:25,196 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4650, loss[loss=0.0687, simple_loss=0.1005, pruned_loss=0.0136, audio_tagging_loss=0.004862, over 15163.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09124, pruned_loss=0.01304, audio_tagging_loss=0.008849, over 3047283.72 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:59:31,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2916720.0, ans=0.0
2023-11-24 16:59:31,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2916720.0, ans=0.0
2023-11-24 16:59:34,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2916720.0, ans=0.0
2023-11-24 16:59:42,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2916786.6666666665, ans=0.2
2023-11-24 16:59:44,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2916786.6666666665, ans=0.1
2023-11-24 16:59:46,877 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.291e+01 9.050e+01 9.890e+01 1.291e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 16:59:59,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2916853.3333333335, ans=0.125
2023-11-24 17:00:01,918 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.43 vs. limit=15.0
2023-11-24 17:00:14,693 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437550
2023-11-24 17:00:17,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2916986.6666666665, ans=0.1
2023-11-24 17:00:27,360 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4700, loss[loss=0.0633, simple_loss=0.08611, pruned_loss=0.009201, audio_tagging_loss=0.01105, over 15697.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09075, pruned_loss=0.01301, audio_tagging_loss=0.008961, over 3052450.74 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:00:51,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2917186.6666666665, ans=0.125
2023-11-24 17:01:06,879 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.04 vs. limit=12.0
2023-11-24 17:01:16,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2917320.0, ans=0.0
2023-11-24 17:01:17,423 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437600
2023-11-24 17:01:25,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2917320.0, ans=0.0
2023-11-24 17:01:30,268 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4750, loss[loss=0.05291, simple_loss=0.06699, pruned_loss=0.01069, audio_tagging_loss=0.008722, over 13459.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09019, pruned_loss=0.01296, audio_tagging_loss=0.009016, over 3052872.95 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:01:30,477 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:01:52,692 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.496e+01 8.751e+01 9.417e+01 1.046e+02 1.255e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-24 17:02:15,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2917586.6666666665, ans=0.0
2023-11-24 17:02:17,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2917586.6666666665, ans=0.0
2023-11-24 17:02:20,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437650
2023-11-24 17:02:28,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2917653.3333333335, ans=0.0
2023-11-24 17:02:31,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2917720.0, ans=0.1
2023-11-24 17:02:32,320 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4800, loss[loss=0.06688, simple_loss=0.09037, pruned_loss=0.01118, audio_tagging_loss=0.01051, over 15340.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.08975, pruned_loss=0.01291, audio_tagging_loss=0.009118, over 3046902.11 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:02:32,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2917720.0, ans=0.0
2023-11-24 17:02:36,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2917720.0, ans=0.0
2023-11-24 17:02:53,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=14.42 vs. limit=15.0
2023-11-24 17:02:56,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten.whitening_limit, batch_count=2917853.3333333335, ans=22.5
2023-11-24 17:03:02,957 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.40 vs. limit=12.0
2023-11-24 17:03:21,875 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437700
2023-11-24 17:03:23,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2917986.6666666665, ans=0.125
2023-11-24 17:03:34,100 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4850, loss[loss=0.05627, simple_loss=0.07014, pruned_loss=0.009809, audio_tagging_loss=0.01139, over 14955.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.08992, pruned_loss=0.01304, audio_tagging_loss=0.009256, over 3050438.16 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:03:39,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2918053.3333333335, ans=0.0
2023-11-24 17:03:58,340 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.659e+01 9.301e+01 9.915e+01 1.482e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 17:04:23,219 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:04:24,707 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437750
2023-11-24 17:04:36,848 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4900, loss[loss=0.04616, simple_loss=0.0602, pruned_loss=0.006736, audio_tagging_loss=0.009322, over 14464.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09066, pruned_loss=0.01294, audio_tagging_loss=0.009085, over 3043435.27 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:04:43,990 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.12 vs. limit=22.5
2023-11-24 17:04:52,185 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2918453.3333333335, ans=0.1
2023-11-24 17:04:57,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2918453.3333333335, ans=0.1
2023-11-24 17:05:12,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2918520.0, ans=0.0
2023-11-24 17:05:17,544 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2918586.6666666665, ans=0.125
2023-11-24 17:05:19,161 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.03 vs. limit=15.0
2023-11-24 17:05:23,047 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2918586.6666666665, ans=0.1
2023-11-24 17:05:27,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437800
2023-11-24 17:05:33,688 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.13 vs. limit=15.0
2023-11-24 17:05:39,980 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 4950, loss[loss=0.07712, simple_loss=0.09681, pruned_loss=0.01981, audio_tagging_loss=0.008909, over 15450.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09023, pruned_loss=0.01293, audio_tagging_loss=0.009007, over 3043462.96 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:05:47,609 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.58 vs. limit=22.5
2023-11-24 17:06:04,042 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.424e+01 9.084e+01 9.605e+01 1.394e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 17:06:15,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2918853.3333333335, ans=0.0
2023-11-24 17:06:18,856 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=15.0
2023-11-24 17:06:30,240 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437850
2023-11-24 17:06:42,630 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5000, loss[loss=0.08563, simple_loss=0.1163, pruned_loss=0.01933, audio_tagging_loss=0.008147, over 14748.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09132, pruned_loss=0.01314, audio_tagging_loss=0.008928, over 3036872.86 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:06:52,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2919053.3333333335, ans=0.0
2023-11-24 17:07:00,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2919120.0, ans=0.125
2023-11-24 17:07:08,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2919186.6666666665, ans=0.125
2023-11-24 17:07:08,281 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:07:17,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2919186.6666666665, ans=0.07
2023-11-24 17:07:18,771 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2919253.3333333335, ans=0.125
2023-11-24 17:07:21,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-24 17:07:32,496 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437900
2023-11-24 17:07:45,494 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5050, loss[loss=0.06819, simple_loss=0.09777, pruned_loss=0.01288, audio_tagging_loss=0.006426, over 15720.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09131, pruned_loss=0.01324, audio_tagging_loss=0.00881, over 3042941.79 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:07:49,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2919386.6666666665, ans=0.1
2023-11-24 17:08:08,402 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.782e+01 8.595e+01 9.107e+01 9.818e+01 1.374e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 17:08:13,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2919520.0, ans=0.07
2023-11-24 17:08:35,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 437950
2023-11-24 17:08:40,811 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2919653.3333333335, ans=0.025
2023-11-24 17:08:47,802 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5100, loss[loss=0.06742, simple_loss=0.09098, pruned_loss=0.01207, audio_tagging_loss=0.009856, over 14258.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.08994, pruned_loss=0.01295, audio_tagging_loss=0.008884, over 3039289.63 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:08:48,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2919720.0, ans=0.125
2023-11-24 17:09:11,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2919853.3333333335, ans=0.2
2023-11-24 17:09:29,271 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2919920.0, ans=0.04949747468305833
2023-11-24 17:09:31,982 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.46 vs. limit=22.5
2023-11-24 17:09:37,345 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438000
2023-11-24 17:09:49,265 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5150, loss[loss=0.0786, simple_loss=0.09755, pruned_loss=0.01826, audio_tagging_loss=0.01157, over 13913.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09157, pruned_loss=0.01328, audio_tagging_loss=0.008849, over 3039955.75 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:10:13,399 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.605e+01 9.201e+01 1.005e+02 1.322e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 17:10:23,295 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.60 vs. limit=15.0
2023-11-24 17:10:39,116 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438050
2023-11-24 17:10:52,003 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5200, loss[loss=0.07226, simple_loss=0.1011, pruned_loss=0.0152, audio_tagging_loss=0.006519, over 16183.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09188, pruned_loss=0.01333, audio_tagging_loss=0.00878, over 3038734.39 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:11:00,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2920386.6666666665, ans=0.0
2023-11-24 17:11:06,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2920453.3333333335, ans=0.125
2023-11-24 17:11:29,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2920586.6666666665, ans=0.0
2023-11-24 17:11:33,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2920586.6666666665, ans=0.125
2023-11-24 17:11:35,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2920586.6666666665, ans=0.125
2023-11-24 17:11:42,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438100
2023-11-24 17:11:46,284 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2920653.3333333335, ans=0.125
2023-11-24 17:11:50,330 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.54 vs. limit=22.5
2023-11-24 17:11:55,030 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5250, loss[loss=0.07894, simple_loss=0.1154, pruned_loss=0.01376, audio_tagging_loss=0.007458, over 16075.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09187, pruned_loss=0.01319, audio_tagging_loss=0.00872, over 3046942.35 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:12:05,032 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.14 vs. limit=15.0
2023-11-24 17:12:07,294 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2920786.6666666665, ans=0.125
2023-11-24 17:12:11,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2920786.6666666665, ans=0.2
2023-11-24 17:12:17,093 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:12:17,929 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.449e+01 8.931e+01 9.765e+01 1.225e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 17:12:45,680 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438150
2023-11-24 17:12:57,286 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5300, loss[loss=0.07204, simple_loss=0.09634, pruned_loss=0.0145, audio_tagging_loss=0.00937, over 14476.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09205, pruned_loss=0.01309, audio_tagging_loss=0.008636, over 3049861.31 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:13:39,738 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=12.0
2023-11-24 17:13:47,409 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438200
2023-11-24 17:13:48,085 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.63 vs. limit=15.0
2023-11-24 17:14:00,116 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5350, loss[loss=0.07206, simple_loss=0.09819, pruned_loss=0.01406, audio_tagging_loss=0.008899, over 13501.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.0917, pruned_loss=0.013, audio_tagging_loss=0.008726, over 3047288.52 frames. ], batch size: 52, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:14:07,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2921386.6666666665, ans=0.125
2023-11-24 17:14:13,994 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.04 vs. limit=15.0
2023-11-24 17:14:24,403 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.013e+01 8.487e+01 9.193e+01 9.991e+01 1.472e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 17:14:31,973 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2921520.0, ans=0.125
2023-11-24 17:14:34,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2921520.0, ans=0.5
2023-11-24 17:14:36,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2921586.6666666665, ans=0.0
2023-11-24 17:14:41,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2921586.6666666665, ans=0.0
2023-11-24 17:14:49,944 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438250
2023-11-24 17:15:03,523 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5400, loss[loss=0.06712, simple_loss=0.09128, pruned_loss=0.01121, audio_tagging_loss=0.01028, over 15377.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09121, pruned_loss=0.01307, audio_tagging_loss=0.008762, over 3051570.53 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:15:05,568 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.44 vs. limit=15.0
2023-11-24 17:15:09,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2921720.0, ans=0.05
2023-11-24 17:15:12,498 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.13 vs. limit=15.0
2023-11-24 17:15:31,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2921853.3333333335, ans=0.0
2023-11-24 17:15:32,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2921853.3333333335, ans=0.125
2023-11-24 17:15:33,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2921853.3333333335, ans=0.125
2023-11-24 17:15:53,237 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438300
2023-11-24 17:15:59,309 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2921986.6666666665, ans=0.0
2023-11-24 17:16:00,418 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2921986.6666666665, ans=0.0
2023-11-24 17:16:04,903 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5450, loss[loss=0.05601, simple_loss=0.07181, pruned_loss=0.009547, audio_tagging_loss=0.01056, over 16167.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.0912, pruned_loss=0.01313, audio_tagging_loss=0.008764, over 3057432.66 frames. ], batch size: 63, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:16:05,201 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2922053.3333333335, ans=0.0
2023-11-24 17:16:28,926 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.393e+01 9.258e+01 1.002e+02 1.486e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 17:16:45,436 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.15 vs. limit=10.0
2023-11-24 17:16:56,002 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438350
2023-11-24 17:17:01,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2922320.0, ans=0.2
2023-11-24 17:17:02,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2922320.0, ans=0.2
2023-11-24 17:17:04,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.15 vs. limit=12.0
2023-11-24 17:17:08,719 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5500, loss[loss=0.06249, simple_loss=0.08218, pruned_loss=0.01241, audio_tagging_loss=0.008986, over 15453.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09141, pruned_loss=0.01315, audio_tagging_loss=0.008883, over 3056267.62 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:17:14,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2922386.6666666665, ans=0.125
2023-11-24 17:17:42,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2922520.0, ans=0.125
2023-11-24 17:17:58,583 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438400
2023-11-24 17:18:06,935 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.17 vs. limit=15.0
2023-11-24 17:18:09,001 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:18:11,635 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5550, loss[loss=0.05046, simple_loss=0.07192, pruned_loss=0.007561, audio_tagging_loss=0.006942, over 15047.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09059, pruned_loss=0.01293, audio_tagging_loss=0.008918, over 3048162.96 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:18:34,539 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.462e+01 8.681e+01 9.449e+01 1.002e+02 1.118e+02, threshold=1.890e+02, percent-clipped=0.0
2023-11-24 17:18:54,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2922920.0, ans=0.0
2023-11-24 17:19:01,978 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438450
2023-11-24 17:19:04,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2922986.6666666665, ans=0.09899494936611666
2023-11-24 17:19:10,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2922986.6666666665, ans=0.125
2023-11-24 17:19:13,616 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5600, loss[loss=0.06983, simple_loss=0.09875, pruned_loss=0.01135, audio_tagging_loss=0.009097, over 15025.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09106, pruned_loss=0.01301, audio_tagging_loss=0.008958, over 3051016.95 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:19:22,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2923053.3333333335, ans=0.125
2023-11-24 17:19:22,420 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.67 vs. limit=12.0
2023-11-24 17:19:45,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2923186.6666666665, ans=0.0
2023-11-24 17:19:56,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2923253.3333333335, ans=0.125
2023-11-24 17:19:57,408 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 17:20:01,395 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:20:03,543 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438500
2023-11-24 17:20:13,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2923320.0, ans=0.125
2023-11-24 17:20:15,353 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5650, loss[loss=0.08436, simple_loss=0.1121, pruned_loss=0.02091, audio_tagging_loss=0.007385, over 15165.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09225, pruned_loss=0.0132, audio_tagging_loss=0.009041, over 3057086.76 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:20:15,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2923386.6666666665, ans=0.2
2023-11-24 17:20:37,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2923453.3333333335, ans=0.0
2023-11-24 17:20:39,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2923453.3333333335, ans=0.0
2023-11-24 17:20:40,185 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.503e+01 9.097e+01 9.762e+01 1.252e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 17:20:49,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2923520.0, ans=0.1
2023-11-24 17:20:53,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2923586.6666666665, ans=0.125
2023-11-24 17:20:58,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2923586.6666666665, ans=0.125
2023-11-24 17:21:00,984 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.98 vs. limit=15.0
2023-11-24 17:21:05,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438550
2023-11-24 17:21:18,236 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5700, loss[loss=0.06582, simple_loss=0.0896, pruned_loss=0.01265, audio_tagging_loss=0.008368, over 16019.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.0904, pruned_loss=0.01283, audio_tagging_loss=0.009171, over 3057296.61 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:21:21,402 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.45 vs. limit=15.0
2023-11-24 17:21:31,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2923786.6666666665, ans=0.125
2023-11-24 17:21:42,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2923853.3333333335, ans=0.0
2023-11-24 17:21:44,154 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.67 vs. limit=15.0
2023-11-24 17:21:49,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2923853.3333333335, ans=0.125
2023-11-24 17:21:52,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2923853.3333333335, ans=0.125
2023-11-24 17:21:55,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.98 vs. limit=15.0
2023-11-24 17:22:00,600 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.94 vs. limit=22.5
2023-11-24 17:22:01,572 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.38 vs. limit=15.0
2023-11-24 17:22:07,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2923986.6666666665, ans=0.0
2023-11-24 17:22:08,407 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438600
2023-11-24 17:22:15,610 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2923986.6666666665, ans=0.1
2023-11-24 17:22:21,773 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5750, loss[loss=0.06956, simple_loss=0.1026, pruned_loss=0.0114, audio_tagging_loss=0.006869, over 15097.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09075, pruned_loss=0.01294, audio_tagging_loss=0.009059, over 3051167.85 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:22:25,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2924053.3333333335, ans=0.2
2023-11-24 17:22:28,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2924053.3333333335, ans=0.0
2023-11-24 17:22:44,763 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.490e+01 8.615e+01 9.261e+01 9.857e+01 1.309e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 17:23:11,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438650
2023-11-24 17:23:12,596 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2924320.0, ans=0.025
2023-11-24 17:23:22,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2924386.6666666665, ans=0.0
2023-11-24 17:23:23,079 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5800, loss[loss=0.05887, simple_loss=0.07725, pruned_loss=0.009785, audio_tagging_loss=0.01046, over 14041.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.0903, pruned_loss=0.01285, audio_tagging_loss=0.008968, over 3056866.42 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:23:37,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2924453.3333333335, ans=0.125
2023-11-24 17:23:38,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2924453.3333333335, ans=0.2
2023-11-24 17:23:42,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2924453.3333333335, ans=0.125
2023-11-24 17:23:48,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2924520.0, ans=0.125
2023-11-24 17:24:12,463 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438700
2023-11-24 17:24:24,481 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2924720.0, ans=0.125
2023-11-24 17:24:24,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2924720.0, ans=0.125
2023-11-24 17:24:25,407 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5850, loss[loss=0.08039, simple_loss=0.1104, pruned_loss=0.01826, audio_tagging_loss=0.006899, over 15799.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09047, pruned_loss=0.01291, audio_tagging_loss=0.00887, over 3055099.10 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:24:40,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2924786.6666666665, ans=0.0
2023-11-24 17:24:50,114 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.501e+01 8.938e+01 9.729e+01 1.237e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 17:24:50,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2924853.3333333335, ans=0.0
2023-11-24 17:24:55,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2924853.3333333335, ans=0.125
2023-11-24 17:24:57,568 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2924853.3333333335, ans=0.2
2023-11-24 17:25:01,715 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2924920.0, ans=0.2
2023-11-24 17:25:02,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2924920.0, ans=0.125
2023-11-24 17:25:15,069 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438750
2023-11-24 17:25:23,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2924986.6666666665, ans=0.0
2023-11-24 17:25:27,317 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5900, loss[loss=0.05469, simple_loss=0.07254, pruned_loss=0.01199, audio_tagging_loss=0.006433, over 15284.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09027, pruned_loss=0.01292, audio_tagging_loss=0.008841, over 3050454.51 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:25:31,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2925053.3333333335, ans=0.125
2023-11-24 17:25:32,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2925053.3333333335, ans=0.0
2023-11-24 17:25:57,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2925186.6666666665, ans=0.125
2023-11-24 17:26:16,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438800
2023-11-24 17:26:28,399 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2925386.6666666665, ans=0.125
2023-11-24 17:26:29,245 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 5950, loss[loss=0.05342, simple_loss=0.06775, pruned_loss=0.01225, audio_tagging_loss=0.007298, over 14093.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09112, pruned_loss=0.01305, audio_tagging_loss=0.008699, over 3053997.17 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:26:39,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2925386.6666666665, ans=0.09899494936611666
2023-11-24 17:26:40,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2925453.3333333335, ans=0.0
2023-11-24 17:26:54,496 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.448e+01 8.647e+01 9.220e+01 9.867e+01 1.210e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 17:27:01,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2925520.0, ans=0.125
2023-11-24 17:27:04,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2925520.0, ans=0.125
2023-11-24 17:27:18,850 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438850
2023-11-24 17:27:31,679 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6000, loss[loss=0.05109, simple_loss=0.05971, pruned_loss=0.009156, audio_tagging_loss=0.01208, over 14637.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09095, pruned_loss=0.01316, audio_tagging_loss=0.008691, over 3049728.67 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:27:31,680 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 17:28:14,005 INFO [train_asr.py:1253] (2/4) Epoch 37, validation: loss=0.05829, simple_loss=0.05083, pruned_loss=0.00526, audio_tagging_loss=0.02761, over 4681554.00 frames. 
2023-11-24 17:28:14,006 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 17:28:41,522 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.90 vs. limit=15.0
2023-11-24 17:28:44,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2925853.3333333335, ans=0.125
2023-11-24 17:28:46,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2925853.3333333335, ans=0.125
2023-11-24 17:28:49,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2925853.3333333335, ans=0.125
2023-11-24 17:28:59,038 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 17:28:59,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2925920.0, ans=0.07
2023-11-24 17:29:01,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2925920.0, ans=10.0
2023-11-24 17:29:03,771 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438900
2023-11-24 17:29:16,162 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6050, loss[loss=0.06716, simple_loss=0.09488, pruned_loss=0.01148, audio_tagging_loss=0.008242, over 15778.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.08983, pruned_loss=0.01306, audio_tagging_loss=0.008797, over 3051286.94 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:29:21,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2926053.3333333335, ans=0.125
2023-11-24 17:29:21,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2926053.3333333335, ans=0.0
2023-11-24 17:29:41,491 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.423e+01 8.485e+01 9.097e+01 9.869e+01 1.305e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 17:29:48,798 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2926186.6666666665, ans=0.125
2023-11-24 17:29:57,169 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2926253.3333333335, ans=0.0
2023-11-24 17:29:59,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2926253.3333333335, ans=15.0
2023-11-24 17:30:06,407 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 438950
2023-11-24 17:30:13,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2926320.0, ans=0.0
2023-11-24 17:30:17,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2926386.6666666665, ans=0.0
2023-11-24 17:30:18,679 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6100, loss[loss=0.04635, simple_loss=0.0621, pruned_loss=0.007124, audio_tagging_loss=0.008176, over 15182.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09017, pruned_loss=0.01302, audio_tagging_loss=0.008847, over 3052925.36 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:30:24,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2926386.6666666665, ans=0.2
2023-11-24 17:30:37,813 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.67 vs. limit=15.0
2023-11-24 17:30:45,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2926520.0, ans=0.125
2023-11-24 17:30:48,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2926520.0, ans=0.125
2023-11-24 17:31:08,377 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439000
2023-11-24 17:31:15,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2926653.3333333335, ans=0.2
2023-11-24 17:31:19,565 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.34 vs. limit=6.0
2023-11-24 17:31:21,178 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6150, loss[loss=0.08871, simple_loss=0.1184, pruned_loss=0.02083, audio_tagging_loss=0.00868, over 15767.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09011, pruned_loss=0.013, audio_tagging_loss=0.008886, over 3053518.51 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:31:46,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.702e+01 9.525e+01 1.018e+02 1.166e+02, threshold=1.905e+02, percent-clipped=0.0
2023-11-24 17:31:52,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2926853.3333333335, ans=0.0
2023-11-24 17:31:55,628 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.02 vs. limit=10.0
2023-11-24 17:31:57,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2926920.0, ans=0.0
2023-11-24 17:32:09,086 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.48 vs. limit=15.0
2023-11-24 17:32:10,845 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439050
2023-11-24 17:32:12,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2926986.6666666665, ans=0.0
2023-11-24 17:32:15,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2926986.6666666665, ans=0.0
2023-11-24 17:32:22,653 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6200, loss[loss=0.08587, simple_loss=0.1226, pruned_loss=0.01797, audio_tagging_loss=0.006594, over 14727.00 frames. ], tot_loss[loss=0.06657, simple_loss=0.08972, pruned_loss=0.0128, audio_tagging_loss=0.008915, over 3047741.60 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:32:38,583 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2927120.0, ans=0.125
2023-11-24 17:32:49,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2927186.6666666665, ans=0.125
2023-11-24 17:32:57,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2927186.6666666665, ans=0.04949747468305833
2023-11-24 17:33:12,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439100
2023-11-24 17:33:25,905 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6250, loss[loss=0.07156, simple_loss=0.09905, pruned_loss=0.01118, audio_tagging_loss=0.01086, over 14548.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09026, pruned_loss=0.01297, audio_tagging_loss=0.009035, over 3042224.60 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:33:28,447 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2927386.6666666665, ans=0.125
2023-11-24 17:33:37,784 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2927453.3333333335, ans=0.125
2023-11-24 17:33:50,392 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 8.621e+01 9.212e+01 1.010e+02 1.296e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 17:34:15,145 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439150
2023-11-24 17:34:21,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2927653.3333333335, ans=0.125
2023-11-24 17:34:27,454 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6300, loss[loss=0.07365, simple_loss=0.1009, pruned_loss=0.01533, audio_tagging_loss=0.007872, over 15970.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09095, pruned_loss=0.01314, audio_tagging_loss=0.009065, over 3041950.25 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:34:28,259 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.84 vs. limit=15.0
2023-11-24 17:34:31,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2927720.0, ans=0.125
2023-11-24 17:34:33,739 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2927720.0, ans=10.0
2023-11-24 17:34:34,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2927720.0, ans=0.1
2023-11-24 17:34:40,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2927786.6666666665, ans=0.125
2023-11-24 17:35:05,327 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.32 vs. limit=12.0
2023-11-24 17:35:07,345 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2927920.0, ans=0.0
2023-11-24 17:35:08,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=2927920.0, ans=0.025
2023-11-24 17:35:09,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2927920.0, ans=0.04949747468305833
2023-11-24 17:35:11,559 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2927920.0, ans=0.125
2023-11-24 17:35:17,205 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439200
2023-11-24 17:35:21,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2927986.6666666665, ans=0.1
2023-11-24 17:35:24,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2927986.6666666665, ans=0.125
2023-11-24 17:35:29,266 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6350, loss[loss=0.07262, simple_loss=0.1018, pruned_loss=0.01113, audio_tagging_loss=0.01056, over 14318.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09092, pruned_loss=0.01308, audio_tagging_loss=0.008999, over 3043737.40 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:35:56,864 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.531e+01 9.107e+01 9.829e+01 2.915e+02, threshold=1.821e+02, percent-clipped=1.0
2023-11-24 17:36:18,705 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439250
2023-11-24 17:36:24,193 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2928320.0, ans=0.125
2023-11-24 17:36:27,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2928320.0, ans=0.5
2023-11-24 17:36:31,550 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6400, loss[loss=0.06739, simple_loss=0.09421, pruned_loss=0.01264, audio_tagging_loss=0.007637, over 15435.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.08965, pruned_loss=0.01289, audio_tagging_loss=0.009134, over 3041522.03 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:36:36,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.45 vs. limit=15.0
2023-11-24 17:36:36,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2928386.6666666665, ans=0.0
2023-11-24 17:36:36,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2928386.6666666665, ans=0.0
2023-11-24 17:36:42,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2928453.3333333335, ans=0.125
2023-11-24 17:36:43,906 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2928453.3333333335, ans=0.125
2023-11-24 17:36:50,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2928453.3333333335, ans=0.125
2023-11-24 17:37:02,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2928520.0, ans=0.0
2023-11-24 17:37:14,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2928586.6666666665, ans=0.125
2023-11-24 17:37:15,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2928586.6666666665, ans=0.0
2023-11-24 17:37:21,549 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439300
2023-11-24 17:37:27,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2928653.3333333335, ans=0.1
2023-11-24 17:37:33,848 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6450, loss[loss=0.07429, simple_loss=0.1111, pruned_loss=0.01144, audio_tagging_loss=0.007301, over 15299.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.0905, pruned_loss=0.013, audio_tagging_loss=0.00913, over 3042216.36 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:37:34,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2928720.0, ans=0.0
2023-11-24 17:37:55,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2928786.6666666665, ans=0.0
2023-11-24 17:38:00,168 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.603e+01 9.218e+01 1.012e+02 1.215e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 17:38:00,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2928853.3333333335, ans=0.0
2023-11-24 17:38:01,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2928853.3333333335, ans=0.125
2023-11-24 17:38:02,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2928853.3333333335, ans=0.0
2023-11-24 17:38:02,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2928853.3333333335, ans=0.0
2023-11-24 17:38:16,252 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2928920.0, ans=0.125
2023-11-24 17:38:22,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439350
2023-11-24 17:38:24,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2928986.6666666665, ans=0.125
2023-11-24 17:38:28,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2928986.6666666665, ans=0.125
2023-11-24 17:38:34,342 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6500, loss[loss=0.07223, simple_loss=0.09357, pruned_loss=0.01319, audio_tagging_loss=0.01225, over 14944.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09032, pruned_loss=0.01308, audio_tagging_loss=0.009188, over 3042139.40 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:38:42,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2929053.3333333335, ans=0.1
2023-11-24 17:38:50,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2929120.0, ans=0.1
2023-11-24 17:38:57,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2929120.0, ans=0.0
2023-11-24 17:39:12,870 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.05 vs. limit=10.0
2023-11-24 17:39:17,384 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2929253.3333333335, ans=0.0
2023-11-24 17:39:24,197 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439400
2023-11-24 17:39:33,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2929320.0, ans=0.0
2023-11-24 17:39:36,745 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6550, loss[loss=0.06354, simple_loss=0.08269, pruned_loss=0.01357, audio_tagging_loss=0.00863, over 13754.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09055, pruned_loss=0.01309, audio_tagging_loss=0.009006, over 3037541.75 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:39:37,308 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.19 vs. limit=15.0
2023-11-24 17:39:38,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2929386.6666666665, ans=0.0
2023-11-24 17:39:47,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2929386.6666666665, ans=0.0
2023-11-24 17:39:51,184 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2929453.3333333335, ans=0.0
2023-11-24 17:40:05,708 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.772e+01 9.362e+01 9.895e+01 1.833e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 17:40:26,640 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439450
2023-11-24 17:40:37,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2929653.3333333335, ans=0.1
2023-11-24 17:40:39,591 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6600, loss[loss=0.07449, simple_loss=0.1088, pruned_loss=0.01355, audio_tagging_loss=0.006572, over 16292.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09093, pruned_loss=0.01308, audio_tagging_loss=0.00891, over 3043292.43 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:40:56,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2929786.6666666665, ans=0.125
2023-11-24 17:41:09,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2929853.3333333335, ans=0.2
2023-11-24 17:41:19,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2929920.0, ans=0.0
2023-11-24 17:41:29,662 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439500
2023-11-24 17:41:36,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2929986.6666666665, ans=0.0
2023-11-24 17:41:41,332 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6650, loss[loss=0.06308, simple_loss=0.08084, pruned_loss=0.0119, audio_tagging_loss=0.01076, over 16549.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08971, pruned_loss=0.01283, audio_tagging_loss=0.008984, over 3040157.79 frames. ], batch size: 63, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:41:54,246 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2930120.0, ans=0.0
2023-11-24 17:42:00,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2930120.0, ans=0.0
2023-11-24 17:42:10,022 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.532e+01 9.137e+01 1.001e+02 1.205e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 17:42:21,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2930253.3333333335, ans=0.125
2023-11-24 17:42:22,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2930253.3333333335, ans=0.125
2023-11-24 17:42:31,781 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439550
2023-11-24 17:42:41,240 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.29 vs. limit=22.5
2023-11-24 17:42:44,155 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6700, loss[loss=0.05719, simple_loss=0.07475, pruned_loss=0.009823, audio_tagging_loss=0.009991, over 14946.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09014, pruned_loss=0.01289, audio_tagging_loss=0.008934, over 3041240.68 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:43:20,383 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:43:20,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2930586.6666666665, ans=0.2
2023-11-24 17:43:22,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2930586.6666666665, ans=0.0
2023-11-24 17:43:23,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2930586.6666666665, ans=15.0
2023-11-24 17:43:33,913 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439600
2023-11-24 17:43:35,943 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.88 vs. limit=12.0
2023-11-24 17:43:45,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_na.min_abs, batch_count=2930720.0, ans=0.02
2023-11-24 17:43:46,509 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6750, loss[loss=0.05878, simple_loss=0.07847, pruned_loss=0.01231, audio_tagging_loss=0.007233, over 16311.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.09017, pruned_loss=0.01292, audio_tagging_loss=0.008793, over 3040595.40 frames. ], batch size: 63, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:43:57,711 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2930720.0, ans=0.125
2023-11-24 17:44:02,375 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2930786.6666666665, ans=0.0
2023-11-24 17:44:08,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2930786.6666666665, ans=0.2
2023-11-24 17:44:08,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2930786.6666666665, ans=0.2
2023-11-24 17:44:15,653 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.260e+01 8.901e+01 9.551e+01 1.159e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-24 17:44:37,014 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439650
2023-11-24 17:44:40,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2930986.6666666665, ans=0.0
2023-11-24 17:44:49,447 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6800, loss[loss=0.05529, simple_loss=0.06446, pruned_loss=0.01297, audio_tagging_loss=0.01009, over 15107.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.0901, pruned_loss=0.01295, audio_tagging_loss=0.008695, over 3039588.48 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:45:00,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2931120.0, ans=0.0
2023-11-24 17:45:08,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2931120.0, ans=0.95
2023-11-24 17:45:39,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439700
2023-11-24 17:45:51,918 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6850, loss[loss=0.06287, simple_loss=0.08599, pruned_loss=0.01138, audio_tagging_loss=0.008491, over 15889.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.09046, pruned_loss=0.01293, audio_tagging_loss=0.008602, over 3041555.06 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:45:52,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2931386.6666666665, ans=0.125
2023-11-24 17:45:52,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2931386.6666666665, ans=0.2
2023-11-24 17:46:21,190 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.185e+01 8.495e+01 8.934e+01 9.864e+01 1.145e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-24 17:46:23,899 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2931520.0, ans=0.09899494936611666
2023-11-24 17:46:26,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2931520.0, ans=0.0
2023-11-24 17:46:27,806 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.52 vs. limit=15.0
2023-11-24 17:46:35,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.25 vs. limit=15.0
2023-11-24 17:46:37,729 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.23 vs. limit=15.0
2023-11-24 17:46:40,000 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2931586.6666666665, ans=0.035
2023-11-24 17:46:42,331 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439750
2023-11-24 17:46:54,418 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.52 vs. limit=12.0
2023-11-24 17:46:55,079 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6900, loss[loss=0.07543, simple_loss=0.09915, pruned_loss=0.01656, audio_tagging_loss=0.009296, over 15518.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09106, pruned_loss=0.013, audio_tagging_loss=0.00865, over 3051304.43 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:46:57,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2931720.0, ans=0.125
2023-11-24 17:47:00,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2931720.0, ans=0.125
2023-11-24 17:47:20,991 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2931853.3333333335, ans=0.0
2023-11-24 17:47:25,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2931853.3333333335, ans=0.125
2023-11-24 17:47:28,970 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.73 vs. limit=15.0
2023-11-24 17:47:30,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0
2023-11-24 17:47:40,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2931920.0, ans=0.125
2023-11-24 17:47:42,855 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 17:47:45,309 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439800
2023-11-24 17:47:58,631 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 6950, loss[loss=0.0681, simple_loss=0.08931, pruned_loss=0.01117, audio_tagging_loss=0.01227, over 15514.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09184, pruned_loss=0.01329, audio_tagging_loss=0.008695, over 3049885.95 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:48:05,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2932053.3333333335, ans=0.125
2023-11-24 17:48:18,538 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2932120.0, ans=0.125
2023-11-24 17:48:21,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=2932120.0, ans=0.02
2023-11-24 17:48:24,560 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.95 vs. limit=22.5
2023-11-24 17:48:26,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.54 vs. limit=15.0
2023-11-24 17:48:27,200 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.428e+01 8.692e+01 9.069e+01 1.003e+02 1.264e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 17:48:31,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2932186.6666666665, ans=0.0
2023-11-24 17:48:43,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2932253.3333333335, ans=0.125
2023-11-24 17:48:48,740 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439850
2023-11-24 17:48:49,021 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2932320.0, ans=0.125
2023-11-24 17:48:52,520 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2932320.0, ans=0.125
2023-11-24 17:48:57,002 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2932320.0, ans=0.125
2023-11-24 17:48:58,620 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.74 vs. limit=22.5
2023-11-24 17:49:00,499 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7000, loss[loss=0.07908, simple_loss=0.1126, pruned_loss=0.0147, audio_tagging_loss=0.008098, over 15139.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09131, pruned_loss=0.01322, audio_tagging_loss=0.008746, over 3055776.58 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:49:02,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2932386.6666666665, ans=0.125
2023-11-24 17:49:03,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2932386.6666666665, ans=0.1
2023-11-24 17:49:22,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2932453.3333333335, ans=0.125
2023-11-24 17:49:26,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2932520.0, ans=0.125
2023-11-24 17:49:28,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2932520.0, ans=0.0
2023-11-24 17:49:33,941 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.73 vs. limit=15.0
2023-11-24 17:49:34,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2932520.0, ans=0.125
2023-11-24 17:49:39,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2932586.6666666665, ans=0.1
2023-11-24 17:49:51,092 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439900
2023-11-24 17:49:54,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2932653.3333333335, ans=0.125
2023-11-24 17:50:03,477 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7050, loss[loss=0.06158, simple_loss=0.0832, pruned_loss=0.01103, audio_tagging_loss=0.00895, over 14482.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09007, pruned_loss=0.01298, audio_tagging_loss=0.008915, over 3064419.33 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:50:03,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2932720.0, ans=0.1
2023-11-24 17:50:04,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2932720.0, ans=0.125
2023-11-24 17:50:05,059 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2932720.0, ans=0.0
2023-11-24 17:50:09,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2932720.0, ans=0.1
2023-11-24 17:50:31,453 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.50 vs. limit=10.0
2023-11-24 17:50:31,913 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.508e+01 9.071e+01 9.763e+01 1.227e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 17:50:32,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2932853.3333333335, ans=0.125
2023-11-24 17:50:53,371 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 439950
2023-11-24 17:50:54,620 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2932986.6666666665, ans=0.0
2023-11-24 17:50:54,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2932986.6666666665, ans=0.0
2023-11-24 17:51:03,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2932986.6666666665, ans=0.125
2023-11-24 17:51:05,601 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7100, loss[loss=0.06282, simple_loss=0.0792, pruned_loss=0.01318, audio_tagging_loss=0.01003, over 15114.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09108, pruned_loss=0.01302, audio_tagging_loss=0.008997, over 3065072.78 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:51:07,078 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2933053.3333333335, ans=0.125
2023-11-24 17:51:21,908 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2933120.0, ans=0.125
2023-11-24 17:51:35,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.84 vs. limit=22.5
2023-11-24 17:51:36,476 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.61 vs. limit=6.0
2023-11-24 17:51:54,661 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:51:55,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440000
2023-11-24 17:52:02,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2933320.0, ans=0.125
2023-11-24 17:52:02,977 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.25 vs. limit=15.0
2023-11-24 17:52:12,084 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7150, loss[loss=0.04615, simple_loss=0.05659, pruned_loss=0.006848, audio_tagging_loss=0.011, over 14523.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09127, pruned_loss=0.01295, audio_tagging_loss=0.009027, over 3062491.45 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:52:20,985 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2933386.6666666665, ans=0.0
2023-11-24 17:52:21,195 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2933386.6666666665, ans=0.5
2023-11-24 17:52:22,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2933386.6666666665, ans=0.2
2023-11-24 17:52:40,902 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.362e+01 8.675e+01 9.215e+01 9.966e+01 1.271e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 17:52:42,379 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2933520.0, ans=0.95
2023-11-24 17:52:45,206 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.34 vs. limit=10.0
2023-11-24 17:52:47,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2933520.0, ans=0.1
2023-11-24 17:52:50,591 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:52:53,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2933586.6666666665, ans=0.125
2023-11-24 17:52:59,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2933586.6666666665, ans=0.0
2023-11-24 17:53:01,801 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440050
2023-11-24 17:53:14,518 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7200, loss[loss=0.06513, simple_loss=0.08565, pruned_loss=0.01082, audio_tagging_loss=0.01149, over 14636.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.0908, pruned_loss=0.01284, audio_tagging_loss=0.009145, over 3052420.39 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:53:14,843 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2933720.0, ans=0.125
2023-11-24 17:53:19,707 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2933720.0, ans=0.125
2023-11-24 17:53:25,613 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2933786.6666666665, ans=0.5
2023-11-24 17:53:35,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2933786.6666666665, ans=0.0
2023-11-24 17:53:40,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2933853.3333333335, ans=0.125
2023-11-24 17:53:54,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2933920.0, ans=0.0
2023-11-24 17:53:57,558 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.73 vs. limit=5.0
2023-11-24 17:53:58,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2933920.0, ans=0.0
2023-11-24 17:54:04,234 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440100
2023-11-24 17:54:05,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2933986.6666666665, ans=0.125
2023-11-24 17:54:09,890 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.99 vs. limit=22.5
2023-11-24 17:54:16,761 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7250, loss[loss=0.07593, simple_loss=0.103, pruned_loss=0.01667, audio_tagging_loss=0.007767, over 15347.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09135, pruned_loss=0.01301, audio_tagging_loss=0.009145, over 3056619.47 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:54:26,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.24 vs. limit=22.5
2023-11-24 17:54:41,130 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2934186.6666666665, ans=0.2
2023-11-24 17:54:47,441 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.141e+01 8.488e+01 9.107e+01 9.916e+01 1.399e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 17:54:49,043 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2934186.6666666665, ans=0.0
2023-11-24 17:55:04,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2934253.3333333335, ans=0.125
2023-11-24 17:55:06,877 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440150
2023-11-24 17:55:18,550 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7300, loss[loss=0.05377, simple_loss=0.06362, pruned_loss=0.009623, audio_tagging_loss=0.01233, over 14838.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09088, pruned_loss=0.01282, audio_tagging_loss=0.009065, over 3044231.25 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:55:25,798 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.35 vs. limit=12.0
2023-11-24 17:55:27,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2934386.6666666665, ans=0.2
2023-11-24 17:55:51,639 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.84 vs. limit=15.0
2023-11-24 17:56:09,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440200
2023-11-24 17:56:12,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2934653.3333333335, ans=0.0
2023-11-24 17:56:13,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2934653.3333333335, ans=0.1
2023-11-24 17:56:22,499 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7350, loss[loss=0.056, simple_loss=0.07216, pruned_loss=0.01055, audio_tagging_loss=0.009368, over 15069.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09056, pruned_loss=0.01287, audio_tagging_loss=0.008996, over 3041511.74 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:56:33,752 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.58 vs. limit=15.0
2023-11-24 17:56:35,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-24 17:56:37,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2934786.6666666665, ans=0.025
2023-11-24 17:56:44,334 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2934786.6666666665, ans=0.0
2023-11-24 17:56:51,147 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.582e+01 8.969e+01 9.550e+01 1.265e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-24 17:57:05,901 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.13 vs. limit=15.0
2023-11-24 17:57:11,769 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440250
2023-11-24 17:57:24,029 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7400, loss[loss=0.06197, simple_loss=0.07485, pruned_loss=0.01292, audio_tagging_loss=0.01162, over 14009.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.08991, pruned_loss=0.01287, audio_tagging_loss=0.008904, over 3036458.20 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:58:01,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2935253.3333333335, ans=0.0
2023-11-24 17:58:06,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2935253.3333333335, ans=0.125
2023-11-24 17:58:14,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440300
2023-11-24 17:58:16,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=2935320.0, ans=15.0
2023-11-24 17:58:17,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2935320.0, ans=0.125
2023-11-24 17:58:26,500 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7450, loss[loss=0.05792, simple_loss=0.07831, pruned_loss=0.008981, audio_tagging_loss=0.009784, over 15032.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08969, pruned_loss=0.01278, audio_tagging_loss=0.008821, over 3036889.23 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:58:26,862 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2935386.6666666665, ans=0.125
2023-11-24 17:58:41,070 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2935453.3333333335, ans=0.125
2023-11-24 17:58:56,702 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.405e+01 8.534e+01 9.145e+01 9.758e+01 1.407e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 17:59:16,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440350
2023-11-24 17:59:28,786 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7500, loss[loss=0.0717, simple_loss=0.09625, pruned_loss=0.01587, audio_tagging_loss=0.007706, over 15112.00 frames. ], tot_loss[loss=0.06654, simple_loss=0.08997, pruned_loss=0.01282, audio_tagging_loss=0.008734, over 3041413.88 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:59:31,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2935720.0, ans=0.1
2023-11-24 17:59:39,150 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:59:39,462 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0
2023-11-24 17:59:39,690 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.01 vs. limit=15.0
2023-11-24 17:59:49,097 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2935786.6666666665, ans=0.0
2023-11-24 17:59:55,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2935853.3333333335, ans=0.2
2023-11-24 18:00:03,355 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2935853.3333333335, ans=0.125
2023-11-24 18:00:18,940 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440400
2023-11-24 18:00:31,110 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7550, loss[loss=0.06717, simple_loss=0.09534, pruned_loss=0.01185, audio_tagging_loss=0.007643, over 15107.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09097, pruned_loss=0.01297, audio_tagging_loss=0.008705, over 3047891.73 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:00:32,598 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2936053.3333333335, ans=0.125
2023-11-24 18:00:34,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2936053.3333333335, ans=0.125
2023-11-24 18:01:00,293 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.354e+01 8.524e+01 9.162e+01 9.767e+01 1.233e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 18:01:21,334 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440450
2023-11-24 18:01:32,927 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7600, loss[loss=0.06355, simple_loss=0.08541, pruned_loss=0.01104, audio_tagging_loss=0.009806, over 14244.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.09024, pruned_loss=0.01275, audio_tagging_loss=0.008696, over 3045257.85 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:01:52,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2936453.3333333335, ans=0.2
2023-11-24 18:02:02,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2936520.0, ans=0.0
2023-11-24 18:02:07,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2936520.0, ans=0.125
2023-11-24 18:02:23,249 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440500
2023-11-24 18:02:35,610 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7650, loss[loss=0.08237, simple_loss=0.1126, pruned_loss=0.01784, audio_tagging_loss=0.008233, over 15377.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.09024, pruned_loss=0.01286, audio_tagging_loss=0.008728, over 3041142.61 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:02:46,372 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2936720.0, ans=0.125
2023-11-24 18:03:05,428 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.749e+01 9.173e+01 9.808e+01 1.366e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-24 18:03:08,119 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2936853.3333333335, ans=0.125
2023-11-24 18:03:25,528 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440550
2023-11-24 18:03:32,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2936986.6666666665, ans=0.125
2023-11-24 18:03:33,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2936986.6666666665, ans=0.0
2023-11-24 18:03:37,736 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7700, loss[loss=0.09073, simple_loss=0.1285, pruned_loss=0.01906, audio_tagging_loss=0.00742, over 14783.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08983, pruned_loss=0.01271, audio_tagging_loss=0.008719, over 3031436.22 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:03:43,739 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.27 vs. limit=15.0
2023-11-24 18:03:46,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2937053.3333333335, ans=0.125
2023-11-24 18:03:52,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2937120.0, ans=0.04949747468305833
2023-11-24 18:03:55,069 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2937120.0, ans=10.0
2023-11-24 18:04:00,112 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=22.5
2023-11-24 18:04:03,140 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2937186.6666666665, ans=0.1
2023-11-24 18:04:05,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2937186.6666666665, ans=0.125
2023-11-24 18:04:12,059 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:04:16,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2937253.3333333335, ans=0.1
2023-11-24 18:04:19,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2937253.3333333335, ans=0.125
2023-11-24 18:04:23,884 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2937253.3333333335, ans=0.2
2023-11-24 18:04:26,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2937320.0, ans=0.0
2023-11-24 18:04:27,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440600
2023-11-24 18:04:39,862 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7750, loss[loss=0.06114, simple_loss=0.08812, pruned_loss=0.00972, audio_tagging_loss=0.007357, over 14663.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09078, pruned_loss=0.0128, audio_tagging_loss=0.00874, over 3036095.82 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:04:46,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2937386.6666666665, ans=0.125
2023-11-24 18:04:57,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2937453.3333333335, ans=0.125
2023-11-24 18:04:59,178 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.28 vs. limit=6.0
2023-11-24 18:05:02,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2937453.3333333335, ans=0.125
2023-11-24 18:05:03,321 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.40 vs. limit=15.0
2023-11-24 18:05:10,504 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.466e+01 8.710e+01 9.479e+01 9.926e+01 1.240e+02, threshold=1.896e+02, percent-clipped=0.0
2023-11-24 18:05:12,663 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.64 vs. limit=5.0
2023-11-24 18:05:13,162 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2937520.0, ans=0.1
2023-11-24 18:05:26,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2937586.6666666665, ans=0.0
2023-11-24 18:05:30,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440650
2023-11-24 18:05:42,089 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7800, loss[loss=0.06097, simple_loss=0.07773, pruned_loss=0.01255, audio_tagging_loss=0.009548, over 14559.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09152, pruned_loss=0.01293, audio_tagging_loss=0.008625, over 3039314.36 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:05:53,199 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2937720.0, ans=6.0
2023-11-24 18:05:59,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2937786.6666666665, ans=0.0
2023-11-24 18:06:21,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2937920.0, ans=0.0
2023-11-24 18:06:24,151 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.36 vs. limit=15.0
2023-11-24 18:06:31,174 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2937986.6666666665, ans=0.0
2023-11-24 18:06:32,216 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440700
2023-11-24 18:06:39,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2937986.6666666665, ans=0.04949747468305833
2023-11-24 18:06:45,216 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7850, loss[loss=0.07897, simple_loss=0.1136, pruned_loss=0.01471, audio_tagging_loss=0.007455, over 14996.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.0909, pruned_loss=0.01292, audio_tagging_loss=0.008699, over 3042713.71 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:07:14,183 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.363e+01 8.824e+01 9.632e+01 1.042e+02 1.245e+02, threshold=1.926e+02, percent-clipped=0.0
2023-11-24 18:07:23,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2938253.3333333335, ans=0.0
2023-11-24 18:07:32,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2938253.3333333335, ans=0.2
2023-11-24 18:07:34,013 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-24 18:07:34,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440750
2023-11-24 18:07:47,153 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7900, loss[loss=0.05194, simple_loss=0.07497, pruned_loss=0.006914, audio_tagging_loss=0.007545, over 14443.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09149, pruned_loss=0.01319, audio_tagging_loss=0.008722, over 3044762.18 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:07:52,250 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2938386.6666666665, ans=0.0
2023-11-24 18:07:52,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2938386.6666666665, ans=0.125
2023-11-24 18:08:09,353 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2938453.3333333335, ans=0.0
2023-11-24 18:08:14,673 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2938520.0, ans=0.125
2023-11-24 18:08:28,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2938586.6666666665, ans=0.2
2023-11-24 18:08:29,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2938586.6666666665, ans=0.125
2023-11-24 18:08:36,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2938653.3333333335, ans=0.0
2023-11-24 18:08:37,012 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440800
2023-11-24 18:08:42,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2938653.3333333335, ans=0.0
2023-11-24 18:08:49,000 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 7950, loss[loss=0.08083, simple_loss=0.1055, pruned_loss=0.01911, audio_tagging_loss=0.008955, over 15441.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09106, pruned_loss=0.01303, audio_tagging_loss=0.008873, over 3052842.34 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:08:57,119 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=5.03 vs. limit=15.0
2023-11-24 18:08:58,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2938720.0, ans=0.125
2023-11-24 18:09:00,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2938786.6666666665, ans=0.125
2023-11-24 18:09:04,171 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:09:10,778 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0
2023-11-24 18:09:20,727 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.553e+01 8.522e+01 9.138e+01 9.780e+01 1.184e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 18:09:38,581 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440850
2023-11-24 18:09:38,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2938986.6666666665, ans=0.0
2023-11-24 18:09:50,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2939053.3333333335, ans=0.0
2023-11-24 18:09:51,434 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8000, loss[loss=0.04382, simple_loss=0.05847, pruned_loss=0.004766, audio_tagging_loss=0.009823, over 15787.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09035, pruned_loss=0.01294, audio_tagging_loss=0.008975, over 3052175.85 frames. ], batch size: 62, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:10:19,289 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.44 vs. limit=15.0
2023-11-24 18:10:29,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2939253.3333333335, ans=0.125
2023-11-24 18:10:40,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2939320.0, ans=0.1
2023-11-24 18:10:41,047 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440900
2023-11-24 18:10:41,586 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.27 vs. limit=22.5
2023-11-24 18:10:52,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2939386.6666666665, ans=0.0
2023-11-24 18:10:53,882 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8050, loss[loss=0.09037, simple_loss=0.1281, pruned_loss=0.0201, audio_tagging_loss=0.006196, over 15996.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09078, pruned_loss=0.01306, audio_tagging_loss=0.009024, over 3053741.81 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:11:25,411 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.434e+01 8.492e+01 9.262e+01 1.003e+02 1.241e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 18:11:30,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2939586.6666666665, ans=0.09899494936611666
2023-11-24 18:11:43,203 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 440950
2023-11-24 18:11:51,545 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:11:54,750 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8100, loss[loss=0.07491, simple_loss=0.09861, pruned_loss=0.01759, audio_tagging_loss=0.008017, over 14574.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09163, pruned_loss=0.01334, audio_tagging_loss=0.008948, over 3054010.40 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:11:58,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2939720.0, ans=0.125
2023-11-24 18:12:01,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2939720.0, ans=0.1
2023-11-24 18:12:02,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2939720.0, ans=0.125
2023-11-24 18:12:07,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2939786.6666666665, ans=0.125
2023-11-24 18:12:44,250 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441000
2023-11-24 18:12:57,403 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8150, loss[loss=0.06864, simple_loss=0.09115, pruned_loss=0.01736, audio_tagging_loss=0.005708, over 15400.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09203, pruned_loss=0.01331, audio_tagging_loss=0.008878, over 3047878.91 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:13:00,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2940053.3333333335, ans=0.125
2023-11-24 18:13:06,414 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.40 vs. limit=6.0
2023-11-24 18:13:27,450 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.86 vs. limit=10.0
2023-11-24 18:13:29,019 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.802e+01 9.331e+01 9.892e+01 1.682e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-24 18:13:46,760 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441050
2023-11-24 18:13:49,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2940320.0, ans=0.125
2023-11-24 18:13:49,552 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2940320.0, ans=0.1
2023-11-24 18:13:59,370 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8200, loss[loss=0.07834, simple_loss=0.1078, pruned_loss=0.01735, audio_tagging_loss=0.007109, over 15089.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09148, pruned_loss=0.0132, audio_tagging_loss=0.008785, over 3047339.14 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:13:59,405 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:14:25,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2940520.0, ans=0.0
2023-11-24 18:14:29,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2940520.0, ans=0.1
2023-11-24 18:14:43,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2940586.6666666665, ans=0.125
2023-11-24 18:14:48,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441100
2023-11-24 18:15:01,161 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8250, loss[loss=0.04989, simple_loss=0.05669, pruned_loss=0.01269, audio_tagging_loss=0.008851, over 14846.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09124, pruned_loss=0.01332, audio_tagging_loss=0.008785, over 3041955.87 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:15:13,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2940786.6666666665, ans=0.0
2023-11-24 18:15:33,535 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.363e+01 8.417e+01 9.119e+01 9.803e+01 1.778e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 18:15:40,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2940920.0, ans=0.125
2023-11-24 18:15:43,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2940920.0, ans=0.025
2023-11-24 18:15:50,720 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441150
2023-11-24 18:16:03,892 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8300, loss[loss=0.04643, simple_loss=0.06201, pruned_loss=0.004249, audio_tagging_loss=0.01118, over 15287.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09183, pruned_loss=0.01332, audio_tagging_loss=0.00872, over 3050750.44 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:16:07,799 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2941053.3333333335, ans=0.05
2023-11-24 18:16:09,290 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=15.0
2023-11-24 18:16:11,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2941053.3333333335, ans=0.1
2023-11-24 18:16:46,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2941253.3333333335, ans=0.125
2023-11-24 18:16:54,691 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441200
2023-11-24 18:17:07,480 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8350, loss[loss=0.05982, simple_loss=0.07868, pruned_loss=0.009806, audio_tagging_loss=0.01068, over 15411.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09142, pruned_loss=0.01319, audio_tagging_loss=0.008747, over 3050312.65 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:17:10,533 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.09 vs. limit=15.0
2023-11-24 18:17:14,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2941386.6666666665, ans=0.125
2023-11-24 18:17:30,087 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.92 vs. limit=15.0
2023-11-24 18:17:35,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2941520.0, ans=0.05
2023-11-24 18:17:39,540 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2941520.0, ans=0.0
2023-11-24 18:17:40,295 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.269e+01 8.581e+01 9.277e+01 1.007e+02 1.908e+02, threshold=1.855e+02, percent-clipped=1.0
2023-11-24 18:17:40,508 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2941520.0, ans=0.125
2023-11-24 18:17:57,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441250
2023-11-24 18:17:59,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2941653.3333333335, ans=0.125
2023-11-24 18:18:04,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2941653.3333333335, ans=0.09899494936611666
2023-11-24 18:18:09,319 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8400, loss[loss=0.08284, simple_loss=0.1111, pruned_loss=0.0202, audio_tagging_loss=0.00711, over 15336.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09095, pruned_loss=0.01296, audio_tagging_loss=0.008731, over 3060106.39 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:18:13,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2941720.0, ans=0.0
2023-11-24 18:18:15,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2941720.0, ans=0.125
2023-11-24 18:18:19,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2941720.0, ans=0.05
2023-11-24 18:18:22,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2941786.6666666665, ans=0.0
2023-11-24 18:18:51,355 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.71 vs. limit=15.0
2023-11-24 18:18:51,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2941920.0, ans=0.0
2023-11-24 18:18:55,452 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2941920.0, ans=0.125
2023-11-24 18:18:59,406 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441300
2023-11-24 18:19:11,676 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8450, loss[loss=0.0541, simple_loss=0.07722, pruned_loss=0.008433, audio_tagging_loss=0.007055, over 15495.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09031, pruned_loss=0.01301, audio_tagging_loss=0.008785, over 3048983.38 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:19:42,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2942186.6666666665, ans=0.0
2023-11-24 18:19:43,711 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.476e+01 8.739e+01 9.324e+01 1.024e+02 1.265e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 18:19:48,589 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.47 vs. limit=15.0
2023-11-24 18:20:02,305 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441350
2023-11-24 18:20:03,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2942320.0, ans=0.125
2023-11-24 18:20:10,794 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2942320.0, ans=0.2
2023-11-24 18:20:13,964 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8500, loss[loss=0.05366, simple_loss=0.06541, pruned_loss=0.01076, audio_tagging_loss=0.01019, over 15402.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09142, pruned_loss=0.01317, audio_tagging_loss=0.008796, over 3053691.71 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:20:45,049 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2942520.0, ans=0.125
2023-11-24 18:21:01,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2942586.6666666665, ans=0.2
2023-11-24 18:21:04,358 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2942653.3333333335, ans=0.125
2023-11-24 18:21:05,364 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441400
2023-11-24 18:21:08,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2942653.3333333335, ans=0.125
2023-11-24 18:21:09,973 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.35 vs. limit=12.0
2023-11-24 18:21:17,644 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8550, loss[loss=0.06101, simple_loss=0.08794, pruned_loss=0.008702, audio_tagging_loss=0.008342, over 14696.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09149, pruned_loss=0.01331, audio_tagging_loss=0.008765, over 3054937.24 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:21:21,457 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2942720.0, ans=0.125
2023-11-24 18:21:35,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2942786.6666666665, ans=0.125
2023-11-24 18:21:52,034 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 8.580e+01 9.059e+01 9.638e+01 1.247e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 18:21:55,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2942920.0, ans=0.1
2023-11-24 18:21:56,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.68 vs. limit=15.0
2023-11-24 18:22:04,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2942920.0, ans=0.0
2023-11-24 18:22:07,640 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441450
2023-11-24 18:22:18,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2942986.6666666665, ans=0.125
2023-11-24 18:22:21,156 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8600, loss[loss=0.04069, simple_loss=0.05046, pruned_loss=0.005626, audio_tagging_loss=0.009839, over 14646.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09117, pruned_loss=0.01315, audio_tagging_loss=0.008819, over 3050313.64 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:22:34,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2943120.0, ans=0.125
2023-11-24 18:23:12,939 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441500
2023-11-24 18:23:25,579 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8650, loss[loss=0.06815, simple_loss=0.0844, pruned_loss=0.01536, audio_tagging_loss=0.01059, over 15958.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09033, pruned_loss=0.01307, audio_tagging_loss=0.008935, over 3046529.72 frames. ], batch size: 62, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:23:28,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2943386.6666666665, ans=0.125
2023-11-24 18:23:56,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2943520.0, ans=0.05
2023-11-24 18:23:59,833 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.252e+01 8.635e+01 9.102e+01 9.726e+01 1.317e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 18:24:16,896 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441550
2023-11-24 18:24:21,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2943653.3333333335, ans=0.125
2023-11-24 18:24:22,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2943653.3333333335, ans=0.125
2023-11-24 18:24:25,456 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2943653.3333333335, ans=0.125
2023-11-24 18:24:28,931 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8700, loss[loss=0.1016, simple_loss=0.1292, pruned_loss=0.02832, audio_tagging_loss=0.008729, over 15269.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09157, pruned_loss=0.01328, audio_tagging_loss=0.008968, over 3041824.20 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:24:42,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2943786.6666666665, ans=0.0
2023-11-24 18:25:11,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2943920.0, ans=0.5
2023-11-24 18:25:11,159 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2943920.0, ans=0.125
2023-11-24 18:25:19,304 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441600
2023-11-24 18:25:26,749 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff3.min_abs, batch_count=2943986.6666666665, ans=0.2
2023-11-24 18:25:27,222 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.20 vs. limit=12.0
2023-11-24 18:25:31,362 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8750, loss[loss=0.06766, simple_loss=0.08334, pruned_loss=0.01353, audio_tagging_loss=0.01247, over 15461.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09232, pruned_loss=0.01336, audio_tagging_loss=0.009072, over 3038806.25 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:25:54,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2944120.0, ans=0.125
2023-11-24 18:26:05,776 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.362e+01 8.931e+01 9.559e+01 1.034e+02 1.434e+02, threshold=1.912e+02, percent-clipped=0.0
2023-11-24 18:26:06,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2944186.6666666665, ans=0.125
2023-11-24 18:26:16,286 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.00 vs. limit=15.0
2023-11-24 18:26:22,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441650
2023-11-24 18:26:28,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2944320.0, ans=0.0
2023-11-24 18:26:34,972 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8800, loss[loss=0.06526, simple_loss=0.08331, pruned_loss=0.01323, audio_tagging_loss=0.01038, over 14965.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.0924, pruned_loss=0.01327, audio_tagging_loss=0.009173, over 3036468.74 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:26:46,002 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.96 vs. limit=15.0
2023-11-24 18:26:50,276 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:26:57,729 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:27:12,567 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2944586.6666666665, ans=0.125
2023-11-24 18:27:23,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.07 vs. limit=22.5
2023-11-24 18:27:25,865 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441700
2023-11-24 18:27:26,176 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2944653.3333333335, ans=0.125
2023-11-24 18:27:38,747 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8850, loss[loss=0.05344, simple_loss=0.06091, pruned_loss=0.01272, audio_tagging_loss=0.01026, over 14184.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09084, pruned_loss=0.01314, audio_tagging_loss=0.009222, over 3035120.33 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:27:39,392 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.74 vs. limit=15.0
2023-11-24 18:27:40,137 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2944720.0, ans=0.2
2023-11-24 18:27:42,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2944720.0, ans=0.125
2023-11-24 18:27:49,699 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:27:54,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2944786.6666666665, ans=0.125
2023-11-24 18:28:00,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2944786.6666666665, ans=0.0
2023-11-24 18:28:12,406 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.596e+01 9.206e+01 9.875e+01 1.365e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 18:28:22,098 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.41 vs. limit=22.5
2023-11-24 18:28:28,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441750
2023-11-24 18:28:32,682 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2944986.6666666665, ans=0.125
2023-11-24 18:28:40,888 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8900, loss[loss=0.06641, simple_loss=0.08492, pruned_loss=0.01291, audio_tagging_loss=0.01104, over 14311.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09132, pruned_loss=0.0132, audio_tagging_loss=0.009103, over 3039833.71 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:28:50,329 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2945053.3333333335, ans=0.125
2023-11-24 18:29:02,928 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.32 vs. limit=10.0
2023-11-24 18:29:05,727 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.09 vs. limit=6.0
2023-11-24 18:29:15,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2945186.6666666665, ans=0.015
2023-11-24 18:29:19,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2945253.3333333335, ans=0.2
2023-11-24 18:29:19,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2945253.3333333335, ans=0.125
2023-11-24 18:29:22,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2945253.3333333335, ans=0.125
2023-11-24 18:29:27,014 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.35 vs. limit=15.0
2023-11-24 18:29:32,487 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441800
2023-11-24 18:29:41,684 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2945320.0, ans=0.0
2023-11-24 18:29:46,781 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 8950, loss[loss=0.06917, simple_loss=0.09264, pruned_loss=0.01392, audio_tagging_loss=0.008923, over 14435.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09188, pruned_loss=0.01328, audio_tagging_loss=0.008922, over 3042383.41 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:30:04,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2945453.3333333335, ans=0.125
2023-11-24 18:30:20,104 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.368e+01 8.707e+01 9.494e+01 1.019e+02 1.254e+02, threshold=1.899e+02, percent-clipped=0.0
2023-11-24 18:30:21,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2945520.0, ans=0.0
2023-11-24 18:30:24,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2945586.6666666665, ans=0.0
2023-11-24 18:30:30,080 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2945586.6666666665, ans=0.125
2023-11-24 18:30:37,486 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441850
2023-11-24 18:30:46,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2945653.3333333335, ans=0.0
2023-11-24 18:30:48,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.38 vs. limit=15.0
2023-11-24 18:30:50,102 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9000, loss[loss=0.04925, simple_loss=0.0638, pruned_loss=0.008205, audio_tagging_loss=0.009144, over 14645.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09112, pruned_loss=0.01308, audio_tagging_loss=0.008818, over 3046875.47 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:30:50,102 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 18:31:31,493 INFO [train_asr.py:1253] (2/4) Epoch 37, validation: loss=0.05871, simple_loss=0.05072, pruned_loss=0.005135, audio_tagging_loss=0.02821, over 4681554.00 frames. 
2023-11-24 18:31:31,494 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 18:31:47,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2945786.6666666665, ans=0.125
2023-11-24 18:31:54,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2945786.6666666665, ans=0.1
2023-11-24 18:31:57,359 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.67 vs. limit=15.0
2023-11-24 18:32:04,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.19 vs. limit=12.0
2023-11-24 18:32:06,605 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:32:22,541 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441900
2023-11-24 18:32:31,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2945986.6666666665, ans=0.0
2023-11-24 18:32:34,937 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9050, loss[loss=0.07982, simple_loss=0.1074, pruned_loss=0.01748, audio_tagging_loss=0.008667, over 16896.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09062, pruned_loss=0.01293, audio_tagging_loss=0.008833, over 3045919.40 frames. ], batch size: 62, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:32:48,271 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:32:49,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=2946120.0, ans=0.2
2023-11-24 18:32:52,758 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2946120.0, ans=0.1
2023-11-24 18:33:09,596 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.312e+01 8.396e+01 9.037e+01 9.861e+01 1.283e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 18:33:14,172 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2946253.3333333335, ans=0.0
2023-11-24 18:33:24,532 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 441950
2023-11-24 18:33:25,153 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.85 vs. limit=22.5
2023-11-24 18:33:26,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2946320.0, ans=0.0
2023-11-24 18:33:37,552 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9100, loss[loss=0.06456, simple_loss=0.06182, pruned_loss=0.01941, audio_tagging_loss=0.01425, over 13290.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.09023, pruned_loss=0.01286, audio_tagging_loss=0.008789, over 3042321.60 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:33:54,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2946453.3333333335, ans=0.125
2023-11-24 18:34:10,468 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.58 vs. limit=15.0
2023-11-24 18:34:20,661 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2946586.6666666665, ans=0.125
2023-11-24 18:34:27,579 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442000
2023-11-24 18:34:31,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2946653.3333333335, ans=0.125
2023-11-24 18:34:35,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2946653.3333333335, ans=0.0
2023-11-24 18:34:39,636 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9150, loss[loss=0.06651, simple_loss=0.09292, pruned_loss=0.01218, audio_tagging_loss=0.007872, over 15821.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.0906, pruned_loss=0.01294, audio_tagging_loss=0.008747, over 3040242.60 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:34:46,858 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-24 18:34:47,729 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2946720.0, ans=0.0
2023-11-24 18:34:57,227 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.28 vs. limit=15.0
2023-11-24 18:35:13,234 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:35:14,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2946853.3333333335, ans=0.125
2023-11-24 18:35:15,209 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.510e+01 9.159e+01 9.829e+01 1.251e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 18:35:16,687 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2946920.0, ans=0.0
2023-11-24 18:35:30,166 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442050
2023-11-24 18:35:32,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2946986.6666666665, ans=0.2
2023-11-24 18:35:40,428 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2946986.6666666665, ans=0.125
2023-11-24 18:35:42,497 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9200, loss[loss=0.04758, simple_loss=0.05869, pruned_loss=0.009951, audio_tagging_loss=0.008287, over 13795.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.08983, pruned_loss=0.01288, audio_tagging_loss=0.008824, over 3037376.58 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:36:05,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2947120.0, ans=0.125
2023-11-24 18:36:32,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442100
2023-11-24 18:36:34,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2947320.0, ans=0.0
2023-11-24 18:36:45,618 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9250, loss[loss=0.07568, simple_loss=0.1133, pruned_loss=0.01363, audio_tagging_loss=0.00538, over 14733.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.09006, pruned_loss=0.01288, audio_tagging_loss=0.008842, over 3045609.61 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:37:07,793 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2947453.3333333335, ans=0.125
2023-11-24 18:37:19,954 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.271e+01 8.726e+01 9.340e+01 9.852e+01 1.345e+02, threshold=1.868e+02, percent-clipped=0.0
2023-11-24 18:37:21,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2947586.6666666665, ans=0.0
2023-11-24 18:37:35,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442150
2023-11-24 18:37:38,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2947653.3333333335, ans=0.125
2023-11-24 18:37:47,444 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9300, loss[loss=0.04912, simple_loss=0.06139, pruned_loss=0.006807, audio_tagging_loss=0.01162, over 14056.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08973, pruned_loss=0.01279, audio_tagging_loss=0.008817, over 3037324.15 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:37:50,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2947720.0, ans=0.125
2023-11-24 18:38:00,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2947786.6666666665, ans=0.0
2023-11-24 18:38:37,554 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442200
2023-11-24 18:38:42,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2947986.6666666665, ans=0.0
2023-11-24 18:38:51,060 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9350, loss[loss=0.08451, simple_loss=0.1055, pruned_loss=0.02252, audio_tagging_loss=0.009254, over 14909.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08926, pruned_loss=0.01281, audio_tagging_loss=0.008914, over 3040499.09 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:38:54,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2948053.3333333335, ans=0.125
2023-11-24 18:39:00,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2948053.3333333335, ans=0.125
2023-11-24 18:39:10,247 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2948120.0, ans=0.125
2023-11-24 18:39:22,483 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.78 vs. limit=15.0
2023-11-24 18:39:26,935 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.625e+01 9.145e+01 9.875e+01 1.374e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 18:39:35,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2948253.3333333335, ans=0.125
2023-11-24 18:39:37,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2948253.3333333335, ans=0.2
2023-11-24 18:39:40,955 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442250
2023-11-24 18:39:45,848 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2948320.0, ans=0.125
2023-11-24 18:39:52,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2948386.6666666665, ans=0.0
2023-11-24 18:39:53,308 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9400, loss[loss=0.06467, simple_loss=0.08506, pruned_loss=0.01459, audio_tagging_loss=0.007551, over 14178.00 frames. ], tot_loss[loss=0.06641, simple_loss=0.08919, pruned_loss=0.01284, audio_tagging_loss=0.008978, over 3048306.66 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:40:41,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2948586.6666666665, ans=0.125
2023-11-24 18:40:44,044 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442300
2023-11-24 18:40:53,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2948653.3333333335, ans=0.1
2023-11-24 18:40:54,213 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:40:55,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2948720.0, ans=0.1
2023-11-24 18:40:56,575 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9450, loss[loss=0.07424, simple_loss=0.1001, pruned_loss=0.01511, audio_tagging_loss=0.009088, over 15723.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.08984, pruned_loss=0.01301, audio_tagging_loss=0.009005, over 3054875.24 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:41:01,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2948720.0, ans=0.125
2023-11-24 18:41:03,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2948720.0, ans=0.125
2023-11-24 18:41:15,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2948786.6666666665, ans=0.125
2023-11-24 18:41:24,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2948853.3333333335, ans=0.09899494936611666
2023-11-24 18:41:24,430 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2948853.3333333335, ans=0.125
2023-11-24 18:41:25,990 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2948853.3333333335, ans=0.0
2023-11-24 18:41:33,031 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.255e+01 8.857e+01 9.505e+01 1.052e+02 1.346e+02, threshold=1.901e+02, percent-clipped=0.0
2023-11-24 18:41:43,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2948920.0, ans=0.0
2023-11-24 18:41:47,122 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442350
2023-11-24 18:41:48,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2948986.6666666665, ans=0.125
2023-11-24 18:41:51,363 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2948986.6666666665, ans=0.125
2023-11-24 18:41:52,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2948986.6666666665, ans=0.125
2023-11-24 18:41:57,327 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2948986.6666666665, ans=0.2
2023-11-24 18:41:59,892 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9500, loss[loss=0.04217, simple_loss=0.05627, pruned_loss=0.005875, audio_tagging_loss=0.00816, over 14711.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09009, pruned_loss=0.01307, audio_tagging_loss=0.009057, over 3053900.25 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:42:33,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2949186.6666666665, ans=0.125
2023-11-24 18:42:49,688 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442400
2023-11-24 18:43:02,617 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9550, loss[loss=0.05603, simple_loss=0.07807, pruned_loss=0.008431, audio_tagging_loss=0.00857, over 15088.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08932, pruned_loss=0.01276, audio_tagging_loss=0.009248, over 3049376.17 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:43:05,208 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:43:11,343 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2949386.6666666665, ans=0.125
2023-11-24 18:43:35,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2949520.0, ans=0.05
2023-11-24 18:43:38,731 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.675e+01 9.207e+01 9.929e+01 1.581e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 18:43:43,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2949586.6666666665, ans=0.125
2023-11-24 18:43:46,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2949586.6666666665, ans=0.1
2023-11-24 18:43:52,558 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442450
2023-11-24 18:43:52,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2949653.3333333335, ans=0.09899494936611666
2023-11-24 18:44:02,945 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.14 vs. limit=8.0
2023-11-24 18:44:04,431 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9600, loss[loss=0.07188, simple_loss=0.0992, pruned_loss=0.01154, audio_tagging_loss=0.01074, over 15225.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.08965, pruned_loss=0.01284, audio_tagging_loss=0.009274, over 3045369.32 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:44:10,252 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.99 vs. limit=15.0
2023-11-24 18:44:12,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2949720.0, ans=0.0
2023-11-24 18:44:17,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2949786.6666666665, ans=0.0
2023-11-24 18:44:21,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2949786.6666666665, ans=0.05
2023-11-24 18:44:24,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.03 vs. limit=15.0
2023-11-24 18:44:41,208 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2949920.0, ans=0.0
2023-11-24 18:44:45,008 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0
2023-11-24 18:44:54,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442500
2023-11-24 18:45:07,131 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9650, loss[loss=0.08731, simple_loss=0.1202, pruned_loss=0.01934, audio_tagging_loss=0.007882, over 15572.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.08982, pruned_loss=0.01278, audio_tagging_loss=0.009226, over 3046656.76 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:45:15,607 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.83 vs. limit=22.5
2023-11-24 18:45:23,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2950120.0, ans=0.125
2023-11-24 18:45:26,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2950120.0, ans=0.05
2023-11-24 18:45:37,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2950186.6666666665, ans=0.125
2023-11-24 18:45:43,083 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.493e+01 8.425e+01 9.108e+01 9.681e+01 1.366e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 18:45:57,325 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442550
2023-11-24 18:45:58,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2950320.0, ans=0.125
2023-11-24 18:46:09,222 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9700, loss[loss=0.07014, simple_loss=0.1057, pruned_loss=0.01153, audio_tagging_loss=0.005762, over 14338.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09016, pruned_loss=0.01285, audio_tagging_loss=0.008993, over 3041626.55 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:46:18,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2950386.6666666665, ans=0.2
2023-11-24 18:46:46,763 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2950586.6666666665, ans=0.125
2023-11-24 18:46:56,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2950586.6666666665, ans=0.0
2023-11-24 18:46:57,442 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2950586.6666666665, ans=0.125
2023-11-24 18:46:59,105 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2950653.3333333335, ans=0.125
2023-11-24 18:47:00,182 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442600
2023-11-24 18:47:12,414 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9750, loss[loss=0.06335, simple_loss=0.08876, pruned_loss=0.01021, audio_tagging_loss=0.008756, over 14115.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08985, pruned_loss=0.0127, audio_tagging_loss=0.008963, over 3042265.40 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:47:16,036 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2950720.0, ans=0.125
2023-11-24 18:47:20,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2950720.0, ans=0.1
2023-11-24 18:47:26,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2950786.6666666665, ans=0.2
2023-11-24 18:47:43,069 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.91 vs. limit=15.0
2023-11-24 18:47:48,958 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.947e+01 8.439e+01 9.144e+01 9.970e+01 1.220e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 18:48:02,035 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442650
2023-11-24 18:48:14,352 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9800, loss[loss=0.07334, simple_loss=0.1038, pruned_loss=0.01443, audio_tagging_loss=0.007001, over 15076.00 frames. ], tot_loss[loss=0.06625, simple_loss=0.08938, pruned_loss=0.01259, audio_tagging_loss=0.008971, over 3038424.69 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:48:15,796 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2951053.3333333335, ans=0.125
2023-11-24 18:48:22,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2951053.3333333335, ans=0.0
2023-11-24 18:48:22,919 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.31 vs. limit=15.0
2023-11-24 18:48:32,039 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.92 vs. limit=15.0
2023-11-24 18:48:49,734 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2951253.3333333335, ans=0.0
2023-11-24 18:49:03,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442700
2023-11-24 18:49:09,059 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:49:16,068 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9850, loss[loss=0.05483, simple_loss=0.07415, pruned_loss=0.008699, audio_tagging_loss=0.009057, over 15039.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09146, pruned_loss=0.01296, audio_tagging_loss=0.008835, over 3044611.13 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:49:25,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2951386.6666666665, ans=0.0
2023-11-24 18:49:43,117 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0
2023-11-24 18:49:47,050 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2951520.0, ans=0.125
2023-11-24 18:49:53,629 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.535e+01 9.041e+01 9.752e+01 1.279e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 18:50:05,514 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442750
2023-11-24 18:50:17,726 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9900, loss[loss=0.07477, simple_loss=0.09514, pruned_loss=0.01733, audio_tagging_loss=0.009865, over 14907.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.0922, pruned_loss=0.01291, audio_tagging_loss=0.008789, over 3042897.57 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:50:22,718 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2951720.0, ans=0.035
2023-11-24 18:50:26,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2951720.0, ans=0.125
2023-11-24 18:50:33,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2951786.6666666665, ans=0.0
2023-11-24 18:50:37,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2951786.6666666665, ans=0.125
2023-11-24 18:51:06,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2951986.6666666665, ans=0.1
2023-11-24 18:51:07,622 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442800
2023-11-24 18:51:19,563 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 9950, loss[loss=0.05937, simple_loss=0.07429, pruned_loss=0.01224, audio_tagging_loss=0.009985, over 14944.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09088, pruned_loss=0.01269, audio_tagging_loss=0.008858, over 3045946.78 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:51:21,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2952053.3333333335, ans=0.2
2023-11-24 18:51:22,893 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2952053.3333333335, ans=0.1
2023-11-24 18:51:35,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2952120.0, ans=0.0
2023-11-24 18:51:45,247 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.10 vs. limit=15.0
2023-11-24 18:51:57,276 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.463e+01 9.019e+01 9.662e+01 1.211e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 18:52:03,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2952253.3333333335, ans=0.04949747468305833
2023-11-24 18:52:10,128 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442850
2023-11-24 18:52:15,335 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.12 vs. limit=15.0
2023-11-24 18:52:22,999 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10000, loss[loss=0.07064, simple_loss=0.09397, pruned_loss=0.01485, audio_tagging_loss=0.008806, over 15890.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09136, pruned_loss=0.01297, audio_tagging_loss=0.008774, over 3041457.05 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:52:31,979 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2952386.6666666665, ans=0.125
2023-11-24 18:53:01,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2952586.6666666665, ans=15.0
2023-11-24 18:53:04,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2952586.6666666665, ans=0.09899494936611666
2023-11-24 18:53:11,974 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442900
2023-11-24 18:53:12,541 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.53 vs. limit=15.0
2023-11-24 18:53:24,190 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10050, loss[loss=0.06684, simple_loss=0.09083, pruned_loss=0.0142, audio_tagging_loss=0.007224, over 14441.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09126, pruned_loss=0.01298, audio_tagging_loss=0.008821, over 3044235.89 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:53:40,331 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.28 vs. limit=8.0
2023-11-24 18:53:50,229 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.06 vs. limit=12.0
2023-11-24 18:53:55,340 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2952853.3333333335, ans=0.125
2023-11-24 18:54:00,058 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2952920.0, ans=0.125
2023-11-24 18:54:01,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2952920.0, ans=0.2
2023-11-24 18:54:02,841 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.473e+01 8.442e+01 9.039e+01 9.729e+01 1.134e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 18:54:07,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2952920.0, ans=0.0
2023-11-24 18:54:13,526 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 442950
2023-11-24 18:54:25,232 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10100, loss[loss=0.06478, simple_loss=0.08655, pruned_loss=0.01139, audio_tagging_loss=0.01011, over 15013.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09063, pruned_loss=0.0129, audio_tagging_loss=0.008889, over 3038251.66 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:54:43,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2953120.0, ans=0.125
2023-11-24 18:54:44,804 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2953120.0, ans=0.1
2023-11-24 18:54:51,236 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2953186.6666666665, ans=0.035
2023-11-24 18:54:53,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.44 vs. limit=10.0
2023-11-24 18:54:57,110 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2953186.6666666665, ans=0.125
2023-11-24 18:54:58,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2953186.6666666665, ans=0.0
2023-11-24 18:55:09,413 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.53 vs. limit=15.0
2023-11-24 18:55:14,619 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:55:14,685 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443000
2023-11-24 18:55:28,614 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10150, loss[loss=0.05147, simple_loss=0.06664, pruned_loss=0.009576, audio_tagging_loss=0.008579, over 14406.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09047, pruned_loss=0.01277, audio_tagging_loss=0.008901, over 3044673.51 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:55:43,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2953453.3333333335, ans=0.125
2023-11-24 18:55:53,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.83 vs. limit=15.0
2023-11-24 18:55:56,455 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:56:06,391 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.628e+01 9.087e+01 9.762e+01 1.339e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 18:56:18,462 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443050
2023-11-24 18:56:30,719 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10200, loss[loss=0.05222, simple_loss=0.05891, pruned_loss=0.01029, audio_tagging_loss=0.01247, over 15509.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.08992, pruned_loss=0.01269, audio_tagging_loss=0.009042, over 3048357.33 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:56:47,674 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2953786.6666666665, ans=0.2
2023-11-24 18:56:52,731 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:57:13,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2953920.0, ans=0.0
2023-11-24 18:57:20,394 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443100
2023-11-24 18:57:22,790 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2953986.6666666665, ans=0.125
2023-11-24 18:57:32,195 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10250, loss[loss=0.04514, simple_loss=0.05516, pruned_loss=0.006197, audio_tagging_loss=0.01137, over 15792.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09055, pruned_loss=0.01287, audio_tagging_loss=0.009001, over 3046025.66 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:57:34,977 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2954053.3333333335, ans=0.125
2023-11-24 18:57:41,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2954053.3333333335, ans=0.0
2023-11-24 18:58:10,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2954253.3333333335, ans=0.1
2023-11-24 18:58:11,703 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.337e+01 8.466e+01 9.132e+01 9.893e+01 1.266e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 18:58:12,307 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.43 vs. limit=15.0
2023-11-24 18:58:22,704 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443150
2023-11-24 18:58:35,795 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10300, loss[loss=0.05022, simple_loss=0.06393, pruned_loss=0.005238, audio_tagging_loss=0.01302, over 13930.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.08993, pruned_loss=0.01268, audio_tagging_loss=0.009065, over 3047036.13 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:58:39,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2954386.6666666665, ans=0.1
2023-11-24 18:58:45,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2954386.6666666665, ans=0.125
2023-11-24 18:59:25,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443200
2023-11-24 18:59:39,368 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10350, loss[loss=0.09042, simple_loss=0.1277, pruned_loss=0.02016, audio_tagging_loss=0.006396, over 14127.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09086, pruned_loss=0.01284, audio_tagging_loss=0.009069, over 3035904.04 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:59:44,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2954720.0, ans=0.125
2023-11-24 18:59:58,842 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.07 vs. limit=15.0
2023-11-24 19:00:10,341 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2954853.3333333335, ans=0.125
2023-11-24 19:00:12,607 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2954853.3333333335, ans=0.0
2023-11-24 19:00:15,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2954920.0, ans=0.0
2023-11-24 19:00:16,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2954920.0, ans=0.0
2023-11-24 19:00:17,085 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.601e+01 8.606e+01 9.110e+01 9.960e+01 1.328e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 19:00:28,632 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443250
2023-11-24 19:00:40,314 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10400, loss[loss=0.06069, simple_loss=0.08603, pruned_loss=0.01058, audio_tagging_loss=0.007098, over 15219.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09041, pruned_loss=0.01279, audio_tagging_loss=0.009085, over 3037476.25 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:01:17,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2955253.3333333335, ans=0.1
2023-11-24 19:01:28,115 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:01:30,230 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443300
2023-11-24 19:01:39,554 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.36 vs. limit=15.0
2023-11-24 19:01:40,694 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.75 vs. limit=15.0
2023-11-24 19:01:43,028 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10450, loss[loss=0.06839, simple_loss=0.09889, pruned_loss=0.01152, audio_tagging_loss=0.007423, over 15425.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09116, pruned_loss=0.0129, audio_tagging_loss=0.009024, over 3042224.49 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:01:46,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2955386.6666666665, ans=0.1
2023-11-24 19:02:00,772 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2955453.3333333335, ans=0.125
2023-11-24 19:02:00,817 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2955453.3333333335, ans=0.125
2023-11-24 19:02:22,857 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.090e+01 8.670e+01 9.376e+01 1.004e+02 1.489e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 19:02:24,591 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.83 vs. limit=15.0
2023-11-24 19:02:33,005 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443350
2023-11-24 19:02:37,179 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.58 vs. limit=15.0
2023-11-24 19:02:45,214 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10500, loss[loss=0.0644, simple_loss=0.0811, pruned_loss=0.01319, audio_tagging_loss=0.01066, over 14282.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09119, pruned_loss=0.01273, audio_tagging_loss=0.008902, over 3046755.10 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:02:49,619 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2955720.0, ans=0.2
2023-11-24 19:03:12,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2955853.3333333335, ans=0.125
2023-11-24 19:03:14,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2955853.3333333335, ans=0.125
2023-11-24 19:03:35,019 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443400
2023-11-24 19:03:48,032 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10550, loss[loss=0.06723, simple_loss=0.09101, pruned_loss=0.01273, audio_tagging_loss=0.008995, over 14815.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09105, pruned_loss=0.0128, audio_tagging_loss=0.008777, over 3043057.56 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:04:00,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2956120.0, ans=0.2
2023-11-24 19:04:11,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2956186.6666666665, ans=0.125
2023-11-24 19:04:13,581 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.70 vs. limit=15.0
2023-11-24 19:04:19,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2956186.6666666665, ans=0.125
2023-11-24 19:04:20,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2956186.6666666665, ans=0.0
2023-11-24 19:04:29,168 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.503e+01 9.037e+01 9.861e+01 1.136e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 19:04:30,654 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2956253.3333333335, ans=0.09899494936611666
2023-11-24 19:04:35,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2956253.3333333335, ans=0.125
2023-11-24 19:04:37,540 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443450
2023-11-24 19:04:49,853 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10600, loss[loss=0.0618, simple_loss=0.08134, pruned_loss=0.01241, audio_tagging_loss=0.008713, over 16261.00 frames. ], tot_loss[loss=0.06619, simple_loss=0.08946, pruned_loss=0.01266, audio_tagging_loss=0.008804, over 3040353.83 frames. ], batch size: 62, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:04:54,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2956386.6666666665, ans=0.125
2023-11-24 19:04:58,235 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.70 vs. limit=15.0
2023-11-24 19:05:15,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2956520.0, ans=0.0
2023-11-24 19:05:16,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2956520.0, ans=0.125
2023-11-24 19:05:22,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2956520.0, ans=0.0
2023-11-24 19:05:22,276 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2956520.0, ans=0.125
2023-11-24 19:05:39,982 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443500
2023-11-24 19:05:41,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2956653.3333333335, ans=0.2
2023-11-24 19:05:44,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2956653.3333333335, ans=0.125
2023-11-24 19:05:48,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2956653.3333333335, ans=0.0
2023-11-24 19:05:51,793 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10650, loss[loss=0.0619, simple_loss=0.08294, pruned_loss=0.009636, audio_tagging_loss=0.01079, over 14617.00 frames. ], tot_loss[loss=0.06641, simple_loss=0.09, pruned_loss=0.01271, audio_tagging_loss=0.008699, over 3036613.53 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:05:56,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0
2023-11-24 19:06:06,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2956786.6666666665, ans=0.125
2023-11-24 19:06:06,847 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.42 vs. limit=10.0
2023-11-24 19:06:10,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2956786.6666666665, ans=0.025
2023-11-24 19:06:14,756 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2956786.6666666665, ans=0.0
2023-11-24 19:06:21,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2956853.3333333335, ans=0.0
2023-11-24 19:06:23,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2956853.3333333335, ans=0.125
2023-11-24 19:06:33,393 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.594e+01 8.793e+01 9.512e+01 1.031e+02 1.281e+02, threshold=1.902e+02, percent-clipped=0.0
2023-11-24 19:06:33,688 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2956920.0, ans=0.125
2023-11-24 19:06:34,760 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2956920.0, ans=0.125
2023-11-24 19:06:37,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2956920.0, ans=0.125
2023-11-24 19:06:42,760 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443550
2023-11-24 19:06:55,216 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10700, loss[loss=0.05979, simple_loss=0.08061, pruned_loss=0.01115, audio_tagging_loss=0.008332, over 14977.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.0898, pruned_loss=0.01264, audio_tagging_loss=0.008807, over 3029596.15 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:07:06,357 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.43 vs. limit=12.0
2023-11-24 19:07:13,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.72 vs. limit=12.0
2023-11-24 19:07:31,432 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2957253.3333333335, ans=0.2
2023-11-24 19:07:35,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2957253.3333333335, ans=0.0
2023-11-24 19:07:44,996 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443600
2023-11-24 19:07:49,520 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.95 vs. limit=15.0
2023-11-24 19:07:51,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2957320.0, ans=0.0
2023-11-24 19:07:57,790 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10750, loss[loss=0.06203, simple_loss=0.08588, pruned_loss=0.01086, audio_tagging_loss=0.008233, over 16717.00 frames. ], tot_loss[loss=0.06611, simple_loss=0.08982, pruned_loss=0.01249, audio_tagging_loss=0.00871, over 3035593.60 frames. ], batch size: 63, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:08:12,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2957453.3333333335, ans=0.2
2023-11-24 19:08:17,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2957453.3333333335, ans=0.1
2023-11-24 19:08:18,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2957453.3333333335, ans=0.125
2023-11-24 19:08:37,960 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 8.444e+01 9.135e+01 9.725e+01 3.439e+02, threshold=1.827e+02, percent-clipped=1.0
2023-11-24 19:08:44,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2957586.6666666665, ans=0.125
2023-11-24 19:08:46,961 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443650
2023-11-24 19:08:52,880 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.42 vs. limit=15.0
2023-11-24 19:08:59,153 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10800, loss[loss=0.07631, simple_loss=0.1077, pruned_loss=0.01409, audio_tagging_loss=0.008384, over 13800.00 frames. ], tot_loss[loss=0.06606, simple_loss=0.08984, pruned_loss=0.01244, audio_tagging_loss=0.008709, over 3043362.08 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:09:03,977 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=14.52 vs. limit=22.5
2023-11-24 19:09:05,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2957720.0, ans=0.125
2023-11-24 19:09:10,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2957786.6666666665, ans=0.125
2023-11-24 19:09:14,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2957786.6666666665, ans=0.125
2023-11-24 19:09:26,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2957853.3333333335, ans=0.0
2023-11-24 19:09:35,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2957920.0, ans=0.125
2023-11-24 19:09:37,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2957920.0, ans=0.0
2023-11-24 19:09:48,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443700
2023-11-24 19:09:56,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2957986.6666666665, ans=0.2
2023-11-24 19:10:01,072 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10850, loss[loss=0.06624, simple_loss=0.09137, pruned_loss=0.01431, audio_tagging_loss=0.006255, over 14982.00 frames. ], tot_loss[loss=0.06638, simple_loss=0.0901, pruned_loss=0.01261, audio_tagging_loss=0.008719, over 3047347.68 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:10:23,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2958120.0, ans=0.125
2023-11-24 19:10:29,245 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.26 vs. limit=6.0
2023-11-24 19:10:34,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2958186.6666666665, ans=0.0
2023-11-24 19:10:35,151 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.66 vs. limit=15.0
2023-11-24 19:10:39,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2958253.3333333335, ans=0.025
2023-11-24 19:10:40,068 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2958253.3333333335, ans=0.1
2023-11-24 19:10:43,177 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.404e+01 9.026e+01 9.467e+01 1.240e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-24 19:10:51,539 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443750
2023-11-24 19:10:59,085 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:11:04,357 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10900, loss[loss=0.05966, simple_loss=0.07598, pruned_loss=0.01233, audio_tagging_loss=0.009344, over 13958.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.0904, pruned_loss=0.01263, audio_tagging_loss=0.008756, over 3047865.98 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:11:04,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2958386.6666666665, ans=0.125
2023-11-24 19:11:07,477 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.29 vs. limit=15.0
2023-11-24 19:11:30,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2958520.0, ans=0.0
2023-11-24 19:11:52,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2958586.6666666665, ans=0.2
2023-11-24 19:11:54,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443800
2023-11-24 19:11:59,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2958653.3333333335, ans=0.125
2023-11-24 19:12:06,963 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 10950, loss[loss=0.06324, simple_loss=0.07962, pruned_loss=0.01446, audio_tagging_loss=0.008967, over 14652.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09077, pruned_loss=0.01276, audio_tagging_loss=0.008856, over 3052936.88 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:12:27,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2958786.6666666665, ans=0.0
2023-11-24 19:12:28,244 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:12:48,383 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.471e+01 9.149e+01 9.860e+01 1.271e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 19:12:56,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443850
2023-11-24 19:13:08,965 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11000, loss[loss=0.04971, simple_loss=0.06704, pruned_loss=0.007763, audio_tagging_loss=0.008423, over 15055.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.09032, pruned_loss=0.01281, audio_tagging_loss=0.008878, over 3053094.46 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:13:10,616 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2959053.3333333335, ans=0.125
2023-11-24 19:13:11,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2959053.3333333335, ans=0.2
2023-11-24 19:13:17,566 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:13:25,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2959120.0, ans=0.09899494936611666
2023-11-24 19:13:25,849 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.46 vs. limit=6.0
2023-11-24 19:13:40,695 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2959186.6666666665, ans=0.125
2023-11-24 19:13:58,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443900
2023-11-24 19:13:59,039 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2959320.0, ans=0.125
2023-11-24 19:14:07,299 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2959320.0, ans=0.125
2023-11-24 19:14:10,725 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11050, loss[loss=0.07543, simple_loss=0.1085, pruned_loss=0.01254, audio_tagging_loss=0.008662, over 15135.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.08983, pruned_loss=0.0128, audio_tagging_loss=0.008901, over 3053170.17 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:14:14,453 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.77 vs. limit=22.5
2023-11-24 19:14:19,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2959386.6666666665, ans=0.2
2023-11-24 19:14:30,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2959453.3333333335, ans=0.0
2023-11-24 19:14:38,859 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.31 vs. limit=15.0
2023-11-24 19:14:43,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2959520.0, ans=0.0
2023-11-24 19:14:52,690 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.475e+01 8.793e+01 9.326e+01 1.000e+02 1.252e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 19:14:53,284 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.82 vs. limit=15.0
2023-11-24 19:14:54,121 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2959586.6666666665, ans=0.125
2023-11-24 19:15:01,789 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 443950
2023-11-24 19:15:14,594 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11100, loss[loss=0.05963, simple_loss=0.08056, pruned_loss=0.01105, audio_tagging_loss=0.008299, over 15510.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.08966, pruned_loss=0.01277, audio_tagging_loss=0.009013, over 3054228.51 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:15:14,825 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2959720.0, ans=0.125
2023-11-24 19:15:36,726 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2959786.6666666665, ans=0.5
2023-11-24 19:16:02,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2959920.0, ans=0.125
2023-11-24 19:16:04,352 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444000
2023-11-24 19:16:04,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2959986.6666666665, ans=0.125
2023-11-24 19:16:21,570 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11150, loss[loss=0.05539, simple_loss=0.07438, pruned_loss=0.007812, audio_tagging_loss=0.01039, over 16003.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09031, pruned_loss=0.01293, audio_tagging_loss=0.009103, over 3062673.80 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:16:35,062 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2960120.0, ans=0.0
2023-11-24 19:16:36,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=2960120.0, ans=0.02
2023-11-24 19:17:02,995 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.194e+01 8.502e+01 9.253e+01 1.001e+02 1.275e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 19:17:11,569 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444050
2023-11-24 19:17:17,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2960320.0, ans=0.1
2023-11-24 19:17:17,783 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=15.0
2023-11-24 19:17:22,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2960386.6666666665, ans=0.0
2023-11-24 19:17:23,205 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11200, loss[loss=0.06342, simple_loss=0.07996, pruned_loss=0.01113, audio_tagging_loss=0.0123, over 15165.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09037, pruned_loss=0.01291, audio_tagging_loss=0.009165, over 3058885.96 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:17:28,819 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2960386.6666666665, ans=0.125
2023-11-24 19:18:13,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444100
2023-11-24 19:18:15,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2960653.3333333335, ans=0.0
2023-11-24 19:18:21,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2960653.3333333335, ans=0.1
2023-11-24 19:18:26,576 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11250, loss[loss=0.07412, simple_loss=0.09739, pruned_loss=0.01892, audio_tagging_loss=0.006506, over 14851.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09048, pruned_loss=0.01296, audio_tagging_loss=0.009095, over 3056659.73 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:18:30,656 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.82 vs. limit=22.5
2023-11-24 19:18:40,664 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2960786.6666666665, ans=0.125
2023-11-24 19:18:42,917 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2960786.6666666665, ans=0.125
2023-11-24 19:18:50,268 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.09 vs. limit=15.0
2023-11-24 19:19:01,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2960853.3333333335, ans=0.125
2023-11-24 19:19:08,006 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.311e+01 8.458e+01 9.183e+01 1.013e+02 1.310e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 19:19:16,059 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444150
2023-11-24 19:19:26,594 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.18 vs. limit=15.0
2023-11-24 19:19:28,393 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11300, loss[loss=0.06209, simple_loss=0.08004, pruned_loss=0.01329, audio_tagging_loss=0.008787, over 14914.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09081, pruned_loss=0.01313, audio_tagging_loss=0.008962, over 3046218.77 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:19:33,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2961053.3333333335, ans=0.2
2023-11-24 19:20:02,839 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=15.0
2023-11-24 19:20:06,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-24 19:20:16,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.68 vs. limit=22.5
2023-11-24 19:20:18,494 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444200
2023-11-24 19:20:30,537 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11350, loss[loss=0.092, simple_loss=0.1403, pruned_loss=0.01635, audio_tagging_loss=0.005517, over 14324.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09136, pruned_loss=0.01319, audio_tagging_loss=0.008885, over 3042798.00 frames. ], batch size: 52, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:20:35,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2961386.6666666665, ans=0.125
2023-11-24 19:20:48,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2961453.3333333335, ans=0.2
2023-11-24 19:20:50,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.28 vs. limit=15.0
2023-11-24 19:21:12,884 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.844e+01 9.278e+01 1.039e+02 2.071e+02, threshold=1.856e+02, percent-clipped=1.0
2023-11-24 19:21:14,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2961586.6666666665, ans=0.0
2023-11-24 19:21:20,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444250
2023-11-24 19:21:32,752 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11400, loss[loss=0.05343, simple_loss=0.07667, pruned_loss=0.006155, audio_tagging_loss=0.008938, over 14053.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.09024, pruned_loss=0.01284, audio_tagging_loss=0.008805, over 3040075.73 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:21:39,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2961720.0, ans=0.125
2023-11-24 19:21:43,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2961720.0, ans=0.125
2023-11-24 19:21:52,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2961786.6666666665, ans=0.125
2023-11-24 19:22:10,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2961920.0, ans=0.125
2023-11-24 19:22:22,354 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2961986.6666666665, ans=0.95
2023-11-24 19:22:23,238 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444300
2023-11-24 19:22:36,139 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11450, loss[loss=0.07515, simple_loss=0.1044, pruned_loss=0.01602, audio_tagging_loss=0.006933, over 15625.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09121, pruned_loss=0.01297, audio_tagging_loss=0.008754, over 3041997.98 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:22:37,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2962053.3333333335, ans=0.125
2023-11-24 19:22:40,147 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2962053.3333333335, ans=0.1
2023-11-24 19:23:11,114 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.98 vs. limit=15.0
2023-11-24 19:23:18,425 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.568e+01 8.582e+01 9.161e+01 1.011e+02 1.144e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 19:23:24,422 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=12.0
2023-11-24 19:23:26,166 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444350
2023-11-24 19:23:38,037 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11500, loss[loss=0.0764, simple_loss=0.1075, pruned_loss=0.01454, audio_tagging_loss=0.008094, over 16250.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.09046, pruned_loss=0.0128, audio_tagging_loss=0.008735, over 3045056.51 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:23:38,597 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.25 vs. limit=15.0
2023-11-24 19:23:48,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2962386.6666666665, ans=0.125
2023-11-24 19:23:48,938 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.31 vs. limit=15.0
2023-11-24 19:24:22,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2962586.6666666665, ans=0.125
2023-11-24 19:24:28,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444400
2023-11-24 19:24:40,768 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11550, loss[loss=0.05487, simple_loss=0.06951, pruned_loss=0.009671, audio_tagging_loss=0.01044, over 14933.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09082, pruned_loss=0.01277, audio_tagging_loss=0.008833, over 3047860.11 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:24:46,366 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2962720.0, ans=0.09899494936611666
2023-11-24 19:25:09,111 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:25:17,127 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:25:23,490 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.651e+01 8.528e+01 9.073e+01 9.782e+01 1.310e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 19:25:24,102 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.09 vs. limit=6.0
2023-11-24 19:25:31,202 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444450
2023-11-24 19:25:43,433 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11600, loss[loss=0.06238, simple_loss=0.07851, pruned_loss=0.0137, audio_tagging_loss=0.009423, over 14860.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09152, pruned_loss=0.01292, audio_tagging_loss=0.008756, over 3055709.96 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:25:59,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2963120.0, ans=0.1
2023-11-24 19:26:08,410 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2963186.6666666665, ans=0.2
2023-11-24 19:26:12,485 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.29 vs. limit=12.0
2023-11-24 19:26:20,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2963253.3333333335, ans=0.0
2023-11-24 19:26:22,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2963253.3333333335, ans=0.0
2023-11-24 19:26:33,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444500
2023-11-24 19:26:40,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2963320.0, ans=0.2
2023-11-24 19:26:42,041 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:26:45,179 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11650, loss[loss=0.05625, simple_loss=0.06786, pruned_loss=0.01314, audio_tagging_loss=0.009181, over 13488.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.0912, pruned_loss=0.01294, audio_tagging_loss=0.008804, over 3044059.84 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:26:56,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2963453.3333333335, ans=0.125
2023-11-24 19:26:57,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2963453.3333333335, ans=0.0
2023-11-24 19:27:02,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2963453.3333333335, ans=0.0
2023-11-24 19:27:10,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2963520.0, ans=0.1
2023-11-24 19:27:28,351 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.973e+01 8.230e+01 8.998e+01 9.783e+01 1.223e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 19:27:34,531 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444550
2023-11-24 19:27:46,804 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11700, loss[loss=0.05877, simple_loss=0.07316, pruned_loss=0.01284, audio_tagging_loss=0.009348, over 14557.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09068, pruned_loss=0.01287, audio_tagging_loss=0.008838, over 3044442.90 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:27:53,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2963720.0, ans=0.2
2023-11-24 19:28:02,816 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2963786.6666666665, ans=0.1
2023-11-24 19:28:11,203 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2963853.3333333335, ans=0.2
2023-11-24 19:28:36,645 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444600
2023-11-24 19:28:49,223 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11750, loss[loss=0.05004, simple_loss=0.06499, pruned_loss=0.00834, audio_tagging_loss=0.00921, over 15495.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09069, pruned_loss=0.01294, audio_tagging_loss=0.008908, over 3045734.08 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:29:13,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.95 vs. limit=10.0
2023-11-24 19:29:16,104 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.whiten.whitening_limit, batch_count=2964186.6666666665, ans=15.0
2023-11-24 19:29:32,999 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.583e+01 9.295e+01 1.006e+02 1.151e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 19:29:38,953 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444650
2023-11-24 19:29:51,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2964386.6666666665, ans=0.125
2023-11-24 19:29:52,053 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11800, loss[loss=0.05819, simple_loss=0.0705, pruned_loss=0.01339, audio_tagging_loss=0.009549, over 14679.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09039, pruned_loss=0.01298, audio_tagging_loss=0.008869, over 3047624.41 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:29:54,622 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2964386.6666666665, ans=0.1
2023-11-24 19:29:59,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2964386.6666666665, ans=0.125
2023-11-24 19:29:59,636 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.72 vs. limit=10.0
2023-11-24 19:30:02,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2964453.3333333335, ans=0.0
2023-11-24 19:30:05,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2964453.3333333335, ans=0.125
2023-11-24 19:30:11,064 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.44 vs. limit=22.5
2023-11-24 19:30:42,152 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444700
2023-11-24 19:30:54,561 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11850, loss[loss=0.05002, simple_loss=0.06578, pruned_loss=0.006267, audio_tagging_loss=0.01086, over 14531.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09043, pruned_loss=0.01303, audio_tagging_loss=0.00888, over 3050769.81 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:30:56,045 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:31:06,634 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.63 vs. limit=10.0
2023-11-24 19:31:09,846 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2964786.6666666665, ans=0.09899494936611666
2023-11-24 19:31:37,847 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.569e+01 9.195e+01 9.795e+01 1.501e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 19:31:44,498 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444750
2023-11-24 19:31:56,656 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11900, loss[loss=0.05708, simple_loss=0.08394, pruned_loss=0.008791, audio_tagging_loss=0.006319, over 15040.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09068, pruned_loss=0.01312, audio_tagging_loss=0.008968, over 3058162.86 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:31:58,347 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.68 vs. limit=15.0
2023-11-24 19:32:02,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2965053.3333333335, ans=0.025
2023-11-24 19:32:02,897 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2965053.3333333335, ans=0.125
2023-11-24 19:32:10,774 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2965120.0, ans=22.5
2023-11-24 19:32:11,646 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2965120.0, ans=0.2
2023-11-24 19:32:46,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444800
2023-11-24 19:32:58,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2965386.6666666665, ans=0.1
2023-11-24 19:32:59,208 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 11950, loss[loss=0.05438, simple_loss=0.07771, pruned_loss=0.005723, audio_tagging_loss=0.009799, over 15671.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.08993, pruned_loss=0.01291, audio_tagging_loss=0.009051, over 3057157.27 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:33:07,277 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=15.0
2023-11-24 19:33:08,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2965386.6666666665, ans=0.125
2023-11-24 19:33:08,328 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2965386.6666666665, ans=0.1
2023-11-24 19:33:11,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2965453.3333333335, ans=0.1
2023-11-24 19:33:11,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2965453.3333333335, ans=0.125
2023-11-24 19:33:24,852 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2965520.0, ans=0.125
2023-11-24 19:33:28,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2965520.0, ans=0.125
2023-11-24 19:33:33,107 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2965520.0, ans=0.125
2023-11-24 19:33:41,982 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.648e+01 8.644e+01 9.247e+01 9.996e+01 1.290e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 19:33:47,783 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444850
2023-11-24 19:33:59,359 INFO [train_asr.py:1221] (2/4) Epoch 37, batch 12000, loss[loss=0.085, simple_loss=0.1087, pruned_loss=0.02124, audio_tagging_loss=0.009405, over 15416.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09036, pruned_loss=0.01295, audio_tagging_loss=0.008982, over 3050943.64 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:33:59,360 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 19:34:24,828 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([4.5174, 3.9912, 3.6958, 4.1805, 3.9390, 4.0137, 4.0645, 3.6127],
       device='cuda:2')
2023-11-24 19:34:41,840 INFO [train_asr.py:1253] (2/4) Epoch 37, validation: loss=0.058, simple_loss=0.05081, pruned_loss=0.005169, audio_tagging_loss=0.02743, over 4681554.00 frames. 
2023-11-24 19:34:41,841 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 19:34:44,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2965720.0, ans=0.125
2023-11-24 19:34:47,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2965720.0, ans=0.025
2023-11-24 19:34:54,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2965786.6666666665, ans=0.1
2023-11-24 19:34:55,585 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2965786.6666666665, ans=0.1
2023-11-24 19:35:40,410 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 0, loss[loss=0.07435, simple_loss=0.0844, pruned_loss=0.008797, audio_tagging_loss=0.02335, over 16016.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.0844, pruned_loss=0.008797, audio_tagging_loss=0.02335, over 16016.00 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:35:40,410 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 19:36:04,195 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9564, 3.1595, 2.9050, 3.1657, 3.3396, 2.8127, 3.3823, 2.6056],
       device='cuda:2')
2023-11-24 19:36:08,480 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3517, 4.9974, 4.6081, 5.1813], device='cuda:2')
2023-11-24 19:36:09,814 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.1588, 3.9529, 3.7122, 3.2349], device='cuda:2')
2023-11-24 19:36:10,483 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.7947, 4.9593, 5.0625, 4.8952], device='cuda:2')
2023-11-24 19:36:12,494 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3358, 5.0186, 4.6607, 5.1707], device='cuda:2')
2023-11-24 19:36:13,290 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8082, 4.9687, 5.0553, 4.9049], device='cuda:2')
2023-11-24 19:36:16,571 INFO [train_asr.py:1253] (2/4) Epoch 38, validation: loss=0.05758, simple_loss=0.05072, pruned_loss=0.005057, audio_tagging_loss=0.02716, over 4681554.00 frames. 
2023-11-24 19:36:16,571 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 19:36:20,689 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2965873.3333333335, ans=0.2
2023-11-24 19:36:27,040 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.73 vs. limit=8.0
2023-11-24 19:36:32,326 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2965940.0, ans=0.0
2023-11-24 19:36:37,508 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444900
2023-11-24 19:36:54,979 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.77 vs. limit=15.0
2023-11-24 19:36:56,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.57 vs. limit=10.0
2023-11-24 19:37:02,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0
2023-11-24 19:37:05,952 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2966140.0, ans=0.0
2023-11-24 19:37:06,575 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.57 vs. limit=15.0
2023-11-24 19:37:18,849 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 50, loss[loss=0.07208, simple_loss=0.0883, pruned_loss=0.01046, audio_tagging_loss=0.01747, over 15438.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.08597, pruned_loss=0.01139, audio_tagging_loss=0.01756, over 685853.35 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:37:20,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2966206.6666666665, ans=0.035
2023-11-24 19:37:20,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2966206.6666666665, ans=0.0
2023-11-24 19:37:35,484 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 8.245e+01 9.441e+01 1.025e+02 1.118e+02 1.388e+02, threshold=2.050e+02, percent-clipped=0.0
2023-11-24 19:37:40,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 444950
2023-11-24 19:37:52,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2966340.0, ans=0.125
2023-11-24 19:37:59,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2966406.6666666665, ans=0.2
2023-11-24 19:38:06,714 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2966406.6666666665, ans=0.125
2023-11-24 19:38:19,803 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0
2023-11-24 19:38:21,596 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 100, loss[loss=0.08047, simple_loss=0.1049, pruned_loss=0.01568, audio_tagging_loss=0.01236, over 14963.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.08861, pruned_loss=0.01203, audio_tagging_loss=0.01683, over 1205200.97 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:38:23,532 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0
2023-11-24 19:38:43,054 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445000
2023-11-24 19:39:13,971 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.81 vs. limit=15.0
2023-11-24 19:39:24,345 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 150, loss[loss=0.05575, simple_loss=0.06467, pruned_loss=0.009498, audio_tagging_loss=0.01392, over 15095.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.08991, pruned_loss=0.0125, audio_tagging_loss=0.01483, over 1614612.45 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:39:32,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2966873.3333333335, ans=0.0
2023-11-24 19:39:39,714 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 9.070e+01 9.567e+01 1.041e+02 1.259e+02, threshold=1.913e+02, percent-clipped=0.0
2023-11-24 19:39:43,545 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2966940.0, ans=0.2
2023-11-24 19:39:44,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445050
2023-11-24 19:40:01,386 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0
2023-11-24 19:40:26,140 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 200, loss[loss=0.06405, simple_loss=0.08968, pruned_loss=0.009053, audio_tagging_loss=0.01016, over 13096.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.08996, pruned_loss=0.0126, audio_tagging_loss=0.01304, over 1933341.79 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:40:48,395 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445100
2023-11-24 19:40:50,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2967340.0, ans=0.125
2023-11-24 19:40:50,933 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2967340.0, ans=0.0
2023-11-24 19:41:28,713 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 250, loss[loss=0.07176, simple_loss=0.1076, pruned_loss=0.01154, audio_tagging_loss=0.006412, over 15922.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09184, pruned_loss=0.01298, audio_tagging_loss=0.01176, over 2185375.18 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:41:36,181 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.17 vs. limit=15.0
2023-11-24 19:41:39,490 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2967540.0, ans=0.04949747468305833
2023-11-24 19:41:45,193 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.968e+01 8.613e+01 9.234e+01 1.003e+02 1.300e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 19:41:48,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2967606.6666666665, ans=0.125
2023-11-24 19:41:50,654 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445150
2023-11-24 19:42:00,786 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.17 vs. limit=10.0
2023-11-24 19:42:20,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2967806.6666666665, ans=0.07
2023-11-24 19:42:31,623 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 300, loss[loss=0.08683, simple_loss=0.1218, pruned_loss=0.01908, audio_tagging_loss=0.006826, over 15526.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09264, pruned_loss=0.01335, audio_tagging_loss=0.01084, over 2378147.23 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:42:52,479 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445200
2023-11-24 19:43:05,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2968006.6666666665, ans=0.0
2023-11-24 19:43:07,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.44 vs. limit=15.0
2023-11-24 19:43:07,996 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2968073.3333333335, ans=0.0
2023-11-24 19:43:34,432 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 350, loss[loss=0.05294, simple_loss=0.0684, pruned_loss=0.00929, audio_tagging_loss=0.009451, over 14951.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09255, pruned_loss=0.01326, audio_tagging_loss=0.01027, over 2532766.16 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:43:35,978 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:43:35,994 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2968206.6666666665, ans=0.0
2023-11-24 19:43:43,734 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.36 vs. limit=15.0
2023-11-24 19:43:48,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2968273.3333333335, ans=0.1
2023-11-24 19:43:51,715 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.795e+01 8.774e+01 9.424e+01 1.013e+02 1.303e+02, threshold=1.885e+02, percent-clipped=0.0
2023-11-24 19:43:55,374 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445250
2023-11-24 19:43:57,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.69 vs. limit=22.5
2023-11-24 19:44:08,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2968340.0, ans=0.125
2023-11-24 19:44:15,517 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=15.13 vs. limit=15.0
2023-11-24 19:44:25,643 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2968473.3333333335, ans=0.0
2023-11-24 19:44:25,690 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2968473.3333333335, ans=0.125
2023-11-24 19:44:25,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2968473.3333333335, ans=0.1
2023-11-24 19:44:29,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2968473.3333333335, ans=0.1
2023-11-24 19:44:36,725 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 400, loss[loss=0.06068, simple_loss=0.0794, pruned_loss=0.01102, audio_tagging_loss=0.009964, over 15609.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09154, pruned_loss=0.01307, audio_tagging_loss=0.009949, over 2644262.48 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:44:48,716 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2968606.6666666665, ans=0.1
2023-11-24 19:44:58,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445300
2023-11-24 19:44:59,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2968606.6666666665, ans=22.5
2023-11-24 19:45:04,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2968673.3333333335, ans=0.125
2023-11-24 19:45:05,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2968673.3333333335, ans=0.0
2023-11-24 19:45:09,205 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2968673.3333333335, ans=0.0
2023-11-24 19:45:11,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.23 vs. limit=15.0
2023-11-24 19:45:14,584 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.87 vs. limit=10.0
2023-11-24 19:45:35,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=2968806.6666666665, ans=0.02
2023-11-24 19:45:39,738 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 450, loss[loss=0.06147, simple_loss=0.08695, pruned_loss=0.01089, audio_tagging_loss=0.007104, over 15157.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09059, pruned_loss=0.01292, audio_tagging_loss=0.009792, over 2730228.32 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:45:56,758 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.512e+01 9.246e+01 9.935e+01 1.248e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 19:46:00,380 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445350
2023-11-24 19:46:11,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2969006.6666666665, ans=0.0
2023-11-24 19:46:41,815 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 500, loss[loss=0.08445, simple_loss=0.1111, pruned_loss=0.01854, audio_tagging_loss=0.01034, over 16327.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.0906, pruned_loss=0.01302, audio_tagging_loss=0.009596, over 2797076.43 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:46:48,227 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2969206.6666666665, ans=0.0
2023-11-24 19:46:48,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2969206.6666666665, ans=15.0
2023-11-24 19:46:54,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2969273.3333333335, ans=0.125
2023-11-24 19:46:59,849 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2969273.3333333335, ans=0.2
2023-11-24 19:47:03,049 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445400
2023-11-24 19:47:10,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2969340.0, ans=0.2
2023-11-24 19:47:15,443 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2969340.0, ans=0.2
2023-11-24 19:47:29,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2969406.6666666665, ans=0.0
2023-11-24 19:47:38,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2969473.3333333335, ans=0.125
2023-11-24 19:47:42,217 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2969473.3333333335, ans=0.95
2023-11-24 19:47:44,364 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 550, loss[loss=0.06061, simple_loss=0.07606, pruned_loss=0.01285, audio_tagging_loss=0.009725, over 13698.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09083, pruned_loss=0.01287, audio_tagging_loss=0.009407, over 2858643.96 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:48:04,004 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 8.415e+01 9.096e+01 9.896e+01 1.420e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 19:48:04,519 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.51 vs. limit=15.0
2023-11-24 19:48:06,504 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445450
2023-11-24 19:48:21,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2969740.0, ans=0.1
2023-11-24 19:48:38,665 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2969806.6666666665, ans=0.2
2023-11-24 19:48:44,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2969806.6666666665, ans=0.0
2023-11-24 19:48:47,447 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 600, loss[loss=0.07386, simple_loss=0.1004, pruned_loss=0.01535, audio_tagging_loss=0.008312, over 14985.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09131, pruned_loss=0.01318, audio_tagging_loss=0.009333, over 2894816.34 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:48:47,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2969873.3333333335, ans=0.125
2023-11-24 19:48:47,691 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2969873.3333333335, ans=0.0
2023-11-24 19:49:05,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2969940.0, ans=0.125
2023-11-24 19:49:06,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2969940.0, ans=0.1
2023-11-24 19:49:08,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445500
2023-11-24 19:49:13,709 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2970006.6666666665, ans=0.125
2023-11-24 19:49:26,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2970073.3333333335, ans=0.125
2023-11-24 19:49:45,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2970140.0, ans=0.09899494936611666
2023-11-24 19:49:49,977 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 650, loss[loss=0.05983, simple_loss=0.08005, pruned_loss=0.007796, audio_tagging_loss=0.01201, over 15425.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09094, pruned_loss=0.01305, audio_tagging_loss=0.009256, over 2928137.72 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:49:51,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2970206.6666666665, ans=0.2
2023-11-24 19:50:01,093 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2970273.3333333335, ans=0.0
2023-11-24 19:50:07,961 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.559e+01 9.252e+01 1.019e+02 1.352e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 19:50:10,980 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445550
2023-11-24 19:50:40,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2970473.3333333335, ans=0.125
2023-11-24 19:50:43,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2970473.3333333335, ans=0.1
2023-11-24 19:50:51,226 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 700, loss[loss=0.06907, simple_loss=0.08744, pruned_loss=0.01285, audio_tagging_loss=0.01249, over 17069.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09075, pruned_loss=0.01281, audio_tagging_loss=0.00929, over 2959648.13 frames. ], batch size: 65, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:50:56,968 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2970540.0, ans=0.1
2023-11-24 19:51:01,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2970540.0, ans=0.125
2023-11-24 19:51:13,021 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445600
2023-11-24 19:51:18,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2970673.3333333335, ans=0.125
2023-11-24 19:51:24,331 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:51:29,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2970740.0, ans=0.125
2023-11-24 19:51:55,294 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 750, loss[loss=0.0616, simple_loss=0.07548, pruned_loss=0.01324, audio_tagging_loss=0.01063, over 14359.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09047, pruned_loss=0.01274, audio_tagging_loss=0.009291, over 2985535.77 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:52:00,270 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2970873.3333333335, ans=0.125
2023-11-24 19:52:02,787 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2970873.3333333335, ans=0.1
2023-11-24 19:52:10,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.20 vs. limit=10.0
2023-11-24 19:52:14,351 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.538e+01 8.682e+01 9.156e+01 1.004e+02 1.177e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 19:52:16,865 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445650
2023-11-24 19:52:19,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2971006.6666666665, ans=0.2
2023-11-24 19:52:22,988 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2971006.6666666665, ans=0.125
2023-11-24 19:52:25,581 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2971006.6666666665, ans=0.2
2023-11-24 19:52:37,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.81 vs. limit=10.0
2023-11-24 19:52:38,368 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2971073.3333333335, ans=0.2
2023-11-24 19:52:53,086 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2971140.0, ans=0.0
2023-11-24 19:52:55,208 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.71 vs. limit=15.0
2023-11-24 19:52:56,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.36 vs. limit=15.0
2023-11-24 19:52:58,243 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 800, loss[loss=0.07558, simple_loss=0.104, pruned_loss=0.0128, audio_tagging_loss=0.01078, over 15522.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09057, pruned_loss=0.01257, audio_tagging_loss=0.009233, over 2997987.58 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:53:05,127 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:53:19,199 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445700
2023-11-24 19:53:33,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2971340.0, ans=0.125
2023-11-24 19:53:36,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2971406.6666666665, ans=0.125
2023-11-24 19:53:44,947 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2971406.6666666665, ans=0.0
2023-11-24 19:53:58,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2971473.3333333335, ans=0.125
2023-11-24 19:53:59,189 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.89 vs. limit=15.0
2023-11-24 19:54:00,945 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 850, loss[loss=0.06861, simple_loss=0.096, pruned_loss=0.01325, audio_tagging_loss=0.007353, over 15184.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.091, pruned_loss=0.01264, audio_tagging_loss=0.00927, over 3011145.56 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:54:19,780 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.517e+01 9.234e+01 9.613e+01 1.155e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 19:54:22,297 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445750
2023-11-24 19:54:24,920 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2971673.3333333335, ans=0.125
2023-11-24 19:54:27,308 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2971673.3333333335, ans=0.09899494936611666
2023-11-24 19:54:48,034 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0
2023-11-24 19:55:03,977 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 900, loss[loss=0.07698, simple_loss=0.107, pruned_loss=0.01569, audio_tagging_loss=0.007804, over 14823.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09074, pruned_loss=0.01263, audio_tagging_loss=0.009347, over 3015208.76 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:55:04,401 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2971873.3333333335, ans=0.125
2023-11-24 19:55:04,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2971873.3333333335, ans=0.2
2023-11-24 19:55:08,383 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-24 19:55:22,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2971940.0, ans=0.0
2023-11-24 19:55:23,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2971940.0, ans=0.125
2023-11-24 19:55:25,413 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445800
2023-11-24 19:55:43,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2972073.3333333335, ans=0.1
2023-11-24 19:55:49,083 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.22 vs. limit=22.5
2023-11-24 19:56:02,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2972140.0, ans=0.125
2023-11-24 19:56:07,363 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 950, loss[loss=0.08685, simple_loss=0.1172, pruned_loss=0.01979, audio_tagging_loss=0.008475, over 15584.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.0911, pruned_loss=0.0126, audio_tagging_loss=0.009213, over 3022594.14 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:56:21,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2972273.3333333335, ans=0.0
2023-11-24 19:56:21,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2972273.3333333335, ans=0.125
2023-11-24 19:56:26,835 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.519e+01 9.243e+01 9.717e+01 1.374e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 19:56:28,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445850
2023-11-24 19:56:28,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2972273.3333333335, ans=0.1
2023-11-24 19:56:47,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2972406.6666666665, ans=0.07
2023-11-24 19:57:09,556 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1000, loss[loss=0.06528, simple_loss=0.08452, pruned_loss=0.01404, audio_tagging_loss=0.008978, over 14920.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09107, pruned_loss=0.01283, audio_tagging_loss=0.009059, over 3023113.89 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:57:14,578 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2972540.0, ans=0.125
2023-11-24 19:57:24,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2972606.6666666665, ans=0.0
2023-11-24 19:57:30,764 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445900
2023-11-24 19:57:35,514 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:57:56,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2972740.0, ans=0.125
2023-11-24 19:58:01,292 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.59 vs. limit=22.5
2023-11-24 19:58:05,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2972806.6666666665, ans=0.0
2023-11-24 19:58:07,894 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.52 vs. limit=22.5
2023-11-24 19:58:11,864 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1050, loss[loss=0.04747, simple_loss=0.06464, pruned_loss=0.005223, audio_tagging_loss=0.009931, over 14558.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08975, pruned_loss=0.01263, audio_tagging_loss=0.008965, over 3030478.33 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:58:13,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2972873.3333333335, ans=0.125
2023-11-24 19:58:32,047 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.733e+01 9.359e+01 1.012e+02 1.444e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 19:58:33,433 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 445950
2023-11-24 19:58:38,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2973006.6666666665, ans=0.125
2023-11-24 19:59:03,467 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.48 vs. limit=10.0
2023-11-24 19:59:14,420 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1100, loss[loss=0.07702, simple_loss=0.1123, pruned_loss=0.0114, audio_tagging_loss=0.009477, over 15832.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08998, pruned_loss=0.01262, audio_tagging_loss=0.008864, over 3041801.39 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:59:17,480 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:59:21,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2973206.6666666665, ans=0.125
2023-11-24 19:59:35,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446000
2023-11-24 19:59:35,941 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2973273.3333333335, ans=0.125
2023-11-24 19:59:35,973 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:59:55,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2973406.6666666665, ans=0.125
2023-11-24 20:00:16,990 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1150, loss[loss=0.07288, simple_loss=0.1029, pruned_loss=0.01252, audio_tagging_loss=0.0089, over 15827.00 frames. ], tot_loss[loss=0.06657, simple_loss=0.09017, pruned_loss=0.01265, audio_tagging_loss=0.008836, over 3044536.07 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:00:37,597 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.661e+01 8.560e+01 9.055e+01 9.595e+01 1.732e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 20:00:38,883 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446050
2023-11-24 20:00:56,776 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.50 vs. limit=22.5
2023-11-24 20:01:19,756 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1200, loss[loss=0.09604, simple_loss=0.1391, pruned_loss=0.01985, audio_tagging_loss=0.006633, over 15147.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.09006, pruned_loss=0.01268, audio_tagging_loss=0.008745, over 3036377.58 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:01:25,526 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.71 vs. limit=15.0
2023-11-24 20:01:28,942 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2973873.3333333335, ans=0.0
2023-11-24 20:01:32,391 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2973940.0, ans=0.0
2023-11-24 20:01:34,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2973940.0, ans=0.0
2023-11-24 20:01:39,030 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0
2023-11-24 20:01:41,049 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446100
2023-11-24 20:01:42,342 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:01:42,486 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2973940.0, ans=0.07
2023-11-24 20:01:50,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2974006.6666666665, ans=0.125
2023-11-24 20:02:05,275 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2974073.3333333335, ans=0.125
2023-11-24 20:02:07,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2974073.3333333335, ans=0.0
2023-11-24 20:02:08,640 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2974140.0, ans=0.1
2023-11-24 20:02:13,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2974140.0, ans=0.125
2023-11-24 20:02:22,313 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1250, loss[loss=0.08142, simple_loss=0.118, pruned_loss=0.0153, audio_tagging_loss=0.007152, over 15738.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.09015, pruned_loss=0.01272, audio_tagging_loss=0.008765, over 3038773.05 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:02:42,693 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.510e+01 9.127e+01 9.845e+01 1.206e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 20:02:42,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446150
2023-11-24 20:02:58,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2974340.0, ans=0.1
2023-11-24 20:03:07,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.40 vs. limit=22.5
2023-11-24 20:03:14,079 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.95 vs. limit=12.0
2023-11-24 20:03:24,240 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1300, loss[loss=0.05504, simple_loss=0.07672, pruned_loss=0.007467, audio_tagging_loss=0.009218, over 13982.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.09015, pruned_loss=0.0128, audio_tagging_loss=0.008744, over 3042697.59 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:03:45,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2974606.6666666665, ans=0.125
2023-11-24 20:03:46,023 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446200
2023-11-24 20:04:01,150 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2974740.0, ans=0.1
2023-11-24 20:04:07,523 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=17.46 vs. limit=22.5
2023-11-24 20:04:22,600 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2974806.6666666665, ans=0.2
2023-11-24 20:04:25,185 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.64 vs. limit=15.0
2023-11-24 20:04:26,367 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1350, loss[loss=0.04953, simple_loss=0.06316, pruned_loss=0.007321, audio_tagging_loss=0.01062, over 14843.00 frames. ], tot_loss[loss=0.06578, simple_loss=0.08905, pruned_loss=0.01247, audio_tagging_loss=0.00879, over 3039428.23 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:04:29,713 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2974873.3333333335, ans=0.2
2023-11-24 20:04:48,067 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.451e+01 8.554e+01 9.036e+01 9.639e+01 2.297e+02, threshold=1.807e+02, percent-clipped=1.0
2023-11-24 20:04:48,207 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446250
2023-11-24 20:04:54,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=2975006.6666666665, ans=0.1
2023-11-24 20:04:59,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2975006.6666666665, ans=0.1
2023-11-24 20:05:00,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2975006.6666666665, ans=0.0
2023-11-24 20:05:10,677 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:05:19,813 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2975140.0, ans=0.0
2023-11-24 20:05:22,487 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.46 vs. limit=22.5
2023-11-24 20:05:26,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2975140.0, ans=0.125
2023-11-24 20:05:29,136 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1400, loss[loss=0.06257, simple_loss=0.07752, pruned_loss=0.01329, audio_tagging_loss=0.01052, over 14997.00 frames. ], tot_loss[loss=0.06561, simple_loss=0.08861, pruned_loss=0.01251, audio_tagging_loss=0.0088, over 3047593.67 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:05:36,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2975206.6666666665, ans=0.1
2023-11-24 20:05:49,959 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446300
2023-11-24 20:06:31,098 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1450, loss[loss=0.05795, simple_loss=0.07946, pruned_loss=0.009732, audio_tagging_loss=0.008488, over 15249.00 frames. ], tot_loss[loss=0.06615, simple_loss=0.08936, pruned_loss=0.01269, audio_tagging_loss=0.008781, over 3044804.11 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:06:33,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2975540.0, ans=0.125
2023-11-24 20:06:43,301 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2975606.6666666665, ans=0.125
2023-11-24 20:06:51,819 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.634e+01 9.284e+01 1.040e+02 1.664e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 20:06:52,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446350
2023-11-24 20:06:53,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2975606.6666666665, ans=0.0
2023-11-24 20:07:06,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2975673.3333333335, ans=0.125
2023-11-24 20:07:24,743 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2975806.6666666665, ans=0.1
2023-11-24 20:07:29,672 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2975806.6666666665, ans=0.1
2023-11-24 20:07:33,036 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1500, loss[loss=0.08333, simple_loss=0.1229, pruned_loss=0.01447, audio_tagging_loss=0.007396, over 15319.00 frames. ], tot_loss[loss=0.06627, simple_loss=0.08931, pruned_loss=0.01271, audio_tagging_loss=0.008905, over 3038788.58 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:07:33,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2975873.3333333335, ans=0.125
2023-11-24 20:07:54,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446400
2023-11-24 20:08:04,226 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.50 vs. limit=10.0
2023-11-24 20:08:15,770 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2976073.3333333335, ans=0.125
2023-11-24 20:08:17,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2976073.3333333335, ans=0.2
2023-11-24 20:08:20,650 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2976073.3333333335, ans=0.0
2023-11-24 20:08:29,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2976140.0, ans=0.125
2023-11-24 20:08:33,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2976140.0, ans=0.0
2023-11-24 20:08:36,381 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1550, loss[loss=0.067, simple_loss=0.08462, pruned_loss=0.01138, audio_tagging_loss=0.01331, over 15392.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.08991, pruned_loss=0.01283, audio_tagging_loss=0.009029, over 3036319.03 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:08:56,985 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446450
2023-11-24 20:08:58,028 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.670e+01 9.483e+01 1.014e+02 1.933e+02, threshold=1.897e+02, percent-clipped=2.0
2023-11-24 20:09:18,907 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.61 vs. limit=15.0
2023-11-24 20:09:23,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2976406.6666666665, ans=0.0
2023-11-24 20:09:37,983 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1600, loss[loss=0.06571, simple_loss=0.07837, pruned_loss=0.01494, audio_tagging_loss=0.01159, over 15457.00 frames. ], tot_loss[loss=0.06629, simple_loss=0.08903, pruned_loss=0.01261, audio_tagging_loss=0.009164, over 3036981.26 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:09:48,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2976606.6666666665, ans=0.125
2023-11-24 20:09:58,756 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446500
2023-11-24 20:10:04,257 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2976673.3333333335, ans=0.2
2023-11-24 20:10:10,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2976673.3333333335, ans=0.0
2023-11-24 20:10:39,341 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1650, loss[loss=0.06672, simple_loss=0.0803, pruned_loss=0.01668, audio_tagging_loss=0.009885, over 14166.00 frames. ], tot_loss[loss=0.06603, simple_loss=0.08844, pruned_loss=0.01262, audio_tagging_loss=0.00919, over 3043502.08 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:11:01,264 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446550
2023-11-24 20:11:02,253 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.314e+01 8.733e+01 9.209e+01 9.885e+01 1.194e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 20:11:11,939 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.85 vs. limit=10.0
2023-11-24 20:11:18,597 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2977073.3333333335, ans=0.0
2023-11-24 20:11:20,823 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2977073.3333333335, ans=0.1
2023-11-24 20:11:36,409 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2977140.0, ans=0.07
2023-11-24 20:11:40,974 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2977206.6666666665, ans=0.125
2023-11-24 20:11:41,964 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1700, loss[loss=0.06965, simple_loss=0.09825, pruned_loss=0.01136, audio_tagging_loss=0.009162, over 15347.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08945, pruned_loss=0.01279, audio_tagging_loss=0.009165, over 3042354.66 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:11:47,668 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2977206.6666666665, ans=0.125
2023-11-24 20:11:59,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2977273.3333333335, ans=0.125
2023-11-24 20:12:03,386 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446600
2023-11-24 20:12:09,865 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2977340.0, ans=0.125
2023-11-24 20:12:15,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2977340.0, ans=0.1
2023-11-24 20:12:16,750 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2977340.0, ans=0.0
2023-11-24 20:12:31,048 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2977473.3333333335, ans=0.125
2023-11-24 20:12:45,064 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1750, loss[loss=0.06141, simple_loss=0.08144, pruned_loss=0.01212, audio_tagging_loss=0.008569, over 15198.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08907, pruned_loss=0.01281, audio_tagging_loss=0.009087, over 3035552.81 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:12:46,511 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2977540.0, ans=0.025
2023-11-24 20:12:49,447 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.90 vs. limit=8.0
2023-11-24 20:12:57,072 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2977606.6666666665, ans=0.035
2023-11-24 20:13:05,284 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446650
2023-11-24 20:13:08,151 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.455e+01 9.155e+01 9.735e+01 1.313e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 20:13:11,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2977673.3333333335, ans=0.0
2023-11-24 20:13:27,348 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2977740.0, ans=0.125
2023-11-24 20:13:27,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2977740.0, ans=0.125
2023-11-24 20:13:27,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2977740.0, ans=0.0
2023-11-24 20:13:36,177 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2977806.6666666665, ans=0.1
2023-11-24 20:13:39,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2977806.6666666665, ans=0.1
2023-11-24 20:13:46,500 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1800, loss[loss=0.04808, simple_loss=0.06958, pruned_loss=0.006416, audio_tagging_loss=0.006867, over 16128.00 frames. ], tot_loss[loss=0.06608, simple_loss=0.08914, pruned_loss=0.01261, audio_tagging_loss=0.008906, over 3038431.61 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:13:57,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2977873.3333333335, ans=0.0
2023-11-24 20:14:03,423 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2977940.0, ans=0.125
2023-11-24 20:14:07,947 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446700
2023-11-24 20:14:17,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2978006.6666666665, ans=0.125
2023-11-24 20:14:27,237 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:14:33,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.51 vs. limit=15.0
2023-11-24 20:14:49,177 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1850, loss[loss=0.06761, simple_loss=0.08372, pruned_loss=0.01371, audio_tagging_loss=0.01203, over 15253.00 frames. ], tot_loss[loss=0.0663, simple_loss=0.08931, pruned_loss=0.01275, audio_tagging_loss=0.008897, over 3044342.81 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:14:52,945 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2978206.6666666665, ans=0.125
2023-11-24 20:14:54,221 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2978206.6666666665, ans=0.125
2023-11-24 20:14:55,925 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.46 vs. limit=15.0
2023-11-24 20:15:10,412 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446750
2023-11-24 20:15:10,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=2978273.3333333335, ans=0.02
2023-11-24 20:15:12,638 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.609e+01 9.248e+01 1.012e+02 1.189e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 20:15:15,362 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2978340.0, ans=0.125
2023-11-24 20:15:20,190 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2978340.0, ans=0.125
2023-11-24 20:15:23,706 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2978340.0, ans=0.125
2023-11-24 20:15:30,240 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2978406.6666666665, ans=0.125
2023-11-24 20:15:33,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2978406.6666666665, ans=0.0
2023-11-24 20:15:51,161 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1900, loss[loss=0.06766, simple_loss=0.09034, pruned_loss=0.01357, audio_tagging_loss=0.008915, over 14295.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.08998, pruned_loss=0.01296, audio_tagging_loss=0.008785, over 3042433.70 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:15:55,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2978540.0, ans=0.125
2023-11-24 20:16:10,537 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=15.0
2023-11-24 20:16:11,274 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446800
2023-11-24 20:16:31,835 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2978740.0, ans=0.0
2023-11-24 20:16:47,188 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2978806.6666666665, ans=0.125
2023-11-24 20:16:49,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2978806.6666666665, ans=0.125
2023-11-24 20:16:52,749 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 1950, loss[loss=0.06891, simple_loss=0.09601, pruned_loss=0.01274, audio_tagging_loss=0.008168, over 15083.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09087, pruned_loss=0.01309, audio_tagging_loss=0.008655, over 3043360.64 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 4.0
2023-11-24 20:17:13,900 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446850
2023-11-24 20:17:17,336 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.207e+01 8.609e+01 9.375e+01 1.004e+02 1.642e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 20:17:32,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2979073.3333333335, ans=0.125
2023-11-24 20:17:34,395 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2979073.3333333335, ans=0.07
2023-11-24 20:17:35,928 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.75 vs. limit=6.0
2023-11-24 20:17:43,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2979140.0, ans=0.0
2023-11-24 20:17:46,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2979140.0, ans=0.0
2023-11-24 20:17:50,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2979140.0, ans=0.125
2023-11-24 20:17:55,517 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2000, loss[loss=0.05182, simple_loss=0.06345, pruned_loss=0.007833, audio_tagging_loss=0.01226, over 14908.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09028, pruned_loss=0.01299, audio_tagging_loss=0.008795, over 3040906.21 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:18:17,327 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446900
2023-11-24 20:18:38,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2979406.6666666665, ans=0.0
2023-11-24 20:18:39,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2979406.6666666665, ans=0.125
2023-11-24 20:18:43,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2979406.6666666665, ans=0.035
2023-11-24 20:18:57,607 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2050, loss[loss=0.07341, simple_loss=0.101, pruned_loss=0.01336, audio_tagging_loss=0.00957, over 16135.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.09027, pruned_loss=0.01284, audio_tagging_loss=0.008773, over 3038632.84 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:18:58,271 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.03 vs. limit=15.0
2023-11-24 20:19:17,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2979606.6666666665, ans=0.2
2023-11-24 20:19:19,226 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 446950
2023-11-24 20:19:19,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2979606.6666666665, ans=0.125
2023-11-24 20:19:21,014 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.20 vs. limit=22.5
2023-11-24 20:19:22,634 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.128e+01 8.521e+01 9.047e+01 9.602e+01 1.413e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 20:19:37,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2979740.0, ans=0.0
2023-11-24 20:19:39,634 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2979740.0, ans=0.125
2023-11-24 20:19:46,814 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2979806.6666666665, ans=0.125
2023-11-24 20:19:46,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2979806.6666666665, ans=0.125
2023-11-24 20:20:00,619 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2100, loss[loss=0.06149, simple_loss=0.0799, pruned_loss=0.01232, audio_tagging_loss=0.009222, over 15807.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.09013, pruned_loss=0.01283, audio_tagging_loss=0.008753, over 3046983.84 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:20:17,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2979940.0, ans=0.0
2023-11-24 20:20:22,397 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447000
2023-11-24 20:20:24,188 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.80 vs. limit=12.0
2023-11-24 20:21:03,517 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2150, loss[loss=0.06693, simple_loss=0.08263, pruned_loss=0.01379, audio_tagging_loss=0.01183, over 16624.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09085, pruned_loss=0.01295, audio_tagging_loss=0.008802, over 3044978.06 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:21:24,848 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447050
2023-11-24 20:21:28,341 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.688e+01 9.386e+01 1.033e+02 1.815e+02, threshold=1.877e+02, percent-clipped=1.0
2023-11-24 20:21:31,013 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2980340.0, ans=0.0
2023-11-24 20:21:32,113 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2980340.0, ans=0.125
2023-11-24 20:21:40,957 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:21:57,794 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:22:05,664 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2200, loss[loss=0.07675, simple_loss=0.1087, pruned_loss=0.01372, audio_tagging_loss=0.008683, over 14242.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09068, pruned_loss=0.01291, audio_tagging_loss=0.0089, over 3042708.01 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:22:14,061 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.78 vs. limit=15.0
2023-11-24 20:22:26,943 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447100
2023-11-24 20:22:35,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2980673.3333333335, ans=0.0
2023-11-24 20:22:43,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2980740.0, ans=0.2
2023-11-24 20:22:52,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2980740.0, ans=0.1
2023-11-24 20:22:52,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2980740.0, ans=0.125
2023-11-24 20:22:52,142 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2980740.0, ans=0.025
2023-11-24 20:23:07,820 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2250, loss[loss=0.06784, simple_loss=0.1025, pruned_loss=0.01039, audio_tagging_loss=0.006194, over 15300.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.0918, pruned_loss=0.01315, audio_tagging_loss=0.00897, over 3046282.42 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:23:20,876 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2980940.0, ans=0.2
2023-11-24 20:23:24,944 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2980940.0, ans=0.1
2023-11-24 20:23:29,843 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447150
2023-11-24 20:23:30,031 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2980940.0, ans=0.09899494936611666
2023-11-24 20:23:33,354 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.657e+01 9.252e+01 1.018e+02 2.312e+02, threshold=1.850e+02, percent-clipped=2.0
2023-11-24 20:23:35,315 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.57 vs. limit=15.0
2023-11-24 20:23:53,238 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2981073.3333333335, ans=0.125
2023-11-24 20:23:53,773 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.62 vs. limit=22.5
2023-11-24 20:23:55,591 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2981073.3333333335, ans=0.1
2023-11-24 20:24:06,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2981140.0, ans=0.125
2023-11-24 20:24:11,231 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2300, loss[loss=0.08128, simple_loss=0.1122, pruned_loss=0.01759, audio_tagging_loss=0.007602, over 15011.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09247, pruned_loss=0.01327, audio_tagging_loss=0.008868, over 3049282.23 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:24:11,761 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.91 vs. limit=15.0
2023-11-24 20:24:12,099 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.60 vs. limit=15.0
2023-11-24 20:24:14,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2981206.6666666665, ans=0.2
2023-11-24 20:24:21,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2981206.6666666665, ans=0.1
2023-11-24 20:24:32,706 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447200
2023-11-24 20:24:39,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2981340.0, ans=0.125
2023-11-24 20:24:47,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2981406.6666666665, ans=0.125
2023-11-24 20:24:59,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2981406.6666666665, ans=0.125
2023-11-24 20:25:05,801 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2981473.3333333335, ans=0.2
2023-11-24 20:25:06,747 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:25:09,478 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2981473.3333333335, ans=0.125
2023-11-24 20:25:14,037 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2350, loss[loss=0.06751, simple_loss=0.09634, pruned_loss=0.01012, audio_tagging_loss=0.009216, over 15274.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09199, pruned_loss=0.01323, audio_tagging_loss=0.008904, over 3050097.03 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:25:24,648 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2981540.0, ans=0.05
2023-11-24 20:25:25,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2981606.6666666665, ans=0.0
2023-11-24 20:25:31,642 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2981606.6666666665, ans=0.07
2023-11-24 20:25:34,988 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447250
2023-11-24 20:25:39,002 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.625e+01 9.128e+01 9.764e+01 1.329e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 20:25:48,568 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.77 vs. limit=15.0
2023-11-24 20:25:50,016 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2981673.3333333335, ans=0.125
2023-11-24 20:26:16,226 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2400, loss[loss=0.05655, simple_loss=0.07177, pruned_loss=0.009135, audio_tagging_loss=0.01153, over 15776.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09146, pruned_loss=0.0132, audio_tagging_loss=0.008928, over 3054386.95 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:26:22,559 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:26:34,420 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2981940.0, ans=0.125
2023-11-24 20:26:38,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447300
2023-11-24 20:26:51,513 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2982006.6666666665, ans=0.0
2023-11-24 20:27:12,570 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.86 vs. limit=15.0
2023-11-24 20:27:18,567 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2450, loss[loss=0.0526, simple_loss=0.06501, pruned_loss=0.008265, audio_tagging_loss=0.01183, over 16545.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09098, pruned_loss=0.01306, audio_tagging_loss=0.008965, over 3057703.49 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:27:29,382 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2982206.6666666665, ans=0.2
2023-11-24 20:27:40,038 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447350
2023-11-24 20:27:44,110 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.597e+01 9.419e+01 1.015e+02 1.292e+02, threshold=1.884e+02, percent-clipped=0.0
2023-11-24 20:27:57,482 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2982406.6666666665, ans=0.125
2023-11-24 20:28:10,413 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2982473.3333333335, ans=0.125
2023-11-24 20:28:15,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2982473.3333333335, ans=0.125
2023-11-24 20:28:19,254 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2982473.3333333335, ans=0.125
2023-11-24 20:28:20,708 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0
2023-11-24 20:28:20,972 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.55 vs. limit=22.5
2023-11-24 20:28:21,319 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2500, loss[loss=0.06012, simple_loss=0.08364, pruned_loss=0.009111, audio_tagging_loss=0.009186, over 14194.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09073, pruned_loss=0.01292, audio_tagging_loss=0.008992, over 3056103.71 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:28:35,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2982606.6666666665, ans=0.2
2023-11-24 20:28:39,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2982606.6666666665, ans=0.125
2023-11-24 20:28:41,118 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2982606.6666666665, ans=0.0
2023-11-24 20:28:42,236 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447400
2023-11-24 20:29:05,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2982740.0, ans=0.125
2023-11-24 20:29:09,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2982740.0, ans=0.0
2023-11-24 20:29:21,063 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.13 vs. limit=15.0
2023-11-24 20:29:23,915 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2550, loss[loss=0.0486, simple_loss=0.06276, pruned_loss=0.006433, audio_tagging_loss=0.01079, over 15822.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09079, pruned_loss=0.01301, audio_tagging_loss=0.008974, over 3057130.14 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:29:44,533 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447450
2023-11-24 20:29:48,496 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.648e+01 8.553e+01 9.176e+01 1.003e+02 1.865e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-24 20:29:51,256 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:30:25,808 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2600, loss[loss=0.05978, simple_loss=0.0885, pruned_loss=0.008547, audio_tagging_loss=0.006986, over 14920.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.08976, pruned_loss=0.01281, audio_tagging_loss=0.008912, over 3055824.16 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:30:36,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2983206.6666666665, ans=0.125
2023-11-24 20:30:47,065 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2983273.3333333335, ans=0.125
2023-11-24 20:30:48,113 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447500
2023-11-24 20:31:29,456 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2650, loss[loss=0.06977, simple_loss=0.08941, pruned_loss=0.01566, audio_tagging_loss=0.009406, over 16206.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.09013, pruned_loss=0.01285, audio_tagging_loss=0.008781, over 3055468.51 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:31:50,183 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447550
2023-11-24 20:31:52,776 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2983673.3333333335, ans=0.125
2023-11-24 20:31:52,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2983673.3333333335, ans=0.1
2023-11-24 20:31:53,538 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.521e+01 8.637e+01 9.375e+01 1.000e+02 1.273e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 20:31:55,582 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.27 vs. limit=22.5
2023-11-24 20:32:01,223 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.79 vs. limit=22.5
2023-11-24 20:32:06,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2983740.0, ans=0.0
2023-11-24 20:32:30,765 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2700, loss[loss=0.08668, simple_loss=0.1239, pruned_loss=0.01598, audio_tagging_loss=0.008757, over 16624.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09154, pruned_loss=0.013, audio_tagging_loss=0.008665, over 3062059.55 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:32:34,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2983873.3333333335, ans=0.0
2023-11-24 20:32:38,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2983873.3333333335, ans=0.0
2023-11-24 20:32:48,206 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2983940.0, ans=0.125
2023-11-24 20:32:52,360 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447600
2023-11-24 20:33:17,098 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2984073.3333333335, ans=0.2
2023-11-24 20:33:28,285 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.29 vs. limit=15.0
2023-11-24 20:33:33,446 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2750, loss[loss=0.07893, simple_loss=0.1122, pruned_loss=0.01436, audio_tagging_loss=0.008471, over 16695.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.0909, pruned_loss=0.01296, audio_tagging_loss=0.008724, over 3059637.44 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:33:41,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2984206.6666666665, ans=0.0
2023-11-24 20:33:53,394 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.20 vs. limit=22.5
2023-11-24 20:33:55,323 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447650
2023-11-24 20:33:59,989 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.580e+01 9.104e+01 1.006e+02 1.298e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 20:34:01,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2984340.0, ans=0.125
2023-11-24 20:34:06,754 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2984340.0, ans=0.125
2023-11-24 20:34:08,101 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.21 vs. limit=10.0
2023-11-24 20:34:27,042 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:34:32,741 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2984473.3333333335, ans=0.025
2023-11-24 20:34:33,032 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-24 20:34:35,945 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2800, loss[loss=0.06648, simple_loss=0.09849, pruned_loss=0.01118, audio_tagging_loss=0.006053, over 14471.00 frames. ], tot_loss[loss=0.06634, simple_loss=0.08973, pruned_loss=0.01279, audio_tagging_loss=0.008688, over 3047862.55 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:34:51,764 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2984606.6666666665, ans=0.2
2023-11-24 20:34:51,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2984606.6666666665, ans=0.0
2023-11-24 20:34:55,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2984606.6666666665, ans=0.125
2023-11-24 20:34:57,455 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447700
2023-11-24 20:34:57,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2984606.6666666665, ans=0.1
2023-11-24 20:35:38,996 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2850, loss[loss=0.06221, simple_loss=0.07907, pruned_loss=0.01229, audio_tagging_loss=0.01038, over 14723.00 frames. ], tot_loss[loss=0.06614, simple_loss=0.08917, pruned_loss=0.01278, audio_tagging_loss=0.008775, over 3043497.84 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:35:39,445 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2984873.3333333335, ans=0.125
2023-11-24 20:35:43,272 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.21 vs. limit=15.0
2023-11-24 20:35:44,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2984873.3333333335, ans=0.125
2023-11-24 20:35:45,464 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2984873.3333333335, ans=0.04949747468305833
2023-11-24 20:35:49,600 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=12.0
2023-11-24 20:35:55,558 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.70 vs. limit=15.0
2023-11-24 20:35:59,887 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447750
2023-11-24 20:36:00,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2984940.0, ans=0.1
2023-11-24 20:36:05,252 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.522e+01 9.152e+01 9.707e+01 1.376e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 20:36:41,870 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2900, loss[loss=0.06625, simple_loss=0.08449, pruned_loss=0.01455, audio_tagging_loss=0.009461, over 15039.00 frames. ], tot_loss[loss=0.06603, simple_loss=0.08903, pruned_loss=0.01273, audio_tagging_loss=0.008785, over 3037227.40 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:36:52,100 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2985206.6666666665, ans=0.125
2023-11-24 20:37:03,290 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447800
2023-11-24 20:37:19,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2985406.6666666665, ans=0.0
2023-11-24 20:37:19,300 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2985406.6666666665, ans=0.0
2023-11-24 20:37:19,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2985406.6666666665, ans=0.0
2023-11-24 20:37:25,646 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0
2023-11-24 20:37:31,066 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:37:35,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2985473.3333333335, ans=0.5
2023-11-24 20:37:45,067 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 2950, loss[loss=0.06002, simple_loss=0.08008, pruned_loss=0.009621, audio_tagging_loss=0.01036, over 13731.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.0904, pruned_loss=0.01288, audio_tagging_loss=0.008839, over 3037640.49 frames. ], batch size: 52, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:37:45,242 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2985540.0, ans=0.0
2023-11-24 20:38:06,272 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447850
2023-11-24 20:38:10,882 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.599e+01 9.301e+01 9.933e+01 1.313e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 20:38:11,173 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2985673.3333333335, ans=0.05
2023-11-24 20:38:18,706 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.85 vs. limit=22.5
2023-11-24 20:38:28,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2985740.0, ans=0.0
2023-11-24 20:38:44,797 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:38:45,995 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2985873.3333333335, ans=0.125
2023-11-24 20:38:47,532 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3000, loss[loss=0.08944, simple_loss=0.1243, pruned_loss=0.01895, audio_tagging_loss=0.008326, over 15230.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.0908, pruned_loss=0.01316, audio_tagging_loss=0.00885, over 3040094.94 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:38:47,532 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 20:39:26,005 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.5092, 3.2659, 3.7895, 3.6236], device='cuda:2')
2023-11-24 20:39:31,507 INFO [train_asr.py:1253] (2/4) Epoch 38, validation: loss=0.05738, simple_loss=0.0507, pruned_loss=0.005077, audio_tagging_loss=0.02696, over 4681554.00 frames. 
2023-11-24 20:39:31,508 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 20:39:39,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2985873.3333333335, ans=0.0
2023-11-24 20:39:47,437 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.64 vs. limit=15.0
2023-11-24 20:39:51,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2985940.0, ans=0.0
2023-11-24 20:39:52,639 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447900
2023-11-24 20:39:59,903 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:40:14,204 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2986073.3333333335, ans=0.0
2023-11-24 20:40:24,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2986140.0, ans=0.1
2023-11-24 20:40:32,194 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2986140.0, ans=0.0
2023-11-24 20:40:34,365 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3050, loss[loss=0.07854, simple_loss=0.1069, pruned_loss=0.01665, audio_tagging_loss=0.008434, over 15458.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09129, pruned_loss=0.01315, audio_tagging_loss=0.008856, over 3039304.61 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:40:47,424 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2986273.3333333335, ans=0.125
2023-11-24 20:40:52,136 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2986273.3333333335, ans=0.125
2023-11-24 20:40:55,640 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 447950
2023-11-24 20:41:00,358 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.683e+01 8.702e+01 9.218e+01 9.877e+01 1.257e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 20:41:10,488 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:41:18,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2986406.6666666665, ans=0.125
2023-11-24 20:41:37,209 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3100, loss[loss=0.06981, simple_loss=0.1021, pruned_loss=0.01277, audio_tagging_loss=0.005992, over 15493.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09072, pruned_loss=0.01298, audio_tagging_loss=0.008977, over 3040976.03 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:41:52,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2986606.6666666665, ans=0.0
2023-11-24 20:41:52,833 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2986606.6666666665, ans=0.125
2023-11-24 20:41:57,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2986606.6666666665, ans=0.1
2023-11-24 20:41:58,112 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448000
2023-11-24 20:41:58,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2986606.6666666665, ans=0.1
2023-11-24 20:42:21,089 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.99 vs. limit=10.0
2023-11-24 20:42:23,092 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2986740.0, ans=0.0
2023-11-24 20:42:23,446 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.87 vs. limit=15.0
2023-11-24 20:42:30,214 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2986806.6666666665, ans=0.125
2023-11-24 20:42:31,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.27 vs. limit=22.5
2023-11-24 20:42:33,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2986806.6666666665, ans=0.125
2023-11-24 20:42:34,728 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2986806.6666666665, ans=0.1
2023-11-24 20:42:42,696 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3150, loss[loss=0.05876, simple_loss=0.07204, pruned_loss=0.01287, audio_tagging_loss=0.009873, over 15022.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.0911, pruned_loss=0.01296, audio_tagging_loss=0.008946, over 3046530.01 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:42:48,941 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.54 vs. limit=12.0
2023-11-24 20:42:59,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2986940.0, ans=0.0
2023-11-24 20:43:02,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.50 vs. limit=6.0
2023-11-24 20:43:04,322 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448050
2023-11-24 20:43:08,908 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.191e+01 8.654e+01 9.120e+01 9.857e+01 1.344e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 20:43:26,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2987073.3333333335, ans=0.2
2023-11-24 20:43:45,952 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3200, loss[loss=0.05609, simple_loss=0.07917, pruned_loss=0.00709, audio_tagging_loss=0.009412, over 14647.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09109, pruned_loss=0.01293, audio_tagging_loss=0.009006, over 3043007.41 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:44:07,212 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448100
2023-11-24 20:44:16,822 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2987340.0, ans=0.0
2023-11-24 20:44:23,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2987406.6666666665, ans=0.04949747468305833
2023-11-24 20:44:24,985 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.02 vs. limit=22.5
2023-11-24 20:44:29,123 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2987406.6666666665, ans=0.015
2023-11-24 20:44:29,303 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2987406.6666666665, ans=0.0
2023-11-24 20:44:32,186 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2987406.6666666665, ans=0.1
2023-11-24 20:44:47,828 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3250, loss[loss=0.04956, simple_loss=0.05925, pruned_loss=0.008781, audio_tagging_loss=0.01115, over 15749.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09036, pruned_loss=0.01287, audio_tagging_loss=0.00915, over 3045142.14 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:44:58,591 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.60 vs. limit=12.0
2023-11-24 20:45:04,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2987606.6666666665, ans=0.125
2023-11-24 20:45:08,962 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448150
2023-11-24 20:45:15,408 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.226e+01 8.565e+01 9.106e+01 9.950e+01 1.221e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 20:45:21,999 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2987673.3333333335, ans=0.2
2023-11-24 20:45:33,053 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.80 vs. limit=10.0
2023-11-24 20:45:47,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2987806.6666666665, ans=0.125
2023-11-24 20:45:50,292 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3300, loss[loss=0.05894, simple_loss=0.07922, pruned_loss=0.01148, audio_tagging_loss=0.007846, over 14928.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09098, pruned_loss=0.01298, audio_tagging_loss=0.009162, over 3045148.03 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:45:52,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2987873.3333333335, ans=0.125
2023-11-24 20:46:11,538 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448200
2023-11-24 20:46:15,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=2988006.6666666665, ans=10.0
2023-11-24 20:46:15,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2988006.6666666665, ans=0.1
2023-11-24 20:46:29,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2988073.3333333335, ans=10.0
2023-11-24 20:46:33,357 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2988073.3333333335, ans=0.125
2023-11-24 20:46:53,182 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3350, loss[loss=0.05288, simple_loss=0.06169, pruned_loss=0.01049, audio_tagging_loss=0.01154, over 15036.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09084, pruned_loss=0.01288, audio_tagging_loss=0.009071, over 3050607.78 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:47:00,321 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2988206.6666666665, ans=0.125
2023-11-24 20:47:06,774 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0
2023-11-24 20:47:10,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2988273.3333333335, ans=0.125
2023-11-24 20:47:13,912 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448250
2023-11-24 20:47:19,738 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.779e+01 9.397e+01 1.017e+02 1.316e+02, threshold=1.879e+02, percent-clipped=0.0
2023-11-24 20:47:27,936 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2988340.0, ans=0.0
2023-11-24 20:47:29,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2988406.6666666665, ans=0.125
2023-11-24 20:47:34,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2988406.6666666665, ans=0.1
2023-11-24 20:47:47,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2988473.3333333335, ans=0.125
2023-11-24 20:47:54,741 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3400, loss[loss=0.05399, simple_loss=0.0677, pruned_loss=0.007537, audio_tagging_loss=0.01261, over 14466.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09106, pruned_loss=0.01276, audio_tagging_loss=0.008949, over 3049494.03 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:48:00,770 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-24 20:48:16,205 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448300
2023-11-24 20:48:18,874 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2988673.3333333335, ans=0.125
2023-11-24 20:48:57,292 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3450, loss[loss=0.06962, simple_loss=0.1071, pruned_loss=0.008896, audio_tagging_loss=0.007167, over 16768.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09072, pruned_loss=0.01274, audio_tagging_loss=0.008957, over 3043196.58 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:49:14,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2988940.0, ans=0.1
2023-11-24 20:49:19,581 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448350
2023-11-24 20:49:23,627 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2989006.6666666665, ans=0.07
2023-11-24 20:49:25,647 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.675e+01 9.257e+01 9.906e+01 1.659e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 20:49:38,679 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2989073.3333333335, ans=0.125
2023-11-24 20:50:01,055 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3500, loss[loss=0.05977, simple_loss=0.07613, pruned_loss=0.009904, audio_tagging_loss=0.0118, over 14976.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09085, pruned_loss=0.01289, audio_tagging_loss=0.008977, over 3047615.83 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:50:22,410 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448400
2023-11-24 20:50:29,469 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.89 vs. limit=22.5
2023-11-24 20:50:32,102 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:50:32,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2989340.0, ans=0.125
2023-11-24 20:50:39,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2989406.6666666665, ans=0.0
2023-11-24 20:50:57,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2989473.3333333335, ans=0.0
2023-11-24 20:51:03,948 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3550, loss[loss=0.0672, simple_loss=0.0931, pruned_loss=0.01106, audio_tagging_loss=0.009597, over 16011.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09067, pruned_loss=0.01285, audio_tagging_loss=0.008836, over 3048148.92 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:51:25,098 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448450
2023-11-24 20:51:32,065 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.602e+01 9.062e+01 9.844e+01 1.252e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 20:51:32,485 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2989673.3333333335, ans=0.0
2023-11-24 20:51:45,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2989740.0, ans=0.2
2023-11-24 20:51:52,325 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2989740.0, ans=0.125
2023-11-24 20:52:06,924 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3600, loss[loss=0.09419, simple_loss=0.1273, pruned_loss=0.02301, audio_tagging_loss=0.007528, over 15473.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09041, pruned_loss=0.01284, audio_tagging_loss=0.008774, over 3046705.98 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:52:25,786 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2989940.0, ans=0.125
2023-11-24 20:52:29,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448500
2023-11-24 20:52:40,964 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2990006.6666666665, ans=0.125
2023-11-24 20:52:57,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2990140.0, ans=0.1
2023-11-24 20:52:57,551 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=17.96 vs. limit=22.5
2023-11-24 20:52:58,306 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2990140.0, ans=0.1
2023-11-24 20:52:58,446 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2990140.0, ans=0.125
2023-11-24 20:53:07,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2990140.0, ans=0.125
2023-11-24 20:53:09,699 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3650, loss[loss=0.06425, simple_loss=0.08105, pruned_loss=0.01535, audio_tagging_loss=0.008375, over 14434.00 frames. ], tot_loss[loss=0.06617, simple_loss=0.08952, pruned_loss=0.01272, audio_tagging_loss=0.008683, over 3043519.06 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:53:13,075 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2990206.6666666665, ans=0.0
2023-11-24 20:53:30,703 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448550
2023-11-24 20:53:36,941 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.710e+01 8.766e+01 9.382e+01 1.003e+02 1.165e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 20:54:03,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2990473.3333333335, ans=10.0
2023-11-24 20:54:05,940 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2990473.3333333335, ans=0.0
2023-11-24 20:54:11,704 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3700, loss[loss=0.06952, simple_loss=0.09177, pruned_loss=0.01129, audio_tagging_loss=0.01234, over 15247.00 frames. ], tot_loss[loss=0.06649, simple_loss=0.0899, pruned_loss=0.01278, audio_tagging_loss=0.00876, over 3050904.90 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:54:32,477 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448600
2023-11-24 20:54:35,414 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2990673.3333333335, ans=0.0
2023-11-24 20:54:39,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2990673.3333333335, ans=0.0
2023-11-24 20:54:45,376 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.94 vs. limit=12.0
2023-11-24 20:55:05,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2990806.6666666665, ans=0.125
2023-11-24 20:55:06,142 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0
2023-11-24 20:55:14,088 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3750, loss[loss=0.06347, simple_loss=0.08065, pruned_loss=0.01155, audio_tagging_loss=0.0116, over 15727.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.08974, pruned_loss=0.01278, audio_tagging_loss=0.008846, over 3051698.01 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:55:35,637 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448650
2023-11-24 20:55:41,373 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.492e+01 9.238e+01 9.913e+01 1.162e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 20:55:54,928 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:56:15,250 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3800, loss[loss=0.0442, simple_loss=0.0547, pruned_loss=0.007926, audio_tagging_loss=0.008925, over 15008.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.0901, pruned_loss=0.01284, audio_tagging_loss=0.008912, over 3050173.99 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:56:36,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448700
2023-11-24 20:56:48,667 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.02 vs. limit=15.0
2023-11-24 20:56:58,161 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.67 vs. limit=22.5
2023-11-24 20:57:11,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2991473.3333333335, ans=0.125
2023-11-24 20:57:14,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2991473.3333333335, ans=0.0
2023-11-24 20:57:17,965 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3850, loss[loss=0.08311, simple_loss=0.1202, pruned_loss=0.01398, audio_tagging_loss=0.009028, over 14752.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09083, pruned_loss=0.01294, audio_tagging_loss=0.008933, over 3051493.34 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:57:35,061 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2991606.6666666665, ans=0.125
2023-11-24 20:57:38,518 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448750
2023-11-24 20:57:44,192 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.441e+01 8.712e+01 9.172e+01 9.948e+01 1.223e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 20:57:48,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2991673.3333333335, ans=0.2
2023-11-24 20:57:54,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2991740.0, ans=0.125
2023-11-24 20:58:19,600 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3900, loss[loss=0.05712, simple_loss=0.06659, pruned_loss=0.01139, audio_tagging_loss=0.01243, over 14988.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08999, pruned_loss=0.0129, audio_tagging_loss=0.009014, over 3043936.06 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:58:22,666 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.03 vs. limit=6.0
2023-11-24 20:58:23,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2991873.3333333335, ans=0.125
2023-11-24 20:58:24,489 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2991873.3333333335, ans=0.0
2023-11-24 20:58:39,293 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2991940.0, ans=0.2
2023-11-24 20:58:40,283 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448800
2023-11-24 20:58:45,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2992006.6666666665, ans=0.125
2023-11-24 20:58:51,556 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2992006.6666666665, ans=0.1
2023-11-24 20:58:58,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2992073.3333333335, ans=0.125
2023-11-24 20:59:00,943 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2992073.3333333335, ans=0.0
2023-11-24 20:59:12,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2992140.0, ans=0.2
2023-11-24 20:59:14,351 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2992140.0, ans=0.125
2023-11-24 20:59:16,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2992140.0, ans=0.125
2023-11-24 20:59:20,993 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 3950, loss[loss=0.06612, simple_loss=0.08652, pruned_loss=0.01396, audio_tagging_loss=0.008898, over 15769.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.08949, pruned_loss=0.01278, audio_tagging_loss=0.009102, over 3044223.06 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 20:59:25,338 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2992206.6666666665, ans=0.125
2023-11-24 20:59:27,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2992206.6666666665, ans=0.125
2023-11-24 20:59:37,563 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.85 vs. limit=15.0
2023-11-24 20:59:41,926 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2992273.3333333335, ans=0.1
2023-11-24 20:59:42,951 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448850
2023-11-24 20:59:48,704 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.553e+01 9.326e+01 9.949e+01 1.288e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 21:00:02,577 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2992406.6666666665, ans=0.1
2023-11-24 21:00:11,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2992473.3333333335, ans=0.125
2023-11-24 21:00:19,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-24 21:00:23,986 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4000, loss[loss=0.07492, simple_loss=0.101, pruned_loss=0.01415, audio_tagging_loss=0.0103, over 14930.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.08977, pruned_loss=0.01285, audio_tagging_loss=0.009141, over 3049022.00 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:00:41,404 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2992606.6666666665, ans=0.125
2023-11-24 21:00:44,777 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448900
2023-11-24 21:00:54,881 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.90 vs. limit=15.0
2023-11-24 21:01:01,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2992740.0, ans=10.0
2023-11-24 21:01:01,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2992740.0, ans=0.125
2023-11-24 21:01:25,963 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4050, loss[loss=0.06613, simple_loss=0.08372, pruned_loss=0.01427, audio_tagging_loss=0.01, over 14791.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09008, pruned_loss=0.01283, audio_tagging_loss=0.009103, over 3047279.17 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:01:27,222 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:01:47,013 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 448950
2023-11-24 21:01:54,344 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.560e+01 8.711e+01 9.505e+01 1.006e+02 1.290e+02, threshold=1.901e+02, percent-clipped=0.0
2023-11-24 21:02:00,161 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2993006.6666666665, ans=0.0
2023-11-24 21:02:19,940 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.75 vs. limit=6.0
2023-11-24 21:02:27,590 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4100, loss[loss=0.05906, simple_loss=0.07918, pruned_loss=0.01159, audio_tagging_loss=0.00788, over 14820.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.08999, pruned_loss=0.01278, audio_tagging_loss=0.009105, over 3046522.89 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:02:33,696 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.68 vs. limit=10.0
2023-11-24 21:02:37,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2993206.6666666665, ans=0.125
2023-11-24 21:02:50,052 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449000
2023-11-24 21:03:31,083 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4150, loss[loss=0.06448, simple_loss=0.08928, pruned_loss=0.01193, audio_tagging_loss=0.007912, over 16619.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09056, pruned_loss=0.01285, audio_tagging_loss=0.008939, over 3045188.85 frames. ], batch size: 63, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:03:31,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2993540.0, ans=0.125
2023-11-24 21:03:47,959 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2993606.6666666665, ans=0.2
2023-11-24 21:03:49,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2993606.6666666665, ans=0.125
2023-11-24 21:03:52,537 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449050
2023-11-24 21:03:59,448 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.744e+01 8.629e+01 9.149e+01 9.710e+01 1.194e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 21:04:08,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2993740.0, ans=0.2
2023-11-24 21:04:15,001 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:04:15,365 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2993740.0, ans=0.0
2023-11-24 21:04:33,291 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4200, loss[loss=0.06886, simple_loss=0.09354, pruned_loss=0.01451, audio_tagging_loss=0.007578, over 15375.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09078, pruned_loss=0.01302, audio_tagging_loss=0.008766, over 3046926.04 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:04:40,955 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2993873.3333333335, ans=0.0
2023-11-24 21:04:53,868 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449100
2023-11-24 21:04:58,218 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2994006.6666666665, ans=0.0
2023-11-24 21:04:59,397 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2994006.6666666665, ans=0.0
2023-11-24 21:05:04,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2994006.6666666665, ans=0.125
2023-11-24 21:05:31,023 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2994140.0, ans=0.125
2023-11-24 21:05:35,509 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4250, loss[loss=0.08083, simple_loss=0.1101, pruned_loss=0.01702, audio_tagging_loss=0.008757, over 14929.00 frames. ], tot_loss[loss=0.06665, simple_loss=0.09025, pruned_loss=0.01286, audio_tagging_loss=0.008663, over 3046894.01 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:05:36,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2994206.6666666665, ans=0.1
2023-11-24 21:05:41,542 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2994206.6666666665, ans=0.125
2023-11-24 21:05:56,958 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449150
2023-11-24 21:06:04,211 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2994340.0, ans=0.125
2023-11-24 21:06:05,024 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.615e+01 8.581e+01 9.086e+01 9.848e+01 1.238e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 21:06:08,087 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2994340.0, ans=0.2
2023-11-24 21:06:28,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2994473.3333333335, ans=0.025
2023-11-24 21:06:30,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2994473.3333333335, ans=0.07
2023-11-24 21:06:33,193 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.19 vs. limit=10.0
2023-11-24 21:06:37,931 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4300, loss[loss=0.07377, simple_loss=0.1029, pruned_loss=0.01383, audio_tagging_loss=0.00848, over 16514.00 frames. ], tot_loss[loss=0.06665, simple_loss=0.09044, pruned_loss=0.01281, audio_tagging_loss=0.00862, over 3045420.62 frames. ], batch size: 62, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:06:59,456 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449200
2023-11-24 21:07:10,807 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2994673.3333333335, ans=0.0
2023-11-24 21:07:40,223 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4350, loss[loss=0.07633, simple_loss=0.1177, pruned_loss=0.008602, audio_tagging_loss=0.008854, over 15903.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09133, pruned_loss=0.01286, audio_tagging_loss=0.008644, over 3047377.77 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:07:40,700 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=15.31 vs. limit=15.0
2023-11-24 21:07:42,885 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2994873.3333333335, ans=0.2
2023-11-24 21:07:42,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2994873.3333333335, ans=0.1
2023-11-24 21:07:57,106 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.29 vs. limit=15.0
2023-11-24 21:07:57,982 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2994940.0, ans=0.0
2023-11-24 21:08:01,551 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449250
2023-11-24 21:08:10,316 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0
2023-11-24 21:08:10,473 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.676e+01 9.413e+01 1.023e+02 1.276e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-24 21:08:13,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2995006.6666666665, ans=0.0
2023-11-24 21:08:24,088 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2995073.3333333335, ans=0.0
2023-11-24 21:08:40,918 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2995140.0, ans=0.07
2023-11-24 21:08:42,985 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4400, loss[loss=0.07193, simple_loss=0.1086, pruned_loss=0.01187, audio_tagging_loss=0.005757, over 14851.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09189, pruned_loss=0.01299, audio_tagging_loss=0.008608, over 3046064.53 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:08:45,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2995206.6666666665, ans=0.1
2023-11-24 21:09:04,458 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449300
2023-11-24 21:09:06,914 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:09:33,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2995473.3333333335, ans=0.125
2023-11-24 21:09:45,354 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4450, loss[loss=0.05425, simple_loss=0.07784, pruned_loss=0.007738, audio_tagging_loss=0.007591, over 13778.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09234, pruned_loss=0.0131, audio_tagging_loss=0.008529, over 3047385.63 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:09:49,809 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2995540.0, ans=0.125
2023-11-24 21:10:00,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2995606.6666666665, ans=0.0
2023-11-24 21:10:01,503 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2995606.6666666665, ans=0.2
2023-11-24 21:10:06,667 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449350
2023-11-24 21:10:16,933 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.434e+01 8.708e+01 9.388e+01 1.016e+02 1.957e+02, threshold=1.878e+02, percent-clipped=1.0
2023-11-24 21:10:22,202 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2995740.0, ans=0.125
2023-11-24 21:10:47,836 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4500, loss[loss=0.07091, simple_loss=0.09826, pruned_loss=0.01415, audio_tagging_loss=0.00763, over 14735.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09256, pruned_loss=0.01315, audio_tagging_loss=0.008493, over 3041092.82 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:10:58,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2995873.3333333335, ans=0.125
2023-11-24 21:10:59,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2995940.0, ans=0.125
2023-11-24 21:11:09,519 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449400
2023-11-24 21:11:31,467 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2996073.3333333335, ans=0.125
2023-11-24 21:11:38,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.41 vs. limit=15.0
2023-11-24 21:11:43,155 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=15.0
2023-11-24 21:11:43,864 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2996140.0, ans=0.0
2023-11-24 21:11:45,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2996140.0, ans=0.2
2023-11-24 21:11:49,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2996206.6666666665, ans=0.0
2023-11-24 21:11:51,466 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4550, loss[loss=0.07361, simple_loss=0.1015, pruned_loss=0.01424, audio_tagging_loss=0.008606, over 15237.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09134, pruned_loss=0.01294, audio_tagging_loss=0.008637, over 3042709.77 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:12:02,604 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2996273.3333333335, ans=0.1
2023-11-24 21:12:13,012 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449450
2023-11-24 21:12:19,040 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2996340.0, ans=0.125
2023-11-24 21:12:22,245 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.502e+01 8.617e+01 8.939e+01 9.828e+01 1.230e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 21:12:23,800 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2996340.0, ans=0.125
2023-11-24 21:12:31,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2996406.6666666665, ans=0.0
2023-11-24 21:12:35,859 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:12:37,689 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.33 vs. limit=12.0
2023-11-24 21:12:41,473 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2996473.3333333335, ans=0.125
2023-11-24 21:12:46,281 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2996473.3333333335, ans=0.07
2023-11-24 21:12:53,661 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4600, loss[loss=0.09068, simple_loss=0.1239, pruned_loss=0.02133, audio_tagging_loss=0.007407, over 15680.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09115, pruned_loss=0.01299, audio_tagging_loss=0.008684, over 3041546.86 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:13:04,980 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2996606.6666666665, ans=0.07
2023-11-24 21:13:07,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2996606.6666666665, ans=0.0
2023-11-24 21:13:10,349 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.41 vs. limit=15.0
2023-11-24 21:13:14,378 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449500
2023-11-24 21:13:17,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2996673.3333333335, ans=0.0
2023-11-24 21:13:22,637 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.72 vs. limit=10.0
2023-11-24 21:13:49,983 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2996806.6666666665, ans=0.125
2023-11-24 21:13:55,764 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4650, loss[loss=0.07052, simple_loss=0.09519, pruned_loss=0.01467, audio_tagging_loss=0.008258, over 16442.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09102, pruned_loss=0.01291, audio_tagging_loss=0.008824, over 3042134.13 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:14:13,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2996940.0, ans=0.1
2023-11-24 21:14:16,906 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449550
2023-11-24 21:14:23,937 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2997006.6666666665, ans=0.1
2023-11-24 21:14:27,679 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.430e+01 9.300e+01 1.028e+02 1.616e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 21:14:49,887 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=15.0
2023-11-24 21:14:58,456 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4700, loss[loss=0.06786, simple_loss=0.09647, pruned_loss=0.01134, audio_tagging_loss=0.008278, over 15899.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.09006, pruned_loss=0.01283, audio_tagging_loss=0.008988, over 3043304.72 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:15:05,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.14 vs. limit=22.5
2023-11-24 21:15:12,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2997273.3333333335, ans=0.125
2023-11-24 21:15:20,733 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449600
2023-11-24 21:15:26,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2997340.0, ans=0.125
2023-11-24 21:15:37,084 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2997406.6666666665, ans=0.125
2023-11-24 21:15:37,085 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2997406.6666666665, ans=0.2
2023-11-24 21:15:43,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2997406.6666666665, ans=0.07
2023-11-24 21:15:50,587 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2997473.3333333335, ans=0.125
2023-11-24 21:15:50,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2997473.3333333335, ans=15.0
2023-11-24 21:15:59,571 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2997473.3333333335, ans=0.125
2023-11-24 21:16:02,245 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4750, loss[loss=0.0453, simple_loss=0.06477, pruned_loss=0.002895, audio_tagging_loss=0.01003, over 15148.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09039, pruned_loss=0.01298, audio_tagging_loss=0.009172, over 3037891.71 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:16:23,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449650
2023-11-24 21:16:33,253 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.234e+01 8.730e+01 9.403e+01 9.994e+01 1.287e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-24 21:16:56,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2997806.6666666665, ans=0.2
2023-11-24 21:17:04,670 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4800, loss[loss=0.08134, simple_loss=0.1133, pruned_loss=0.01909, audio_tagging_loss=0.005605, over 14854.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.08977, pruned_loss=0.0128, audio_tagging_loss=0.009189, over 3034939.00 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:17:04,912 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2997873.3333333335, ans=0.125
2023-11-24 21:17:09,546 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2997873.3333333335, ans=0.1
2023-11-24 21:17:19,779 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:17:20,943 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:17:25,542 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449700
2023-11-24 21:17:53,697 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.40 vs. limit=15.0
2023-11-24 21:18:06,831 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4850, loss[loss=0.06907, simple_loss=0.09687, pruned_loss=0.01315, audio_tagging_loss=0.007493, over 14942.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09078, pruned_loss=0.01281, audio_tagging_loss=0.009188, over 3041957.73 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:18:28,652 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449750
2023-11-24 21:18:30,337 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0
2023-11-24 21:18:32,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2998340.0, ans=0.125
2023-11-24 21:18:36,453 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2998340.0, ans=0.125
2023-11-24 21:18:37,651 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2998340.0, ans=0.125
2023-11-24 21:18:38,538 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.567e+01 9.194e+01 1.005e+02 1.604e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 21:19:07,663 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.54 vs. limit=22.5
2023-11-24 21:19:09,496 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4900, loss[loss=0.07805, simple_loss=0.1062, pruned_loss=0.01666, audio_tagging_loss=0.0083, over 15388.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09046, pruned_loss=0.01268, audio_tagging_loss=0.009119, over 3042988.99 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:19:12,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2998540.0, ans=0.2
2023-11-24 21:19:18,491 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.87 vs. limit=10.0
2023-11-24 21:19:23,510 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2998606.6666666665, ans=0.125
2023-11-24 21:19:25,702 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2998606.6666666665, ans=0.1
2023-11-24 21:19:25,975 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2998606.6666666665, ans=0.125
2023-11-24 21:19:26,997 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2998606.6666666665, ans=0.07
2023-11-24 21:19:31,517 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449800
2023-11-24 21:19:31,678 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2998606.6666666665, ans=0.2
2023-11-24 21:19:32,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2998606.6666666665, ans=0.0
2023-11-24 21:19:43,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2998673.3333333335, ans=0.125
2023-11-24 21:19:48,115 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2998740.0, ans=0.125
2023-11-24 21:20:00,241 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2998806.6666666665, ans=0.125
2023-11-24 21:20:01,449 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2998806.6666666665, ans=0.125
2023-11-24 21:20:08,440 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2998806.6666666665, ans=0.125
2023-11-24 21:20:13,063 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 4950, loss[loss=0.07303, simple_loss=0.1095, pruned_loss=0.0135, audio_tagging_loss=0.004806, over 14874.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09071, pruned_loss=0.01263, audio_tagging_loss=0.008949, over 3036839.91 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:20:16,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2998873.3333333335, ans=0.1
2023-11-24 21:20:33,949 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449850
2023-11-24 21:20:43,267 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.508e+01 9.344e+01 9.867e+01 1.192e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 21:20:54,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2999073.3333333335, ans=0.1
2023-11-24 21:21:03,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2999140.0, ans=0.125
2023-11-24 21:21:15,138 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5000, loss[loss=0.07597, simple_loss=0.1091, pruned_loss=0.01516, audio_tagging_loss=0.006264, over 14825.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.09052, pruned_loss=0.01254, audio_tagging_loss=0.00878, over 3031986.37 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:21:20,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2999206.6666666665, ans=0.0
2023-11-24 21:21:26,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2999273.3333333335, ans=0.1
2023-11-24 21:21:28,970 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.75 vs. limit=15.0
2023-11-24 21:21:36,316 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449900
2023-11-24 21:21:45,785 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2999340.0, ans=0.025
2023-11-24 21:21:48,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2999340.0, ans=0.0
2023-11-24 21:21:49,469 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2999340.0, ans=0.0
2023-11-24 21:21:59,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2999406.6666666665, ans=0.125
2023-11-24 21:22:00,424 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=6.10 vs. limit=15.0
2023-11-24 21:22:14,256 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.68 vs. limit=15.0
2023-11-24 21:22:14,934 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:22:16,951 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5050, loss[loss=0.06157, simple_loss=0.07917, pruned_loss=0.01345, audio_tagging_loss=0.008538, over 14917.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.09086, pruned_loss=0.01264, audio_tagging_loss=0.00864, over 3034372.72 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:22:25,014 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2999540.0, ans=0.0
2023-11-24 21:22:31,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2999606.6666666665, ans=0.125
2023-11-24 21:22:38,481 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 449950
2023-11-24 21:22:48,383 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.754e+01 9.208e+01 9.764e+01 1.294e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 21:22:57,627 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.54 vs. limit=15.0
2023-11-24 21:23:19,854 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5100, loss[loss=0.06394, simple_loss=0.08393, pruned_loss=0.01242, audio_tagging_loss=0.009554, over 15927.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.09055, pruned_loss=0.01267, audio_tagging_loss=0.008764, over 3042171.71 frames. ], batch size: 62, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:23:22,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2999873.3333333335, ans=0.0
2023-11-24 21:23:27,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=2999873.3333333335, ans=15.0
2023-11-24 21:23:40,656 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450000
2023-11-24 21:23:49,516 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3000006.6666666665, ans=0.125
2023-11-24 21:23:49,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3000006.6666666665, ans=0.125
2023-11-24 21:24:06,613 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.60 vs. limit=22.5
2023-11-24 21:24:22,053 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5150, loss[loss=0.07694, simple_loss=0.1064, pruned_loss=0.01465, audio_tagging_loss=0.009114, over 15801.00 frames. ], tot_loss[loss=0.06612, simple_loss=0.08933, pruned_loss=0.01263, audio_tagging_loss=0.008829, over 3048136.97 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:24:22,692 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.64 vs. limit=15.0
2023-11-24 21:24:29,925 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3000206.6666666665, ans=0.125
2023-11-24 21:24:41,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=3000273.3333333335, ans=0.2
2023-11-24 21:24:42,856 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450050
2023-11-24 21:24:53,307 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.721e+01 9.361e+01 9.842e+01 1.528e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 21:25:05,127 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=3000406.6666666665, ans=0.0
2023-11-24 21:25:05,307 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0
2023-11-24 21:25:11,618 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3000473.3333333335, ans=0.0
2023-11-24 21:25:14,017 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3000473.3333333335, ans=0.0
2023-11-24 21:25:19,896 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3000473.3333333335, ans=0.125
2023-11-24 21:25:24,410 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5200, loss[loss=0.06246, simple_loss=0.085, pruned_loss=0.01053, audio_tagging_loss=0.009439, over 14657.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.09028, pruned_loss=0.01273, audio_tagging_loss=0.008798, over 3042425.41 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:25:32,463 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=3000540.0, ans=0.5
2023-11-24 21:25:34,815 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3000540.0, ans=0.125
2023-11-24 21:25:41,279 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3000606.6666666665, ans=0.125
2023-11-24 21:25:42,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3000606.6666666665, ans=0.125
2023-11-24 21:25:45,787 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450100
2023-11-24 21:25:49,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=3000673.3333333335, ans=10.0
2023-11-24 21:26:00,312 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:26:12,209 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3000740.0, ans=0.0
2023-11-24 21:26:22,361 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3000806.6666666665, ans=0.0
2023-11-24 21:26:23,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3000806.6666666665, ans=0.125
2023-11-24 21:26:27,390 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5250, loss[loss=0.05422, simple_loss=0.06679, pruned_loss=0.01065, audio_tagging_loss=0.01017, over 14501.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.092, pruned_loss=0.01308, audio_tagging_loss=0.008645, over 3044360.68 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:26:48,648 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450150
2023-11-24 21:26:51,616 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.41 vs. limit=15.0
2023-11-24 21:26:59,285 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.195e+01 8.390e+01 9.326e+01 9.807e+01 1.099e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 21:27:08,385 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=3001073.3333333335, ans=0.125
2023-11-24 21:27:16,930 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3001140.0, ans=0.125
2023-11-24 21:27:29,184 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5300, loss[loss=0.07966, simple_loss=0.1088, pruned_loss=0.01779, audio_tagging_loss=0.007494, over 15362.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09192, pruned_loss=0.01316, audio_tagging_loss=0.00866, over 3036092.98 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:27:34,536 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3001206.6666666665, ans=0.0
2023-11-24 21:27:36,965 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3001206.6666666665, ans=0.125
2023-11-24 21:27:40,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3001273.3333333335, ans=0.0
2023-11-24 21:27:49,996 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450200
2023-11-24 21:27:50,302 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3001273.3333333335, ans=0.0
2023-11-24 21:28:14,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=3001406.6666666665, ans=0.2
2023-11-24 21:28:18,289 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3001473.3333333335, ans=0.125
2023-11-24 21:28:30,230 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:28:31,198 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5350, loss[loss=0.06146, simple_loss=0.08696, pruned_loss=0.00961, audio_tagging_loss=0.008369, over 14170.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09286, pruned_loss=0.01325, audio_tagging_loss=0.008678, over 3037737.43 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:28:38,519 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3001540.0, ans=0.1
2023-11-24 21:28:52,487 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450250
2023-11-24 21:28:57,412 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=3001673.3333333335, ans=0.2
2023-11-24 21:29:03,593 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.563e+01 9.107e+01 9.848e+01 1.290e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 21:29:04,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=3001673.3333333335, ans=0.0
2023-11-24 21:29:14,022 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3001740.0, ans=0.0
2023-11-24 21:29:23,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3001806.6666666665, ans=0.125
2023-11-24 21:29:30,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3001806.6666666665, ans=0.125
2023-11-24 21:29:30,422 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.88 vs. limit=15.0
2023-11-24 21:29:32,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3001873.3333333335, ans=0.125
2023-11-24 21:29:33,373 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5400, loss[loss=0.06448, simple_loss=0.09345, pruned_loss=0.009534, audio_tagging_loss=0.008222, over 15344.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09201, pruned_loss=0.0131, audio_tagging_loss=0.008803, over 3040792.48 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:29:38,740 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=3001873.3333333335, ans=0.2
2023-11-24 21:29:38,957 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=3001873.3333333335, ans=0.0
2023-11-24 21:29:39,187 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.78 vs. limit=15.0
2023-11-24 21:29:39,924 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3001873.3333333335, ans=0.0
2023-11-24 21:29:40,241 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=15.0
2023-11-24 21:29:40,253 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.23 vs. limit=15.0
2023-11-24 21:29:55,180 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450300
2023-11-24 21:30:34,890 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5450, loss[loss=0.03757, simple_loss=0.04705, pruned_loss=0.003849, audio_tagging_loss=0.0102, over 15730.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09189, pruned_loss=0.01306, audio_tagging_loss=0.008776, over 3042803.90 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:30:40,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=3002206.6666666665, ans=0.2
2023-11-24 21:30:51,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3002273.3333333335, ans=0.1
2023-11-24 21:30:56,241 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450350
2023-11-24 21:30:57,507 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=3002273.3333333335, ans=0.2
2023-11-24 21:30:57,548 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3002273.3333333335, ans=0.0
2023-11-24 21:31:07,240 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.563e+01 9.171e+01 9.760e+01 1.153e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 21:31:36,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3002540.0, ans=0.1
2023-11-24 21:31:36,962 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5500, loss[loss=0.07069, simple_loss=0.1003, pruned_loss=0.01438, audio_tagging_loss=0.006146, over 15034.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.0915, pruned_loss=0.01297, audio_tagging_loss=0.008877, over 3037617.70 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:31:48,280 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3002606.6666666665, ans=0.125
2023-11-24 21:31:52,310 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.95 vs. limit=22.5
2023-11-24 21:31:57,995 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450400
2023-11-24 21:32:09,662 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.80 vs. limit=22.5
2023-11-24 21:32:12,952 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.24 vs. limit=15.0
2023-11-24 21:32:15,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3002740.0, ans=0.1
2023-11-24 21:32:30,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3002806.6666666665, ans=0.125
2023-11-24 21:32:38,635 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5550, loss[loss=0.05365, simple_loss=0.06876, pruned_loss=0.008287, audio_tagging_loss=0.01098, over 15112.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09075, pruned_loss=0.01293, audio_tagging_loss=0.009036, over 3040231.36 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:32:47,103 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.12 vs. limit=15.0
2023-11-24 21:32:57,631 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=3002940.0, ans=0.125
2023-11-24 21:32:59,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450450
2023-11-24 21:33:12,770 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.457e+01 9.165e+01 9.990e+01 1.185e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 21:33:21,349 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3003073.3333333335, ans=0.125
2023-11-24 21:33:40,966 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5600, loss[loss=0.08479, simple_loss=0.1172, pruned_loss=0.0186, audio_tagging_loss=0.007577, over 14981.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09174, pruned_loss=0.01309, audio_tagging_loss=0.009073, over 3046064.80 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:33:42,538 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:34:02,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450500
2023-11-24 21:34:08,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=3003340.0, ans=0.2
2023-11-24 21:34:15,312 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=3003340.0, ans=0.02
2023-11-24 21:34:15,507 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.75 vs. limit=15.0
2023-11-24 21:34:18,054 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3003406.6666666665, ans=0.125
2023-11-24 21:34:19,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=3003406.6666666665, ans=0.2
2023-11-24 21:34:23,712 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:34:30,253 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3003473.3333333335, ans=0.1
2023-11-24 21:34:30,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.95 vs. limit=22.5
2023-11-24 21:34:44,285 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5650, loss[loss=0.0923, simple_loss=0.1302, pruned_loss=0.01901, audio_tagging_loss=0.008193, over 15315.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09172, pruned_loss=0.01317, audio_tagging_loss=0.009154, over 3045669.41 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:34:44,981 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-24 21:34:51,667 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=3003540.0, ans=0.0
2023-11-24 21:34:52,842 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3003540.0, ans=0.125
2023-11-24 21:34:58,223 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3003606.6666666665, ans=0.1
2023-11-24 21:35:04,791 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3003606.6666666665, ans=0.125
2023-11-24 21:35:05,660 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450550
2023-11-24 21:35:17,496 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.220e+01 8.565e+01 9.205e+01 1.008e+02 1.241e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 21:35:30,131 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=3003740.0, ans=0.2
2023-11-24 21:35:36,923 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.76 vs. limit=15.0
2023-11-24 21:35:41,551 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3003806.6666666665, ans=0.1
2023-11-24 21:35:46,713 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5700, loss[loss=0.05894, simple_loss=0.07697, pruned_loss=0.01297, audio_tagging_loss=0.007496, over 15294.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09127, pruned_loss=0.01318, audio_tagging_loss=0.009109, over 3047551.93 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:36:08,110 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450600
2023-11-24 21:36:09,448 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=3003940.0, ans=0.125
2023-11-24 21:36:09,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3003940.0, ans=0.125
2023-11-24 21:36:24,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3004073.3333333335, ans=0.125
2023-11-24 21:36:48,712 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=3004206.6666666665, ans=0.2
2023-11-24 21:36:49,711 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5750, loss[loss=0.06164, simple_loss=0.08119, pruned_loss=0.009541, audio_tagging_loss=0.0115, over 15427.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09055, pruned_loss=0.01309, audio_tagging_loss=0.009029, over 3045163.98 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:37:05,037 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3004273.3333333335, ans=0.125
2023-11-24 21:37:09,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3004273.3333333335, ans=0.1
2023-11-24 21:37:11,188 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450650
2023-11-24 21:37:23,498 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.366e+01 9.124e+01 1.023e+02 1.214e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 21:37:32,027 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.45 vs. limit=22.5
2023-11-24 21:37:48,727 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3004473.3333333335, ans=0.125
2023-11-24 21:37:52,008 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5800, loss[loss=0.05571, simple_loss=0.07002, pruned_loss=0.008425, audio_tagging_loss=0.01227, over 15686.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.0904, pruned_loss=0.01319, audio_tagging_loss=0.008913, over 3041944.86 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:37:56,053 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.05 vs. limit=22.5
2023-11-24 21:38:00,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3004540.0, ans=0.125
2023-11-24 21:38:13,100 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450700
2023-11-24 21:38:13,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=3004606.6666666665, ans=0.0
2023-11-24 21:38:15,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3004606.6666666665, ans=0.125
2023-11-24 21:38:37,867 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3004740.0, ans=0.125
2023-11-24 21:38:49,139 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=3004806.6666666665, ans=0.07
2023-11-24 21:38:54,022 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5850, loss[loss=0.07596, simple_loss=0.09417, pruned_loss=0.01987, audio_tagging_loss=0.009006, over 14335.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09082, pruned_loss=0.01322, audio_tagging_loss=0.008835, over 3039908.40 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:38:57,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3004873.3333333335, ans=0.125
2023-11-24 21:39:01,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3004873.3333333335, ans=0.125
2023-11-24 21:39:03,608 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=3004873.3333333335, ans=15.0
2023-11-24 21:39:08,328 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.42 vs. limit=15.0
2023-11-24 21:39:13,693 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:39:14,668 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450750
2023-11-24 21:39:26,994 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.612e+01 8.737e+01 9.217e+01 9.924e+01 1.196e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 21:39:33,324 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3005073.3333333335, ans=0.125
2023-11-24 21:39:39,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=3005073.3333333335, ans=0.04949747468305833
2023-11-24 21:39:55,820 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5900, loss[loss=0.059, simple_loss=0.07455, pruned_loss=0.01152, audio_tagging_loss=0.01021, over 15210.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09032, pruned_loss=0.01305, audio_tagging_loss=0.008861, over 3033612.14 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:40:04,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=3005206.6666666665, ans=0.125
2023-11-24 21:40:16,499 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450800
2023-11-24 21:40:17,012 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.22 vs. limit=15.0
2023-11-24 21:40:30,234 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.39 vs. limit=22.5
2023-11-24 21:40:47,389 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.58 vs. limit=12.0
2023-11-24 21:40:57,897 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 5950, loss[loss=0.04863, simple_loss=0.06219, pruned_loss=0.008392, audio_tagging_loss=0.009146, over 16953.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09092, pruned_loss=0.01306, audio_tagging_loss=0.008804, over 3044056.65 frames. ], batch size: 66, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:41:09,961 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=3005606.6666666665, ans=0.2
2023-11-24 21:41:19,144 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450850
2023-11-24 21:41:19,532 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.11 vs. limit=15.0
2023-11-24 21:41:31,421 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.501e+01 8.441e+01 9.152e+01 9.875e+01 1.330e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 21:41:50,298 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.66 vs. limit=15.0
2023-11-24 21:41:58,216 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3005873.3333333335, ans=0.1
2023-11-24 21:41:59,102 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6000, loss[loss=0.05433, simple_loss=0.07858, pruned_loss=0.00632, audio_tagging_loss=0.008721, over 14477.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09111, pruned_loss=0.01299, audio_tagging_loss=0.008821, over 3049057.76 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:41:59,102 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 21:42:40,958 INFO [train_asr.py:1253] (2/4) Epoch 38, validation: loss=0.05788, simple_loss=0.05074, pruned_loss=0.005119, audio_tagging_loss=0.02739, over 4681554.00 frames. 
2023-11-24 21:42:40,959 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 21:42:41,114 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=3005873.3333333335, ans=0.0
2023-11-24 21:42:47,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3005873.3333333335, ans=0.1
2023-11-24 21:42:52,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3005940.0, ans=0.0
2023-11-24 21:42:55,916 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=3005940.0, ans=0.2
2023-11-24 21:43:01,673 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450900
2023-11-24 21:43:14,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3006006.6666666665, ans=0.125
2023-11-24 21:43:23,366 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:43:42,799 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6050, loss[loss=0.07138, simple_loss=0.09183, pruned_loss=0.01816, audio_tagging_loss=0.007307, over 15616.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09059, pruned_loss=0.01295, audio_tagging_loss=0.0088, over 3048468.17 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:44:04,047 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 450950
2023-11-24 21:44:17,480 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.516e+01 8.966e+01 9.859e+01 1.258e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 21:44:31,573 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3006473.3333333335, ans=0.0
2023-11-24 21:44:44,315 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6100, loss[loss=0.0756, simple_loss=0.1057, pruned_loss=0.01438, audio_tagging_loss=0.008341, over 15018.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09134, pruned_loss=0.01302, audio_tagging_loss=0.008749, over 3052804.56 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:44:54,755 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3006540.0, ans=0.125
2023-11-24 21:45:06,018 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451000
2023-11-24 21:45:34,293 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.80 vs. limit=6.0
2023-11-24 21:45:47,787 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6150, loss[loss=0.07443, simple_loss=0.1014, pruned_loss=0.0145, audio_tagging_loss=0.00925, over 15689.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.09042, pruned_loss=0.01283, audio_tagging_loss=0.008807, over 3048896.64 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:46:08,490 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451050
2023-11-24 21:46:14,501 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=3007006.6666666665, ans=0.125
2023-11-24 21:46:21,897 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.214e+01 8.834e+01 9.340e+01 1.011e+02 1.357e+02, threshold=1.868e+02, percent-clipped=0.0
2023-11-24 21:46:31,502 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=3007073.3333333335, ans=0.125
2023-11-24 21:46:33,291 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.70 vs. limit=15.0
2023-11-24 21:46:33,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3007073.3333333335, ans=0.125
2023-11-24 21:46:49,553 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6200, loss[loss=0.0733, simple_loss=0.1038, pruned_loss=0.01276, audio_tagging_loss=0.008629, over 14994.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09043, pruned_loss=0.01285, audio_tagging_loss=0.008851, over 3051149.50 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:46:58,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3007206.6666666665, ans=0.1
2023-11-24 21:47:10,332 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451100
2023-11-24 21:47:10,434 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3007273.3333333335, ans=0.0
2023-11-24 21:47:29,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=3007406.6666666665, ans=0.2
2023-11-24 21:47:30,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=3007406.6666666665, ans=0.0
2023-11-24 21:47:34,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3007406.6666666665, ans=0.125
2023-11-24 21:47:41,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3007473.3333333335, ans=0.1
2023-11-24 21:47:46,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3007473.3333333335, ans=0.125
2023-11-24 21:47:50,830 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6250, loss[loss=0.07011, simple_loss=0.09202, pruned_loss=0.01414, audio_tagging_loss=0.009953, over 15303.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.08966, pruned_loss=0.01268, audio_tagging_loss=0.008947, over 3056007.80 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:48:12,957 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451150
2023-11-24 21:48:19,076 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3007673.3333333335, ans=0.125
2023-11-24 21:48:20,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3007673.3333333335, ans=0.125
2023-11-24 21:48:26,312 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 8.537e+01 9.157e+01 9.825e+01 1.470e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 21:48:45,283 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.33 vs. limit=22.5
2023-11-24 21:48:48,402 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3007806.6666666665, ans=0.125
2023-11-24 21:48:53,248 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6300, loss[loss=0.08439, simple_loss=0.1092, pruned_loss=0.01929, audio_tagging_loss=0.01051, over 15756.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.0899, pruned_loss=0.01273, audio_tagging_loss=0.009091, over 3054965.98 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:48:53,593 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3007873.3333333335, ans=0.1
2023-11-24 21:48:59,299 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.83 vs. limit=10.0
2023-11-24 21:49:04,970 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3007940.0, ans=0.125
2023-11-24 21:49:12,346 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3007940.0, ans=0.1
2023-11-24 21:49:14,663 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451200
2023-11-24 21:49:14,922 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3007940.0, ans=0.0
2023-11-24 21:49:23,624 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3008006.6666666665, ans=0.125
2023-11-24 21:49:27,033 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=3008006.6666666665, ans=0.125
2023-11-24 21:49:35,761 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3008073.3333333335, ans=0.125
2023-11-24 21:49:49,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3008140.0, ans=0.125
2023-11-24 21:49:56,227 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6350, loss[loss=0.06101, simple_loss=0.0814, pruned_loss=0.01099, audio_tagging_loss=0.009317, over 16003.00 frames. ], tot_loss[loss=0.06633, simple_loss=0.08934, pruned_loss=0.01257, audio_tagging_loss=0.00909, over 3053072.57 frames. ], batch size: 62, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:50:15,594 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3008273.3333333335, ans=0.0
2023-11-24 21:50:17,218 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451250
2023-11-24 21:50:18,529 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3008273.3333333335, ans=0.0
2023-11-24 21:50:27,549 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3008340.0, ans=0.1
2023-11-24 21:50:30,742 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.435e+01 8.876e+01 9.772e+01 1.151e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 21:50:35,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=3008406.6666666665, ans=0.0
2023-11-24 21:50:47,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.41 vs. limit=15.0
2023-11-24 21:50:52,149 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3008473.3333333335, ans=0.1
2023-11-24 21:50:57,803 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6400, loss[loss=0.06545, simple_loss=0.08774, pruned_loss=0.01049, audio_tagging_loss=0.01109, over 16634.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.09002, pruned_loss=0.01261, audio_tagging_loss=0.009142, over 3052993.65 frames. ], batch size: 63, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:51:14,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=3008606.6666666665, ans=10.0
2023-11-24 21:51:15,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=3008606.6666666665, ans=0.0
2023-11-24 21:51:18,997 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451300
2023-11-24 21:51:19,555 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.61 vs. limit=15.0
2023-11-24 21:51:43,644 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=3008740.0, ans=0.95
2023-11-24 21:51:59,919 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6450, loss[loss=0.06059, simple_loss=0.08303, pruned_loss=0.009731, audio_tagging_loss=0.009345, over 17258.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.08992, pruned_loss=0.01259, audio_tagging_loss=0.009242, over 3055871.36 frames. ], batch size: 66, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:52:09,319 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.41 vs. limit=12.0
2023-11-24 21:52:21,350 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451350
2023-11-24 21:52:26,237 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3009006.6666666665, ans=0.1
2023-11-24 21:52:34,133 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.612e+01 9.313e+01 1.021e+02 1.233e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 21:52:34,548 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:52:45,543 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=3009073.3333333335, ans=0.025
2023-11-24 21:52:55,470 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3009140.0, ans=0.0
2023-11-24 21:53:01,732 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6500, loss[loss=0.05655, simple_loss=0.07845, pruned_loss=0.008884, audio_tagging_loss=0.008436, over 16220.00 frames. ], tot_loss[loss=0.06648, simple_loss=0.08938, pruned_loss=0.01258, audio_tagging_loss=0.009211, over 3049597.06 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:53:06,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3009206.6666666665, ans=0.0
2023-11-24 21:53:07,316 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=3009206.6666666665, ans=0.035
2023-11-24 21:53:09,806 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=3009206.6666666665, ans=0.0
2023-11-24 21:53:22,644 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451400
2023-11-24 21:53:31,726 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:54:04,588 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6550, loss[loss=0.05129, simple_loss=0.06879, pruned_loss=0.007099, audio_tagging_loss=0.009802, over 14755.00 frames. ], tot_loss[loss=0.06593, simple_loss=0.08883, pruned_loss=0.01244, audio_tagging_loss=0.009072, over 3042668.65 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:54:16,182 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=3009606.6666666665, ans=0.125
2023-11-24 21:54:19,980 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.67 vs. limit=10.0
2023-11-24 21:54:25,934 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451450
2023-11-24 21:54:35,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=3009673.3333333335, ans=0.0
2023-11-24 21:54:39,839 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.039e+01 8.549e+01 9.293e+01 1.004e+02 1.709e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 21:55:06,532 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6600, loss[loss=0.07346, simple_loss=0.09989, pruned_loss=0.01631, audio_tagging_loss=0.007204, over 15119.00 frames. ], tot_loss[loss=0.06623, simple_loss=0.08954, pruned_loss=0.01252, audio_tagging_loss=0.008943, over 3041379.73 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:55:16,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3009873.3333333335, ans=0.1
2023-11-24 21:55:28,502 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451500
2023-11-24 21:55:58,010 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3010140.0, ans=0.125
2023-11-24 21:56:08,549 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6650, loss[loss=0.06531, simple_loss=0.09353, pruned_loss=0.01305, audio_tagging_loss=0.005496, over 14654.00 frames. ], tot_loss[loss=0.06597, simple_loss=0.08899, pruned_loss=0.01257, audio_tagging_loss=0.008901, over 3037214.61 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:56:21,364 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.64 vs. limit=15.0
2023-11-24 21:56:30,285 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451550
2023-11-24 21:56:40,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=3010340.0, ans=0.0
2023-11-24 21:56:44,235 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.352e+01 8.506e+01 9.140e+01 9.928e+01 1.246e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 21:56:44,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3010340.0, ans=0.1
2023-11-24 21:56:48,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3010406.6666666665, ans=0.125
2023-11-24 21:57:04,307 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3010473.3333333335, ans=0.1
2023-11-24 21:57:11,175 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6700, loss[loss=0.07363, simple_loss=0.1036, pruned_loss=0.01174, audio_tagging_loss=0.01009, over 15112.00 frames. ], tot_loss[loss=0.06641, simple_loss=0.08951, pruned_loss=0.0128, audio_tagging_loss=0.008855, over 3037821.60 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:57:32,702 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451600
2023-11-24 21:57:33,304 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.28 vs. limit=15.0
2023-11-24 21:57:35,505 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3010673.3333333335, ans=0.125
2023-11-24 21:57:44,370 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.86 vs. limit=15.0
2023-11-24 21:57:46,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3010673.3333333335, ans=0.125
2023-11-24 21:58:13,671 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6750, loss[loss=0.0581, simple_loss=0.07913, pruned_loss=0.01047, audio_tagging_loss=0.00807, over 15548.00 frames. ], tot_loss[loss=0.06621, simple_loss=0.08938, pruned_loss=0.01268, audio_tagging_loss=0.008838, over 3035902.87 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:58:25,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3010940.0, ans=0.0
2023-11-24 21:58:28,898 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=3010940.0, ans=0.125
2023-11-24 21:58:34,503 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451650
2023-11-24 21:58:49,688 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.221e+01 8.416e+01 8.949e+01 9.766e+01 1.528e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-24 21:58:54,611 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:59:13,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3011140.0, ans=0.0
2023-11-24 21:59:13,820 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0
2023-11-24 21:59:15,544 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6800, loss[loss=0.07581, simple_loss=0.1122, pruned_loss=0.01394, audio_tagging_loss=0.005771, over 15425.00 frames. ], tot_loss[loss=0.06612, simple_loss=0.08939, pruned_loss=0.0126, audio_tagging_loss=0.008818, over 3037691.63 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:59:36,655 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451700
2023-11-24 21:59:48,419 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.62 vs. limit=12.0
2023-11-24 22:00:11,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3011473.3333333335, ans=0.1
2023-11-24 22:00:18,171 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6850, loss[loss=0.0503, simple_loss=0.07061, pruned_loss=0.007738, audio_tagging_loss=0.007259, over 14085.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.09039, pruned_loss=0.01277, audio_tagging_loss=0.008738, over 3031969.92 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:00:39,348 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451750
2023-11-24 22:00:48,993 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=3011673.3333333335, ans=0.0
2023-11-24 22:00:51,532 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=3011673.3333333335, ans=0.0
2023-11-24 22:00:54,781 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.792e+01 9.209e+01 9.855e+01 1.264e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 22:01:12,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=3011806.6666666665, ans=0.125
2023-11-24 22:01:12,900 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3011806.6666666665, ans=0.125
2023-11-24 22:01:15,187 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3011806.6666666665, ans=0.125
2023-11-24 22:01:19,196 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.35 vs. limit=15.0
2023-11-24 22:01:19,653 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6900, loss[loss=0.0559, simple_loss=0.07521, pruned_loss=0.009948, audio_tagging_loss=0.008348, over 13618.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.09052, pruned_loss=0.01276, audio_tagging_loss=0.008728, over 3028628.49 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:01:33,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=3011940.0, ans=0.07
2023-11-24 22:01:41,160 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451800
2023-11-24 22:01:50,499 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=3012006.6666666665, ans=0.05
2023-11-24 22:02:04,156 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3012073.3333333335, ans=0.125
2023-11-24 22:02:05,462 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.09 vs. limit=22.5
2023-11-24 22:02:06,114 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:02:22,848 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 6950, loss[loss=0.08963, simple_loss=0.1279, pruned_loss=0.01872, audio_tagging_loss=0.006948, over 15110.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09178, pruned_loss=0.01286, audio_tagging_loss=0.008698, over 3033020.71 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:02:25,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3012206.6666666665, ans=0.0
2023-11-24 22:02:25,883 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.77 vs. limit=22.5
2023-11-24 22:02:44,293 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451850
2023-11-24 22:02:54,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3012340.0, ans=0.0
2023-11-24 22:03:00,153 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.498e+01 8.404e+01 9.015e+01 9.725e+01 1.456e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 22:03:02,206 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:03:04,518 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3012406.6666666665, ans=0.125
2023-11-24 22:03:15,718 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.47 vs. limit=6.0
2023-11-24 22:03:17,588 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=3012473.3333333335, ans=0.2
2023-11-24 22:03:24,965 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7000, loss[loss=0.07557, simple_loss=0.1017, pruned_loss=0.01745, audio_tagging_loss=0.007249, over 15243.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09182, pruned_loss=0.01297, audio_tagging_loss=0.008809, over 3032899.48 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:03:35,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3012540.0, ans=0.1
2023-11-24 22:03:42,472 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3012606.6666666665, ans=0.0
2023-11-24 22:03:46,409 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451900
2023-11-24 22:03:46,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=3012606.6666666665, ans=0.0
2023-11-24 22:03:54,226 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:04:20,297 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=3012806.6666666665, ans=0.05
2023-11-24 22:04:25,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3012806.6666666665, ans=0.125
2023-11-24 22:04:26,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3012873.3333333335, ans=0.125
2023-11-24 22:04:27,260 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7050, loss[loss=0.07882, simple_loss=0.1108, pruned_loss=0.01606, audio_tagging_loss=0.007386, over 14103.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09109, pruned_loss=0.01285, audio_tagging_loss=0.008855, over 3036418.58 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:04:39,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.76 vs. limit=15.0
2023-11-24 22:04:48,510 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 451950
2023-11-24 22:04:57,987 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0
2023-11-24 22:05:04,197 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.687e+01 9.290e+01 1.031e+02 1.279e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-24 22:05:08,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=3013073.3333333335, ans=0.0
2023-11-24 22:05:26,488 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.77 vs. limit=22.5
2023-11-24 22:05:29,456 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7100, loss[loss=0.06407, simple_loss=0.07749, pruned_loss=0.01484, audio_tagging_loss=0.01048, over 14495.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09109, pruned_loss=0.01299, audio_tagging_loss=0.008949, over 3042822.91 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:05:39,260 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3013206.6666666665, ans=0.1
2023-11-24 22:05:43,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=3013273.3333333335, ans=0.125
2023-11-24 22:05:44,454 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3013273.3333333335, ans=0.0
2023-11-24 22:05:50,279 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452000
2023-11-24 22:06:03,635 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3013340.0, ans=0.125
2023-11-24 22:06:11,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3013406.6666666665, ans=0.1
2023-11-24 22:06:13,506 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=3013406.6666666665, ans=0.125
2023-11-24 22:06:14,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=3013406.6666666665, ans=0.2
2023-11-24 22:06:14,090 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=3013406.6666666665, ans=0.05
2023-11-24 22:06:15,291 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=3013406.6666666665, ans=0.0
2023-11-24 22:06:29,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3013473.3333333335, ans=0.125
2023-11-24 22:06:35,702 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7150, loss[loss=0.05509, simple_loss=0.07551, pruned_loss=0.006568, audio_tagging_loss=0.01077, over 14485.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09175, pruned_loss=0.01309, audio_tagging_loss=0.009007, over 3044589.16 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:06:50,958 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.62 vs. limit=22.5
2023-11-24 22:06:55,296 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=3013606.6666666665, ans=0.125
2023-11-24 22:06:56,888 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452050
2023-11-24 22:07:12,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3013740.0, ans=0.1
2023-11-24 22:07:12,927 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.770e+01 9.367e+01 1.013e+02 1.404e+02, threshold=1.873e+02, percent-clipped=0.0
2023-11-24 22:07:25,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3013806.6666666665, ans=0.125
2023-11-24 22:07:32,871 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3013806.6666666665, ans=0.125
2023-11-24 22:07:36,781 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=3013873.3333333335, ans=15.0
2023-11-24 22:07:37,803 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7200, loss[loss=0.05536, simple_loss=0.07467, pruned_loss=0.00848, audio_tagging_loss=0.009546, over 13970.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09058, pruned_loss=0.01283, audio_tagging_loss=0.009014, over 3046868.02 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:07:46,837 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3013873.3333333335, ans=0.1
2023-11-24 22:07:58,906 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452100
2023-11-24 22:08:01,407 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=3014006.6666666665, ans=0.035
2023-11-24 22:08:11,621 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3014006.6666666665, ans=0.0
2023-11-24 22:08:11,797 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:08:40,396 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7250, loss[loss=0.07566, simple_loss=0.1125, pruned_loss=0.01231, audio_tagging_loss=0.00711, over 16536.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09148, pruned_loss=0.01297, audio_tagging_loss=0.009025, over 3049228.81 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:08:50,593 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.37 vs. limit=15.0
2023-11-24 22:08:57,719 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3014273.3333333335, ans=0.1
2023-11-24 22:09:01,050 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452150
2023-11-24 22:09:17,412 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.25 vs. limit=15.0
2023-11-24 22:09:18,539 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.621e+01 8.553e+01 9.150e+01 9.601e+01 1.170e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 22:09:26,693 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.86 vs. limit=12.0
2023-11-24 22:09:41,859 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7300, loss[loss=0.04937, simple_loss=0.06765, pruned_loss=0.006229, audio_tagging_loss=0.009316, over 14594.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09176, pruned_loss=0.01295, audio_tagging_loss=0.008952, over 3048658.78 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:10:02,577 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452200
2023-11-24 22:10:43,880 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7350, loss[loss=0.08346, simple_loss=0.1255, pruned_loss=0.01325, audio_tagging_loss=0.007446, over 14718.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09171, pruned_loss=0.01281, audio_tagging_loss=0.008807, over 3044416.38 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:10:44,151 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:11:05,585 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452250
2023-11-24 22:11:05,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=3014940.0, ans=0.125
2023-11-24 22:11:22,649 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.188e+01 8.406e+01 8.894e+01 9.896e+01 1.273e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-24 22:11:27,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3015073.3333333335, ans=0.125
2023-11-24 22:11:33,562 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3015140.0, ans=0.125
2023-11-24 22:11:46,720 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7400, loss[loss=0.06029, simple_loss=0.08016, pruned_loss=0.01083, audio_tagging_loss=0.009376, over 15078.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.092, pruned_loss=0.01278, audio_tagging_loss=0.008744, over 3052541.58 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:12:07,418 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452300
2023-11-24 22:12:11,654 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0
2023-11-24 22:12:13,949 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.57 vs. limit=15.0
2023-11-24 22:12:18,812 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.15 vs. limit=15.0
2023-11-24 22:12:28,491 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=3015406.6666666665, ans=0.125
2023-11-24 22:12:48,013 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7450, loss[loss=0.04929, simple_loss=0.06304, pruned_loss=0.007723, audio_tagging_loss=0.01004, over 15644.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09108, pruned_loss=0.01266, audio_tagging_loss=0.00874, over 3041914.88 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:12:50,537 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3015540.0, ans=0.125
2023-11-24 22:12:52,374 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3015540.0, ans=0.125
2023-11-24 22:13:08,730 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452350
2023-11-24 22:13:28,037 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.262e+01 8.690e+01 9.275e+01 1.001e+02 1.240e+02, threshold=1.855e+02, percent-clipped=0.0
2023-11-24 22:13:28,288 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=3015740.0, ans=0.125
2023-11-24 22:13:34,644 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.97 vs. limit=15.0
2023-11-24 22:13:42,919 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=3015806.6666666665, ans=0.125
2023-11-24 22:13:48,891 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3015873.3333333335, ans=0.1
2023-11-24 22:13:49,772 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7500, loss[loss=0.09979, simple_loss=0.1431, pruned_loss=0.02393, audio_tagging_loss=0.004304, over 16136.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09121, pruned_loss=0.01277, audio_tagging_loss=0.008611, over 3050340.31 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:13:52,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3015873.3333333335, ans=0.1
2023-11-24 22:14:09,346 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.67 vs. limit=15.0
2023-11-24 22:14:11,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452400
2023-11-24 22:14:12,187 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:14:39,214 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=15.0
2023-11-24 22:14:43,828 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.32 vs. limit=12.0
2023-11-24 22:14:51,527 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.51 vs. limit=15.0
2023-11-24 22:14:52,040 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7550, loss[loss=0.07137, simple_loss=0.1026, pruned_loss=0.01378, audio_tagging_loss=0.006304, over 16037.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09108, pruned_loss=0.01281, audio_tagging_loss=0.008591, over 3055295.33 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:15:01,998 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=3016206.6666666665, ans=0.125
2023-11-24 22:15:08,129 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=3016273.3333333335, ans=0.0
2023-11-24 22:15:14,480 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452450
2023-11-24 22:15:22,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3016340.0, ans=0.125
2023-11-24 22:15:30,071 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3016406.6666666665, ans=0.125
2023-11-24 22:15:32,171 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.402e+01 8.333e+01 9.038e+01 9.761e+01 1.363e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 22:15:50,428 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.59 vs. limit=10.0
2023-11-24 22:15:55,643 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7600, loss[loss=0.06219, simple_loss=0.08686, pruned_loss=0.01122, audio_tagging_loss=0.007538, over 14908.00 frames. ], tot_loss[loss=0.06626, simple_loss=0.09013, pruned_loss=0.01259, audio_tagging_loss=0.008609, over 3062826.13 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:16:09,352 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=3016606.6666666665, ans=0.1
2023-11-24 22:16:12,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=3016606.6666666665, ans=0.2
2023-11-24 22:16:16,317 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452500
2023-11-24 22:16:32,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=3016740.0, ans=0.0
2023-11-24 22:16:57,866 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7650, loss[loss=0.06784, simple_loss=0.09814, pruned_loss=0.01294, audio_tagging_loss=0.005825, over 15028.00 frames. ], tot_loss[loss=0.0664, simple_loss=0.09013, pruned_loss=0.01267, audio_tagging_loss=0.00866, over 3058881.11 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:16:59,231 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3016873.3333333335, ans=0.1
2023-11-24 22:17:19,339 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452550
2023-11-24 22:17:37,940 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.417e+01 9.176e+01 1.003e+02 1.926e+02, threshold=1.835e+02, percent-clipped=1.0
2023-11-24 22:17:39,450 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3017073.3333333335, ans=0.125
2023-11-24 22:17:49,238 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:18:00,072 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7700, loss[loss=0.06924, simple_loss=0.09724, pruned_loss=0.01281, audio_tagging_loss=0.007808, over 14477.00 frames. ], tot_loss[loss=0.06642, simple_loss=0.09009, pruned_loss=0.01267, audio_tagging_loss=0.008699, over 3055691.99 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:18:09,323 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=3017206.6666666665, ans=0.125
2023-11-24 22:18:11,670 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3017273.3333333335, ans=0.125
2023-11-24 22:18:15,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3017273.3333333335, ans=0.125
2023-11-24 22:18:17,077 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3017273.3333333335, ans=0.125
2023-11-24 22:18:21,458 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452600
2023-11-24 22:18:25,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=3017340.0, ans=0.125
2023-11-24 22:18:27,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=3017340.0, ans=0.2
2023-11-24 22:18:33,189 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3017340.0, ans=0.0
2023-11-24 22:18:52,370 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=3017473.3333333335, ans=0.0
2023-11-24 22:19:02,806 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7750, loss[loss=0.06088, simple_loss=0.0758, pruned_loss=0.01291, audio_tagging_loss=0.01007, over 15080.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08996, pruned_loss=0.01266, audio_tagging_loss=0.00881, over 3049671.83 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:19:07,463 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.56 vs. limit=15.0
2023-11-24 22:19:08,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3017540.0, ans=0.125
2023-11-24 22:19:22,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=3017606.6666666665, ans=0.0
2023-11-24 22:19:23,989 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452650
2023-11-24 22:19:27,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3017673.3333333335, ans=0.125
2023-11-24 22:19:42,995 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.572e+01 9.238e+01 9.882e+01 1.332e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 22:19:46,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3017740.0, ans=0.125
2023-11-24 22:19:54,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3017806.6666666665, ans=0.0
2023-11-24 22:20:05,365 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7800, loss[loss=0.08229, simple_loss=0.1155, pruned_loss=0.01941, audio_tagging_loss=0.005156, over 15171.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.0898, pruned_loss=0.01269, audio_tagging_loss=0.008847, over 3045948.72 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:20:26,853 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452700
2023-11-24 22:20:29,335 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=3018006.6666666665, ans=0.125
2023-11-24 22:20:43,979 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.83 vs. limit=22.5
2023-11-24 22:20:49,179 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=3018073.3333333335, ans=0.0
2023-11-24 22:21:04,073 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.53 vs. limit=22.5
2023-11-24 22:21:06,941 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7850, loss[loss=0.06068, simple_loss=0.08602, pruned_loss=0.009268, audio_tagging_loss=0.008403, over 14111.00 frames. ], tot_loss[loss=0.0663, simple_loss=0.08954, pruned_loss=0.01259, audio_tagging_loss=0.008939, over 3041407.30 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:21:19,487 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.50 vs. limit=10.0
2023-11-24 22:21:28,250 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452750
2023-11-24 22:21:32,151 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3018340.0, ans=0.0
2023-11-24 22:21:40,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=3018340.0, ans=10.0
2023-11-24 22:21:46,948 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.360e+01 8.800e+01 9.383e+01 9.995e+01 1.679e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-24 22:21:51,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=3018406.6666666665, ans=0.2
2023-11-24 22:21:53,269 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=3018406.6666666665, ans=0.0
2023-11-24 22:22:02,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3018473.3333333335, ans=0.125
2023-11-24 22:22:09,006 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3018540.0, ans=0.0
2023-11-24 22:22:09,986 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7900, loss[loss=0.06068, simple_loss=0.07946, pruned_loss=0.009305, audio_tagging_loss=0.01165, over 14854.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09025, pruned_loss=0.0127, audio_tagging_loss=0.008997, over 3044582.17 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:22:12,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3018540.0, ans=0.0
2023-11-24 22:22:17,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3018540.0, ans=0.125
2023-11-24 22:22:30,083 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.60 vs. limit=15.0
2023-11-24 22:22:30,876 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452800
2023-11-24 22:22:56,433 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3018740.0, ans=0.125
2023-11-24 22:22:58,044 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.77 vs. limit=15.0
2023-11-24 22:23:07,181 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=3018806.6666666665, ans=0.2
2023-11-24 22:23:08,305 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3018806.6666666665, ans=0.1
2023-11-24 22:23:12,356 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 7950, loss[loss=0.07176, simple_loss=0.09149, pruned_loss=0.0139, audio_tagging_loss=0.01212, over 15079.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09041, pruned_loss=0.01264, audio_tagging_loss=0.009085, over 3044362.66 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:23:22,261 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3018873.3333333335, ans=0.125
2023-11-24 22:23:25,680 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:23:33,464 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452850
2023-11-24 22:23:40,440 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.70 vs. limit=15.0
2023-11-24 22:23:48,314 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=3019073.3333333335, ans=0.0
2023-11-24 22:23:52,664 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.512e+01 8.496e+01 9.034e+01 9.610e+01 1.973e+02, threshold=1.807e+02, percent-clipped=1.0
2023-11-24 22:23:55,547 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3019073.3333333335, ans=0.0
2023-11-24 22:24:01,192 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3019140.0, ans=0.1
2023-11-24 22:24:11,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=3019140.0, ans=0.2
2023-11-24 22:24:13,981 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8000, loss[loss=0.07022, simple_loss=0.1021, pruned_loss=0.01204, audio_tagging_loss=0.007147, over 14247.00 frames. ], tot_loss[loss=0.06654, simple_loss=0.08946, pruned_loss=0.01254, audio_tagging_loss=0.009273, over 3038107.41 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:24:16,975 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.0
2023-11-24 22:24:35,515 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452900
2023-11-24 22:24:41,701 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3019340.0, ans=0.125
2023-11-24 22:24:50,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3019406.6666666665, ans=0.125
2023-11-24 22:24:52,229 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=3019406.6666666665, ans=0.125
2023-11-24 22:24:57,278 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0
2023-11-24 22:25:01,834 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3019406.6666666665, ans=0.125
2023-11-24 22:25:02,913 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=3019473.3333333335, ans=0.125
2023-11-24 22:25:03,988 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:25:16,637 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8050, loss[loss=0.07303, simple_loss=0.09659, pruned_loss=0.01505, audio_tagging_loss=0.009692, over 14138.00 frames. ], tot_loss[loss=0.06615, simple_loss=0.08872, pruned_loss=0.01248, audio_tagging_loss=0.009316, over 3034638.92 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:25:17,972 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=3019540.0, ans=0.05
2023-11-24 22:25:25,154 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3019540.0, ans=0.1
2023-11-24 22:25:26,653 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=22.5
2023-11-24 22:25:31,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3019606.6666666665, ans=0.0
2023-11-24 22:25:38,349 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 452950
2023-11-24 22:25:44,683 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.24 vs. limit=15.0
2023-11-24 22:25:54,575 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3019740.0, ans=0.125
2023-11-24 22:25:59,473 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.598e+01 8.520e+01 9.232e+01 9.839e+01 1.227e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 22:26:14,317 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=3019806.6666666665, ans=0.125
2023-11-24 22:26:18,639 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8100, loss[loss=0.05265, simple_loss=0.07685, pruned_loss=0.008882, audio_tagging_loss=0.00534, over 14475.00 frames. ], tot_loss[loss=0.06598, simple_loss=0.08856, pruned_loss=0.01245, audio_tagging_loss=0.009241, over 3037435.73 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:26:39,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3019940.0, ans=0.0
2023-11-24 22:26:39,399 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.92 vs. limit=6.0
2023-11-24 22:26:40,120 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453000
2023-11-24 22:26:45,834 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.99 vs. limit=15.0
2023-11-24 22:27:07,018 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.84 vs. limit=15.0
2023-11-24 22:27:21,713 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8150, loss[loss=0.06962, simple_loss=0.09523, pruned_loss=0.01196, audio_tagging_loss=0.01005, over 15169.00 frames. ], tot_loss[loss=0.06595, simple_loss=0.0887, pruned_loss=0.0125, audio_tagging_loss=0.009094, over 3042241.91 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:27:33,768 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3020273.3333333335, ans=0.125
2023-11-24 22:27:38,745 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=3020273.3333333335, ans=0.5
2023-11-24 22:27:43,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453050
2023-11-24 22:27:58,133 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=3020406.6666666665, ans=0.5
2023-11-24 22:28:04,263 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.627e+01 9.105e+01 9.732e+01 1.211e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 22:28:12,810 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3020473.3333333335, ans=0.1
2023-11-24 22:28:23,163 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:28:24,321 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8200, loss[loss=0.07901, simple_loss=0.1143, pruned_loss=0.01497, audio_tagging_loss=0.006868, over 16257.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.08989, pruned_loss=0.01274, audio_tagging_loss=0.008944, over 3044818.28 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:28:25,836 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=3020540.0, ans=0.125
2023-11-24 22:28:44,364 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=3020606.6666666665, ans=0.125
2023-11-24 22:28:45,370 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453100
2023-11-24 22:29:11,128 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3020740.0, ans=0.125
2023-11-24 22:29:26,349 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8250, loss[loss=0.06908, simple_loss=0.09423, pruned_loss=0.01346, audio_tagging_loss=0.008503, over 15678.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.0902, pruned_loss=0.01289, audio_tagging_loss=0.008912, over 3045962.18 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:29:31,860 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:29:47,860 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453150
2023-11-24 22:30:08,935 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.191e+01 8.522e+01 9.040e+01 9.801e+01 1.361e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 22:30:26,512 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=3021140.0, ans=0.05
2023-11-24 22:30:27,751 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=3021206.6666666665, ans=0.0
2023-11-24 22:30:28,673 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8300, loss[loss=0.07732, simple_loss=0.1155, pruned_loss=0.01221, audio_tagging_loss=0.007363, over 17335.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09042, pruned_loss=0.01284, audio_tagging_loss=0.008767, over 3053047.05 frames. ], batch size: 63, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:30:33,026 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3021206.6666666665, ans=0.0
2023-11-24 22:30:40,878 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3021273.3333333335, ans=0.1
2023-11-24 22:30:50,904 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453200
2023-11-24 22:31:14,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=3021406.6666666665, ans=0.0
2023-11-24 22:31:32,468 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8350, loss[loss=0.07871, simple_loss=0.1126, pruned_loss=0.01331, audio_tagging_loss=0.009113, over 15298.00 frames. ], tot_loss[loss=0.06634, simple_loss=0.08993, pruned_loss=0.01268, audio_tagging_loss=0.008697, over 3051220.15 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:31:52,700 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453250
2023-11-24 22:31:59,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=3021673.3333333335, ans=0.0
2023-11-24 22:32:05,390 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=3021673.3333333335, ans=0.95
2023-11-24 22:32:07,641 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=3021740.0, ans=0.2
2023-11-24 22:32:14,492 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.506e+01 8.493e+01 9.150e+01 9.803e+01 1.531e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 22:32:20,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3021740.0, ans=0.125
2023-11-24 22:32:28,286 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=3021806.6666666665, ans=0.0
2023-11-24 22:32:29,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=3021806.6666666665, ans=0.2
2023-11-24 22:32:34,011 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8400, loss[loss=0.07865, simple_loss=0.09716, pruned_loss=0.01955, audio_tagging_loss=0.01051, over 14730.00 frames. ], tot_loss[loss=0.06653, simple_loss=0.09008, pruned_loss=0.01278, audio_tagging_loss=0.008706, over 3051553.57 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:32:54,741 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453300
2023-11-24 22:32:59,141 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=3022006.6666666665, ans=0.2
2023-11-24 22:33:08,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=3022006.6666666665, ans=0.125
2023-11-24 22:33:29,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=3022140.0, ans=0.125
2023-11-24 22:33:36,186 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8450, loss[loss=0.04846, simple_loss=0.06453, pruned_loss=0.006282, audio_tagging_loss=0.009915, over 14598.00 frames. ], tot_loss[loss=0.06615, simple_loss=0.08944, pruned_loss=0.01264, audio_tagging_loss=0.008791, over 3054959.04 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:33:41,111 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=3022206.6666666665, ans=0.125
2023-11-24 22:33:52,461 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3022273.3333333335, ans=0.125
2023-11-24 22:33:58,045 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453350
2023-11-24 22:34:06,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3022340.0, ans=0.125
2023-11-24 22:34:18,673 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.351e+01 8.708e+01 9.231e+01 1.022e+02 1.410e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 22:34:29,494 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3022473.3333333335, ans=0.1
2023-11-24 22:34:38,768 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8500, loss[loss=0.06868, simple_loss=0.08908, pruned_loss=0.01358, audio_tagging_loss=0.01056, over 15415.00 frames. ], tot_loss[loss=0.06654, simple_loss=0.09051, pruned_loss=0.01262, audio_tagging_loss=0.008674, over 3053046.49 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:34:49,746 INFO [scaling.py:1022] (2/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.16 vs. limit=8.0
2023-11-24 22:34:59,569 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453400
2023-11-24 22:35:41,411 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8550, loss[loss=0.0637, simple_loss=0.08543, pruned_loss=0.01227, audio_tagging_loss=0.008724, over 15394.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.09079, pruned_loss=0.01259, audio_tagging_loss=0.008714, over 3048178.35 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:35:41,769 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3022873.3333333335, ans=0.125
2023-11-24 22:35:48,853 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=3022873.3333333335, ans=0.0
2023-11-24 22:35:51,775 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3022873.3333333335, ans=0.0
2023-11-24 22:35:56,383 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3022940.0, ans=0.1
2023-11-24 22:36:02,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453450
2023-11-24 22:36:19,274 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=3023073.3333333335, ans=0.0
2023-11-24 22:36:21,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=3023073.3333333335, ans=0.0
2023-11-24 22:36:23,759 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.604e+01 8.608e+01 9.249e+01 9.944e+01 1.269e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 22:36:30,027 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:36:43,373 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8600, loss[loss=0.06698, simple_loss=0.09391, pruned_loss=0.01147, audio_tagging_loss=0.008551, over 15455.00 frames. ], tot_loss[loss=0.06607, simple_loss=0.08963, pruned_loss=0.01239, audio_tagging_loss=0.008861, over 3045091.97 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:36:47,590 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.58 vs. limit=15.0
2023-11-24 22:37:05,754 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453500
2023-11-24 22:37:30,219 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=3023406.6666666665, ans=0.125
2023-11-24 22:37:40,777 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=3023473.3333333335, ans=0.1
2023-11-24 22:37:45,784 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8650, loss[loss=0.05156, simple_loss=0.06972, pruned_loss=0.008972, audio_tagging_loss=0.007731, over 14868.00 frames. ], tot_loss[loss=0.06617, simple_loss=0.08973, pruned_loss=0.01237, audio_tagging_loss=0.008942, over 3045709.25 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:38:05,493 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.62 vs. limit=10.0
2023-11-24 22:38:07,163 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453550
2023-11-24 22:38:27,882 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.604e+01 9.306e+01 1.022e+02 1.267e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 22:38:28,160 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3023740.0, ans=0.1
2023-11-24 22:38:32,251 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3023740.0, ans=0.125
2023-11-24 22:38:48,886 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8700, loss[loss=0.05613, simple_loss=0.07705, pruned_loss=0.007958, audio_tagging_loss=0.009646, over 15482.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.09017, pruned_loss=0.01248, audio_tagging_loss=0.00898, over 3045934.58 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:38:49,501 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.53 vs. limit=15.0
2023-11-24 22:38:53,971 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3023873.3333333335, ans=0.125
2023-11-24 22:39:09,843 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453600
2023-11-24 22:39:25,736 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3024073.3333333335, ans=0.0
2023-11-24 22:39:51,306 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8750, loss[loss=0.07335, simple_loss=0.1041, pruned_loss=0.01418, audio_tagging_loss=0.007139, over 15913.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09097, pruned_loss=0.01266, audio_tagging_loss=0.009035, over 3048319.51 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:40:12,864 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453650
2023-11-24 22:40:22,064 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.02 vs. limit=22.5
2023-11-24 22:40:23,423 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.82 vs. limit=15.0
2023-11-24 22:40:26,530 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=3024340.0, ans=0.2
2023-11-24 22:40:29,306 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.60 vs. limit=15.0
2023-11-24 22:40:33,324 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.979e+01 9.554e+01 1.051e+02 1.529e+02, threshold=1.911e+02, percent-clipped=0.0
2023-11-24 22:40:35,224 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.37 vs. limit=12.0
2023-11-24 22:40:37,792 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3024406.6666666665, ans=0.125
2023-11-24 22:40:42,380 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=3024473.3333333335, ans=0.025
2023-11-24 22:40:52,680 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8800, loss[loss=0.06989, simple_loss=0.09975, pruned_loss=0.01319, audio_tagging_loss=0.00682, over 15290.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09277, pruned_loss=0.01314, audio_tagging_loss=0.008976, over 3052006.30 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:41:02,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=3024540.0, ans=0.125
2023-11-24 22:41:02,632 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3024540.0, ans=0.1
2023-11-24 22:41:04,748 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3024606.6666666665, ans=0.1
2023-11-24 22:41:09,057 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3024606.6666666665, ans=0.0
2023-11-24 22:41:14,650 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453700
2023-11-24 22:41:37,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3024740.0, ans=0.1
2023-11-24 22:41:56,234 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8850, loss[loss=0.04575, simple_loss=0.05841, pruned_loss=0.004351, audio_tagging_loss=0.0122, over 16151.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09238, pruned_loss=0.01309, audio_tagging_loss=0.008952, over 3053723.22 frames. ], batch size: 64, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:42:05,739 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:42:09,498 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3024940.0, ans=0.125
2023-11-24 22:42:16,863 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453750
2023-11-24 22:42:34,586 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3025073.3333333335, ans=0.125
2023-11-24 22:42:35,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=3025073.3333333335, ans=0.2
2023-11-24 22:42:36,062 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.72 vs. limit=15.0
2023-11-24 22:42:37,811 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.566e+01 8.570e+01 9.101e+01 9.826e+01 1.259e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 22:42:46,041 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3025140.0, ans=0.1
2023-11-24 22:42:51,072 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=6.62 vs. limit=15.0
2023-11-24 22:42:57,362 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8900, loss[loss=0.07815, simple_loss=0.1148, pruned_loss=0.01421, audio_tagging_loss=0.006513, over 15211.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09276, pruned_loss=0.01317, audio_tagging_loss=0.008826, over 3053742.44 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:43:18,677 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453800
2023-11-24 22:43:30,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3025340.0, ans=0.1
2023-11-24 22:43:48,207 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=3025473.3333333335, ans=0.0
2023-11-24 22:43:49,466 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3025473.3333333335, ans=0.125
2023-11-24 22:43:59,729 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 8950, loss[loss=0.06583, simple_loss=0.08357, pruned_loss=0.01543, audio_tagging_loss=0.008621, over 15070.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09306, pruned_loss=0.01345, audio_tagging_loss=0.008652, over 3049643.39 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:44:15,665 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.49 vs. limit=15.0
2023-11-24 22:44:21,872 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453850
2023-11-24 22:44:25,465 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=3025673.3333333335, ans=0.2
2023-11-24 22:44:36,639 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3025740.0, ans=0.0
2023-11-24 22:44:39,067 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3025740.0, ans=0.0
2023-11-24 22:44:43,569 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.683e+01 8.565e+01 9.194e+01 9.956e+01 1.408e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 22:44:55,156 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.49 vs. limit=22.5
2023-11-24 22:45:02,690 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9000, loss[loss=0.0698, simple_loss=0.09216, pruned_loss=0.016, audio_tagging_loss=0.007722, over 15729.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09265, pruned_loss=0.01336, audio_tagging_loss=0.008596, over 3052000.32 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:45:02,691 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 22:45:39,256 INFO [zipformer.py:1873] (2/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3439, 5.0206, 4.6860, 5.1778], device='cuda:2')
2023-11-24 22:45:46,046 INFO [train_asr.py:1253] (2/4) Epoch 38, validation: loss=0.05855, simple_loss=0.05069, pruned_loss=0.005085, audio_tagging_loss=0.02812, over 4681554.00 frames. 
2023-11-24 22:45:46,047 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 22:45:46,243 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3025873.3333333335, ans=0.125
2023-11-24 22:45:47,892 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.32 vs. limit=15.0
2023-11-24 22:46:07,178 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453900
2023-11-24 22:46:14,834 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.01 vs. limit=15.0
2023-11-24 22:46:47,359 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9050, loss[loss=0.0723, simple_loss=0.0935, pruned_loss=0.01554, audio_tagging_loss=0.01002, over 15943.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09255, pruned_loss=0.01335, audio_tagging_loss=0.008574, over 3047500.02 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:46:47,564 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=3026206.6666666665, ans=0.04949747468305833
2023-11-24 22:47:01,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3026273.3333333335, ans=0.0
2023-11-24 22:47:02,100 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.84 vs. limit=10.0
2023-11-24 22:47:08,728 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 453950
2023-11-24 22:47:16,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3026340.0, ans=0.1
2023-11-24 22:47:30,937 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.553e+01 8.549e+01 9.034e+01 9.764e+01 1.451e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 22:47:44,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=3026473.3333333335, ans=0.04949747468305833
2023-11-24 22:47:48,112 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3026473.3333333335, ans=0.125
2023-11-24 22:47:49,215 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=3026540.0, ans=0.2
2023-11-24 22:47:50,112 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9100, loss[loss=0.05604, simple_loss=0.07778, pruned_loss=0.008914, audio_tagging_loss=0.008237, over 15606.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09232, pruned_loss=0.01323, audio_tagging_loss=0.008596, over 3049333.87 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:47:50,711 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.22 vs. limit=15.0
2023-11-24 22:48:11,690 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454000
2023-11-24 22:48:27,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.12 vs. limit=12.0
2023-11-24 22:48:30,004 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3026740.0, ans=0.1
2023-11-24 22:48:35,962 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:48:45,802 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0
2023-11-24 22:48:53,404 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9150, loss[loss=0.04757, simple_loss=0.05514, pruned_loss=0.008745, audio_tagging_loss=0.01125, over 14833.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09243, pruned_loss=0.01318, audio_tagging_loss=0.008585, over 3054928.76 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:49:06,928 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3026940.0, ans=0.1
2023-11-24 22:49:12,170 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3026940.0, ans=0.125
2023-11-24 22:49:14,954 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454050
2023-11-24 22:49:36,302 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.421e+01 9.061e+01 9.734e+01 1.251e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 22:49:53,396 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.04 vs. limit=15.0
2023-11-24 22:49:53,585 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-24 22:49:55,219 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9200, loss[loss=0.06591, simple_loss=0.09244, pruned_loss=0.01254, audio_tagging_loss=0.007157, over 15450.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09169, pruned_loss=0.01302, audio_tagging_loss=0.008608, over 3053409.16 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 22:49:58,287 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3027206.6666666665, ans=0.1
2023-11-24 22:50:02,816 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.69 vs. limit=15.0
2023-11-24 22:50:05,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=3027206.6666666665, ans=0.04949747468305833
2023-11-24 22:50:16,191 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454100
2023-11-24 22:50:28,263 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3027340.0, ans=0.125
2023-11-24 22:50:33,091 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3027406.6666666665, ans=0.125
2023-11-24 22:50:33,277 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff2.min_abs, batch_count=3027406.6666666665, ans=0.1
2023-11-24 22:50:57,383 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9250, loss[loss=0.07819, simple_loss=0.1044, pruned_loss=0.01791, audio_tagging_loss=0.008092, over 16625.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09102, pruned_loss=0.01293, audio_tagging_loss=0.008699, over 3055297.20 frames. ], batch size: 61, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:51:18,290 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454150
2023-11-24 22:51:30,259 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=3027673.3333333335, ans=0.125
2023-11-24 22:51:36,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3027740.0, ans=0.125
2023-11-24 22:51:38,360 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3027740.0, ans=0.1
2023-11-24 22:51:41,754 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.619e+01 9.253e+01 1.002e+02 1.218e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 22:51:43,200 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=3027740.0, ans=0.125
2023-11-24 22:51:47,234 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.92 vs. limit=15.0
2023-11-24 22:51:58,230 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9300, loss[loss=0.06387, simple_loss=0.07648, pruned_loss=0.01541, audio_tagging_loss=0.01022, over 14589.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09061, pruned_loss=0.01287, audio_tagging_loss=0.008805, over 3051313.76 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:51:59,579 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=3027873.3333333335, ans=0.125
2023-11-24 22:52:20,063 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454200
2023-11-24 22:52:20,967 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3027940.0, ans=0.125
2023-11-24 22:52:28,529 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.67 vs. limit=10.0
2023-11-24 22:52:52,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=3028140.0, ans=0.1
2023-11-24 22:53:00,878 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9350, loss[loss=0.08593, simple_loss=0.1186, pruned_loss=0.01882, audio_tagging_loss=0.00779, over 16124.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09046, pruned_loss=0.01284, audio_tagging_loss=0.008839, over 3049566.83 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:53:10,435 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3028206.6666666665, ans=0.125
2023-11-24 22:53:14,066 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3028273.3333333335, ans=0.1
2023-11-24 22:53:16,393 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=3028273.3333333335, ans=0.125
2023-11-24 22:53:17,504 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3028273.3333333335, ans=0.125
2023-11-24 22:53:22,127 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454250
2023-11-24 22:53:34,605 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.94 vs. limit=15.0
2023-11-24 22:53:36,211 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.32 vs. limit=10.0
2023-11-24 22:53:44,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3028406.6666666665, ans=0.125
2023-11-24 22:53:45,746 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.804e+01 9.416e+01 1.006e+02 1.917e+02, threshold=1.883e+02, percent-clipped=1.0
2023-11-24 22:53:47,274 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:53:51,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3028473.3333333335, ans=0.0
2023-11-24 22:54:03,455 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9400, loss[loss=0.0697, simple_loss=0.09793, pruned_loss=0.01346, audio_tagging_loss=0.007268, over 16065.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09072, pruned_loss=0.01297, audio_tagging_loss=0.00887, over 3046016.23 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:54:23,273 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=3028606.6666666665, ans=0.125
2023-11-24 22:54:24,248 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454300
2023-11-24 22:54:41,110 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=12.68 vs. limit=22.5
2023-11-24 22:54:54,678 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.06 vs. limit=10.0
2023-11-24 22:55:01,306 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:55:01,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=3028806.6666666665, ans=0.2
2023-11-24 22:55:05,009 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9450, loss[loss=0.0671, simple_loss=0.09005, pruned_loss=0.0127, audio_tagging_loss=0.009376, over 15867.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09146, pruned_loss=0.01306, audio_tagging_loss=0.008863, over 3049596.50 frames. ], batch size: 61, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:55:08,873 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=3028873.3333333335, ans=0.125
2023-11-24 22:55:26,439 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454350
2023-11-24 22:55:49,905 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.559e+01 8.713e+01 9.194e+01 9.823e+01 1.241e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 22:56:03,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3029140.0, ans=0.1
2023-11-24 22:56:07,685 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9500, loss[loss=0.0735, simple_loss=0.1014, pruned_loss=0.01482, audio_tagging_loss=0.00799, over 15356.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09158, pruned_loss=0.01305, audio_tagging_loss=0.008913, over 3041670.79 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:56:29,251 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454400
2023-11-24 22:56:34,524 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3029340.0, ans=0.0
2023-11-24 22:56:36,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=3029340.0, ans=0.125
2023-11-24 22:56:46,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3029406.6666666665, ans=0.2
2023-11-24 22:56:50,541 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=3029406.6666666665, ans=0.0
2023-11-24 22:56:55,636 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.44 vs. limit=12.0
2023-11-24 22:56:57,028 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3029473.3333333335, ans=0.125
2023-11-24 22:57:10,759 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9550, loss[loss=0.0773, simple_loss=0.1095, pruned_loss=0.01264, audio_tagging_loss=0.009882, over 15514.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09087, pruned_loss=0.01286, audio_tagging_loss=0.009037, over 3044934.83 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:57:15,905 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3029540.0, ans=0.1
2023-11-24 22:57:31,117 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454450
2023-11-24 22:57:43,730 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3029673.3333333335, ans=0.125
2023-11-24 22:57:49,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3029740.0, ans=0.0
2023-11-24 22:57:52,152 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3029740.0, ans=0.125
2023-11-24 22:57:54,550 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.49 vs. limit=22.5
2023-11-24 22:57:56,033 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.670e+01 8.429e+01 9.044e+01 9.550e+01 1.234e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 22:58:12,966 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9600, loss[loss=0.08118, simple_loss=0.1045, pruned_loss=0.01813, audio_tagging_loss=0.01079, over 16229.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09059, pruned_loss=0.01283, audio_tagging_loss=0.009144, over 3054560.81 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 22:58:16,963 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3029873.3333333335, ans=0.1
2023-11-24 22:58:20,722 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3029873.3333333335, ans=0.125
2023-11-24 22:58:24,488 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3029940.0, ans=0.125
2023-11-24 22:58:29,504 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.08 vs. limit=15.0
2023-11-24 22:58:33,503 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454500
2023-11-24 22:58:35,411 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3029940.0, ans=0.125
2023-11-24 22:58:35,528 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=3029940.0, ans=0.2
2023-11-24 22:58:56,071 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0
2023-11-24 22:59:07,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3030140.0, ans=0.125
2023-11-24 22:59:14,825 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9650, loss[loss=0.0885, simple_loss=0.1259, pruned_loss=0.01809, audio_tagging_loss=0.007453, over 15116.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08937, pruned_loss=0.01256, audio_tagging_loss=0.009197, over 3048547.60 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 22:59:36,975 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454550
2023-11-24 22:59:55,535 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=3030406.6666666665, ans=0.2
2023-11-24 22:59:56,675 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3030406.6666666665, ans=0.125
2023-11-24 22:59:59,019 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=3030406.6666666665, ans=0.0
2023-11-24 23:00:01,104 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.723e+01 9.168e+01 1.005e+02 1.303e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 23:00:06,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3030473.3333333335, ans=0.1
2023-11-24 23:00:08,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=3030473.3333333335, ans=0.0
2023-11-24 23:00:17,304 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3030540.0, ans=0.0
2023-11-24 23:00:18,270 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9700, loss[loss=0.07252, simple_loss=0.1039, pruned_loss=0.01478, audio_tagging_loss=0.005763, over 15867.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.08962, pruned_loss=0.01272, audio_tagging_loss=0.008971, over 3050383.13 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:00:27,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=3030540.0, ans=0.2
2023-11-24 23:00:38,625 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454600
2023-11-24 23:00:38,725 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3030606.6666666665, ans=0.0
2023-11-24 23:01:06,592 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3030740.0, ans=0.125
2023-11-24 23:01:12,219 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.36 vs. limit=15.0
2023-11-24 23:01:14,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=3030806.6666666665, ans=0.0
2023-11-24 23:01:21,030 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9750, loss[loss=0.06881, simple_loss=0.09799, pruned_loss=0.01401, audio_tagging_loss=0.005797, over 15194.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08978, pruned_loss=0.01274, audio_tagging_loss=0.008842, over 3047267.62 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:01:31,475 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3030873.3333333335, ans=0.1
2023-11-24 23:01:38,663 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3030940.0, ans=0.125
2023-11-24 23:01:39,669 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=3030940.0, ans=0.2
2023-11-24 23:01:41,972 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454650
2023-11-24 23:01:47,425 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.85 vs. limit=22.5
2023-11-24 23:01:55,144 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:01:56,832 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.84 vs. limit=15.0
2023-11-24 23:02:05,582 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3031073.3333333335, ans=0.125
2023-11-24 23:02:06,524 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.511e+01 8.357e+01 8.880e+01 9.692e+01 1.346e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-24 23:02:22,669 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9800, loss[loss=0.07614, simple_loss=0.112, pruned_loss=0.01188, audio_tagging_loss=0.008279, over 15247.00 frames. ], tot_loss[loss=0.06654, simple_loss=0.09002, pruned_loss=0.01276, audio_tagging_loss=0.008772, over 3046141.10 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:02:25,417 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=3031206.6666666665, ans=0.05
2023-11-24 23:02:35,394 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3031273.3333333335, ans=0.0
2023-11-24 23:02:43,517 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3031273.3333333335, ans=0.125
2023-11-24 23:02:44,562 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454700
2023-11-24 23:03:16,426 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:03:19,267 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3031473.3333333335, ans=0.1
2023-11-24 23:03:19,788 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.16 vs. limit=10.0
2023-11-24 23:03:25,452 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9850, loss[loss=0.06466, simple_loss=0.0852, pruned_loss=0.01313, audio_tagging_loss=0.008929, over 15514.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09137, pruned_loss=0.01301, audio_tagging_loss=0.00863, over 3041866.11 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:03:33,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3031540.0, ans=0.1
2023-11-24 23:03:47,064 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454750
2023-11-24 23:04:13,179 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.582e+01 8.592e+01 9.130e+01 1.016e+02 1.240e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 23:04:16,497 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3031806.6666666665, ans=0.0
2023-11-24 23:04:28,562 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9900, loss[loss=0.0773, simple_loss=0.105, pruned_loss=0.01672, audio_tagging_loss=0.008069, over 15362.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09159, pruned_loss=0.01303, audio_tagging_loss=0.008566, over 3047286.24 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:04:28,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=3031873.3333333335, ans=0.95
2023-11-24 23:04:38,476 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3031873.3333333335, ans=0.125
2023-11-24 23:04:42,820 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3031940.0, ans=0.1
2023-11-24 23:04:49,708 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454800
2023-11-24 23:04:55,134 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3032006.6666666665, ans=0.0
2023-11-24 23:05:11,095 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.60 vs. limit=12.0
2023-11-24 23:05:16,151 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:05:23,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3032140.0, ans=0.0
2023-11-24 23:05:31,867 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 9950, loss[loss=0.07863, simple_loss=0.1151, pruned_loss=0.01428, audio_tagging_loss=0.006804, over 15898.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09166, pruned_loss=0.01287, audio_tagging_loss=0.008526, over 3051921.55 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:05:53,000 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454850
2023-11-24 23:06:06,676 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=3032340.0, ans=0.2
2023-11-24 23:06:16,660 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=3032406.6666666665, ans=15.0
2023-11-24 23:06:18,925 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.463e+01 8.965e+01 9.807e+01 1.516e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 23:06:23,902 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3032473.3333333335, ans=0.1
2023-11-24 23:06:33,769 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10000, loss[loss=0.06368, simple_loss=0.08559, pruned_loss=0.01225, audio_tagging_loss=0.00863, over 15955.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09187, pruned_loss=0.013, audio_tagging_loss=0.008517, over 3057081.20 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:06:45,266 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3032606.6666666665, ans=0.0
2023-11-24 23:06:53,948 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3032606.6666666665, ans=0.1
2023-11-24 23:06:54,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454900
2023-11-24 23:06:58,638 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=3032673.3333333335, ans=0.2
2023-11-24 23:07:18,158 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3032740.0, ans=0.125
2023-11-24 23:07:19,358 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:07:28,883 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.22 vs. limit=15.0
2023-11-24 23:07:35,232 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10050, loss[loss=0.09527, simple_loss=0.1222, pruned_loss=0.02585, audio_tagging_loss=0.008322, over 14968.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09211, pruned_loss=0.01314, audio_tagging_loss=0.008497, over 3050836.38 frames. ], batch size: 53, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:07:40,015 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.42 vs. limit=15.0
2023-11-24 23:07:40,077 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=13.90 vs. limit=22.5
2023-11-24 23:07:45,737 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3032873.3333333335, ans=0.0
2023-11-24 23:07:56,657 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 454950
2023-11-24 23:08:15,767 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3033073.3333333335, ans=0.125
2023-11-24 23:08:19,175 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3033073.3333333335, ans=0.1
2023-11-24 23:08:23,654 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.450e+01 9.036e+01 9.785e+01 1.299e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 23:08:31,774 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.02 vs. limit=12.0
2023-11-24 23:08:36,622 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.78 vs. limit=22.5
2023-11-24 23:08:37,232 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10100, loss[loss=0.05553, simple_loss=0.06919, pruned_loss=0.01077, audio_tagging_loss=0.01017, over 14361.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09199, pruned_loss=0.01311, audio_tagging_loss=0.008542, over 3048195.70 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:08:40,405 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3033206.6666666665, ans=0.1
2023-11-24 23:08:48,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3033273.3333333335, ans=0.125
2023-11-24 23:08:59,051 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455000
2023-11-24 23:09:10,699 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3033340.0, ans=0.1
2023-11-24 23:09:11,052 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.05 vs. limit=15.0
2023-11-24 23:09:26,537 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:09:40,103 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10150, loss[loss=0.08119, simple_loss=0.1026, pruned_loss=0.01864, audio_tagging_loss=0.01123, over 14390.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09184, pruned_loss=0.01317, audio_tagging_loss=0.008705, over 3051628.64 frames. ], batch size: 52, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:09:49,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3033540.0, ans=0.125
2023-11-24 23:10:01,311 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455050
2023-11-24 23:10:07,807 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:10:15,612 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=3033673.3333333335, ans=0.125
2023-11-24 23:10:29,123 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.319e+01 8.639e+01 9.334e+01 9.887e+01 1.234e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-24 23:10:31,872 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3033806.6666666665, ans=0.0
2023-11-24 23:10:31,978 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=3033806.6666666665, ans=0.125
2023-11-24 23:10:43,069 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10200, loss[loss=0.08477, simple_loss=0.1137, pruned_loss=0.02089, audio_tagging_loss=0.007047, over 16207.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09075, pruned_loss=0.01281, audio_tagging_loss=0.008748, over 3048848.23 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:10:48,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3033873.3333333335, ans=0.125
2023-11-24 23:10:59,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=3033940.0, ans=0.035
2023-11-24 23:11:04,512 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:11:04,560 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455100
2023-11-24 23:11:11,040 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.71 vs. limit=12.0
2023-11-24 23:11:29,367 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=3034073.3333333335, ans=0.2
2023-11-24 23:11:45,350 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10250, loss[loss=0.08104, simple_loss=0.1045, pruned_loss=0.01857, audio_tagging_loss=0.01021, over 15986.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08999, pruned_loss=0.01278, audio_tagging_loss=0.008901, over 3045301.56 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:11:54,555 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3034206.6666666665, ans=0.1
2023-11-24 23:11:59,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3034273.3333333335, ans=0.125
2023-11-24 23:12:06,520 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455150
2023-11-24 23:12:27,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=3034406.6666666665, ans=0.125
2023-11-24 23:12:33,688 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.500e+01 8.646e+01 9.167e+01 9.835e+01 1.179e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 23:12:40,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3034473.3333333335, ans=0.125
2023-11-24 23:12:47,006 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10300, loss[loss=0.07501, simple_loss=0.1061, pruned_loss=0.01482, audio_tagging_loss=0.007155, over 16388.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.09006, pruned_loss=0.01276, audio_tagging_loss=0.008991, over 3047994.39 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:12:53,332 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-24 23:12:58,011 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3034540.0, ans=0.125
2023-11-24 23:13:01,780 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:13:02,956 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3034606.6666666665, ans=0.0
2023-11-24 23:13:04,148 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3034606.6666666665, ans=0.125
2023-11-24 23:13:07,818 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3034606.6666666665, ans=0.125
2023-11-24 23:13:08,719 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455200
2023-11-24 23:13:23,969 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=3034740.0, ans=0.2
2023-11-24 23:13:27,005 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3034740.0, ans=0.0
2023-11-24 23:13:32,883 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3034740.0, ans=0.1
2023-11-24 23:13:40,606 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=3034806.6666666665, ans=0.2
2023-11-24 23:13:43,698 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3034806.6666666665, ans=0.125
2023-11-24 23:13:47,245 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3034806.6666666665, ans=0.125
2023-11-24 23:13:48,378 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:13:50,500 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10350, loss[loss=0.06877, simple_loss=0.09651, pruned_loss=0.01133, audio_tagging_loss=0.009179, over 15114.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.09001, pruned_loss=0.0127, audio_tagging_loss=0.00909, over 3051160.48 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:13:52,052 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3034873.3333333335, ans=0.1
2023-11-24 23:14:01,866 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.04 vs. limit=15.0
2023-11-24 23:14:11,330 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455250
2023-11-24 23:14:13,749 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.95 vs. limit=15.0
2023-11-24 23:14:27,694 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3035073.3333333335, ans=0.125
2023-11-24 23:14:33,270 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.56 vs. limit=15.0
2023-11-24 23:14:36,256 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3035073.3333333335, ans=0.0
2023-11-24 23:14:38,308 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.693e+01 9.178e+01 1.025e+02 1.318e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 23:14:45,700 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3035140.0, ans=0.1
2023-11-24 23:14:49,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=3035140.0, ans=0.04949747468305833
2023-11-24 23:14:49,672 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.29 vs. limit=15.0
2023-11-24 23:14:51,328 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10400, loss[loss=0.04679, simple_loss=0.06075, pruned_loss=0.007643, audio_tagging_loss=0.008775, over 14920.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08904, pruned_loss=0.01253, audio_tagging_loss=0.009257, over 3046990.17 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:14:57,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3035206.6666666665, ans=0.125
2023-11-24 23:15:04,765 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=3035273.3333333335, ans=0.0
2023-11-24 23:15:09,886 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.28 vs. limit=15.0
2023-11-24 23:15:11,783 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=3035273.3333333335, ans=0.035
2023-11-24 23:15:12,892 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455300
2023-11-24 23:15:14,766 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3035273.3333333335, ans=0.0
2023-11-24 23:15:22,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3035340.0, ans=0.125
2023-11-24 23:15:49,044 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=3035473.3333333335, ans=0.2
2023-11-24 23:15:53,555 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10450, loss[loss=0.05702, simple_loss=0.07009, pruned_loss=0.01061, audio_tagging_loss=0.01136, over 16017.00 frames. ], tot_loss[loss=0.0664, simple_loss=0.08932, pruned_loss=0.01258, audio_tagging_loss=0.009165, over 3047612.56 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:15:53,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=3035540.0, ans=0.05
2023-11-24 23:16:14,898 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455350
2023-11-24 23:16:18,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3035673.3333333335, ans=0.125
2023-11-24 23:16:19,929 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3035673.3333333335, ans=0.125
2023-11-24 23:16:42,052 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.886e+01 9.517e+01 1.018e+02 1.385e+02, threshold=1.903e+02, percent-clipped=0.0
2023-11-24 23:16:43,671 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3035806.6666666665, ans=0.0
2023-11-24 23:16:56,068 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10500, loss[loss=0.06454, simple_loss=0.09184, pruned_loss=0.01351, audio_tagging_loss=0.005116, over 15658.00 frames. ], tot_loss[loss=0.06582, simple_loss=0.08858, pruned_loss=0.01249, audio_tagging_loss=0.009036, over 3043601.68 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:16:57,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3035873.3333333335, ans=0.1
2023-11-24 23:17:06,976 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=3035940.0, ans=0.125
2023-11-24 23:17:07,626 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-24 23:17:09,437 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3035940.0, ans=0.125
2023-11-24 23:17:11,851 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3035940.0, ans=0.1
2023-11-24 23:17:16,580 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455400
2023-11-24 23:17:22,319 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3036006.6666666665, ans=0.125
2023-11-24 23:17:45,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3036140.0, ans=0.125
2023-11-24 23:17:54,844 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3036140.0, ans=0.0
2023-11-24 23:17:54,857 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3036140.0, ans=0.125
2023-11-24 23:17:58,260 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10550, loss[loss=0.06531, simple_loss=0.08412, pruned_loss=0.01652, audio_tagging_loss=0.006726, over 16063.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08936, pruned_loss=0.01274, audio_tagging_loss=0.008935, over 3039264.00 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:17:58,960 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.59 vs. limit=22.5
2023-11-24 23:18:03,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3036206.6666666665, ans=0.0
2023-11-24 23:18:18,521 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3036273.3333333335, ans=0.1
2023-11-24 23:18:19,453 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455450
2023-11-24 23:18:25,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3036340.0, ans=0.0
2023-11-24 23:18:30,226 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=3036340.0, ans=0.5
2023-11-24 23:18:37,458 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=3036406.6666666665, ans=10.0
2023-11-24 23:18:46,506 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.547e+01 9.162e+01 9.903e+01 1.156e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 23:19:00,180 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10600, loss[loss=0.06808, simple_loss=0.09191, pruned_loss=0.01226, audio_tagging_loss=0.00987, over 14873.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.0902, pruned_loss=0.01289, audio_tagging_loss=0.008774, over 3044335.83 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:19:04,208 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.39 vs. limit=12.0
2023-11-24 23:19:07,867 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.53 vs. limit=15.0
2023-11-24 23:19:09,539 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=3036540.0, ans=0.2
2023-11-24 23:19:11,792 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.73 vs. limit=15.0
2023-11-24 23:19:19,109 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3036606.6666666665, ans=0.125
2023-11-24 23:19:22,455 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455500
2023-11-24 23:19:23,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3036606.6666666665, ans=0.1
2023-11-24 23:19:24,283 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.73 vs. limit=15.0
2023-11-24 23:19:29,652 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=3036673.3333333335, ans=0.0
2023-11-24 23:20:03,938 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10650, loss[loss=0.07203, simple_loss=0.09809, pruned_loss=0.01365, audio_tagging_loss=0.009339, over 14815.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.0903, pruned_loss=0.01287, audio_tagging_loss=0.008737, over 3043848.97 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:20:06,523 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3036873.3333333335, ans=0.0
2023-11-24 23:20:14,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=3036940.0, ans=0.05
2023-11-24 23:20:24,017 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455550
2023-11-24 23:20:35,249 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=3037006.6666666665, ans=0.125
2023-11-24 23:20:38,984 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=3037073.3333333335, ans=0.04949747468305833
2023-11-24 23:20:52,472 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.714e+01 9.529e+01 1.031e+02 1.468e+02, threshold=1.906e+02, percent-clipped=0.0
2023-11-24 23:21:05,483 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10700, loss[loss=0.07988, simple_loss=0.1111, pruned_loss=0.0183, audio_tagging_loss=0.006006, over 15604.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09017, pruned_loss=0.01293, audio_tagging_loss=0.008788, over 3040175.23 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:21:08,038 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=3037206.6666666665, ans=0.125
2023-11-24 23:21:08,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3037206.6666666665, ans=0.0
2023-11-24 23:21:12,881 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=3037206.6666666665, ans=0.0
2023-11-24 23:21:26,149 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455600
2023-11-24 23:21:26,153 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=3037273.3333333335, ans=0.125
2023-11-24 23:21:37,759 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=3037340.0, ans=0.07
2023-11-24 23:21:51,366 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.65 vs. limit=12.0
2023-11-24 23:22:00,310 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3037473.3333333335, ans=0.125
2023-11-24 23:22:07,790 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10750, loss[loss=0.04312, simple_loss=0.05677, pruned_loss=0.005452, audio_tagging_loss=0.009285, over 15335.00 frames. ], tot_loss[loss=0.06648, simple_loss=0.08996, pruned_loss=0.01277, audio_tagging_loss=0.008732, over 3050688.89 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:22:29,818 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455650
2023-11-24 23:22:39,474 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3037673.3333333335, ans=0.0
2023-11-24 23:22:48,868 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3037740.0, ans=0.125
2023-11-24 23:22:56,323 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.544e+01 9.121e+01 9.679e+01 1.132e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 23:23:02,584 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=3037806.6666666665, ans=0.125
2023-11-24 23:23:02,611 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=3037806.6666666665, ans=0.0
2023-11-24 23:23:10,659 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10800, loss[loss=0.06563, simple_loss=0.08915, pruned_loss=0.0129, audio_tagging_loss=0.008146, over 15342.00 frames. ], tot_loss[loss=0.06607, simple_loss=0.08906, pruned_loss=0.0127, audio_tagging_loss=0.00885, over 3049836.43 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:23:21,426 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3037940.0, ans=0.125
2023-11-24 23:23:27,743 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.05 vs. limit=15.0
2023-11-24 23:23:28,500 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=3037940.0, ans=0.2
2023-11-24 23:23:30,757 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455700
2023-11-24 23:24:11,711 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10850, loss[loss=0.04894, simple_loss=0.06676, pruned_loss=0.00702, audio_tagging_loss=0.008544, over 14659.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09039, pruned_loss=0.01288, audio_tagging_loss=0.008738, over 3058555.04 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:24:17,866 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3038206.6666666665, ans=0.125
2023-11-24 23:24:20,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3038206.6666666665, ans=0.125
2023-11-24 23:24:29,001 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3038273.3333333335, ans=0.125
2023-11-24 23:24:32,482 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455750
2023-11-24 23:24:59,976 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.004e+01 8.552e+01 9.146e+01 9.923e+01 1.188e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 23:25:07,127 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:25:10,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3038473.3333333335, ans=0.0
2023-11-24 23:25:13,537 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10900, loss[loss=0.06196, simple_loss=0.08465, pruned_loss=0.01058, audio_tagging_loss=0.00906, over 15374.00 frames. ], tot_loss[loss=0.0663, simple_loss=0.08973, pruned_loss=0.0127, audio_tagging_loss=0.008734, over 3053639.54 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:25:23,166 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3038540.0, ans=0.0
2023-11-24 23:25:24,320 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=3038606.6666666665, ans=0.0
2023-11-24 23:25:35,529 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455800
2023-11-24 23:25:38,388 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=3038673.3333333335, ans=0.0
2023-11-24 23:25:39,553 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=3038673.3333333335, ans=0.0
2023-11-24 23:25:51,653 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=3038740.0, ans=0.125
2023-11-24 23:26:09,065 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.01 vs. limit=15.0
2023-11-24 23:26:14,605 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3038873.3333333335, ans=0.125
2023-11-24 23:26:16,209 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 10950, loss[loss=0.06127, simple_loss=0.07982, pruned_loss=0.01127, audio_tagging_loss=0.01009, over 16331.00 frames. ], tot_loss[loss=0.06633, simple_loss=0.08961, pruned_loss=0.01272, audio_tagging_loss=0.0088, over 3054846.39 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:26:19,509 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3038873.3333333335, ans=0.0
2023-11-24 23:26:22,927 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3038873.3333333335, ans=0.125
2023-11-24 23:26:32,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3038940.0, ans=0.1
2023-11-24 23:26:36,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=3038940.0, ans=0.125
2023-11-24 23:26:37,320 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455850
2023-11-24 23:26:47,081 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3039006.6666666665, ans=0.0
2023-11-24 23:26:57,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3039073.3333333335, ans=0.125
2023-11-24 23:27:05,052 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.762e+01 9.251e+01 9.897e+01 1.242e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 23:27:18,785 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11000, loss[loss=0.05867, simple_loss=0.0797, pruned_loss=0.009786, audio_tagging_loss=0.009035, over 14998.00 frames. ], tot_loss[loss=0.06614, simple_loss=0.08913, pruned_loss=0.01267, audio_tagging_loss=0.00891, over 3053459.90 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:27:26,049 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:27:27,589 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=3039206.6666666665, ans=0.025
2023-11-24 23:27:39,715 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455900
2023-11-24 23:27:46,033 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.99 vs. limit=15.0
2023-11-24 23:27:46,232 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.21 vs. limit=15.0
2023-11-24 23:27:55,178 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:28:01,628 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=3039406.6666666665, ans=0.125
2023-11-24 23:28:02,789 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3039406.6666666665, ans=0.125
2023-11-24 23:28:09,869 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3039473.3333333335, ans=0.1
2023-11-24 23:28:11,029 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3039473.3333333335, ans=0.125
2023-11-24 23:28:14,829 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=3039473.3333333335, ans=0.0
2023-11-24 23:28:20,379 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11050, loss[loss=0.0888, simple_loss=0.1182, pruned_loss=0.02071, audio_tagging_loss=0.008968, over 15523.00 frames. ], tot_loss[loss=0.06665, simple_loss=0.08993, pruned_loss=0.01282, audio_tagging_loss=0.008857, over 3056451.30 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:28:38,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3039606.6666666665, ans=0.125
2023-11-24 23:28:42,212 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 455950
2023-11-24 23:28:45,015 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=3039673.3333333335, ans=0.2
2023-11-24 23:28:57,220 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3039740.0, ans=0.125
2023-11-24 23:29:01,130 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.68 vs. limit=12.0
2023-11-24 23:29:08,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3039740.0, ans=0.125
2023-11-24 23:29:08,950 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.642e+01 9.315e+01 1.003e+02 1.536e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 23:29:22,454 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11100, loss[loss=0.08308, simple_loss=0.1097, pruned_loss=0.01945, audio_tagging_loss=0.008759, over 15031.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09067, pruned_loss=0.01289, audio_tagging_loss=0.008934, over 3055347.48 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:29:42,313 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3039940.0, ans=0.1
2023-11-24 23:29:42,392 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=3039940.0, ans=0.0
2023-11-24 23:29:42,897 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.34 vs. limit=15.0
2023-11-24 23:29:44,640 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456000
2023-11-24 23:29:58,602 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3040006.6666666665, ans=0.0
2023-11-24 23:30:07,396 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3040073.3333333335, ans=0.0
2023-11-24 23:30:30,068 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11150, loss[loss=0.05653, simple_loss=0.07241, pruned_loss=0.0107, audio_tagging_loss=0.009626, over 15756.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08919, pruned_loss=0.01268, audio_tagging_loss=0.009075, over 3053403.58 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:30:30,233 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=3040206.6666666665, ans=0.125
2023-11-24 23:30:42,230 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=3040273.3333333335, ans=0.0
2023-11-24 23:30:46,923 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=3040273.3333333335, ans=0.125
2023-11-24 23:30:47,024 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0
2023-11-24 23:30:50,891 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456050
2023-11-24 23:30:51,034 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3040273.3333333335, ans=0.125
2023-11-24 23:30:52,183 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3040273.3333333335, ans=0.125
2023-11-24 23:31:01,840 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=3040340.0, ans=0.125
2023-11-24 23:31:08,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3040406.6666666665, ans=0.125
2023-11-24 23:31:18,464 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.591e+01 9.320e+01 9.957e+01 1.253e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 23:31:29,725 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0
2023-11-24 23:31:31,379 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11200, loss[loss=0.06776, simple_loss=0.09142, pruned_loss=0.01395, audio_tagging_loss=0.008098, over 15784.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09006, pruned_loss=0.01275, audio_tagging_loss=0.00904, over 3054107.53 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:31:32,854 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3040540.0, ans=0.125
2023-11-24 23:31:33,049 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.87 vs. limit=15.0
2023-11-24 23:31:34,261 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.18 vs. limit=12.0
2023-11-24 23:31:47,522 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3040606.6666666665, ans=0.125
2023-11-24 23:31:53,214 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456100
2023-11-24 23:32:14,455 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=3040740.0, ans=0.0
2023-11-24 23:32:27,951 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff3.min_abs, batch_count=3040806.6666666665, ans=0.2
2023-11-24 23:32:33,945 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11250, loss[loss=0.05962, simple_loss=0.07785, pruned_loss=0.01137, audio_tagging_loss=0.009318, over 14539.00 frames. ], tot_loss[loss=0.06625, simple_loss=0.08903, pruned_loss=0.0127, audio_tagging_loss=0.009039, over 3050449.51 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:32:35,787 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=15.0
2023-11-24 23:32:40,120 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=3040873.3333333335, ans=0.2
2023-11-24 23:32:48,642 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.54 vs. limit=10.0
2023-11-24 23:32:49,730 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=22.5
2023-11-24 23:32:54,342 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3040940.0, ans=0.1
2023-11-24 23:32:55,766 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456150
2023-11-24 23:33:12,168 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3041073.3333333335, ans=0.125
2023-11-24 23:33:17,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3041073.3333333335, ans=0.1
2023-11-24 23:33:23,492 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.642e+01 8.456e+01 9.058e+01 9.747e+01 1.230e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 23:33:30,610 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0
2023-11-24 23:33:35,240 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.19 vs. limit=22.5
2023-11-24 23:33:35,963 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11300, loss[loss=0.06819, simple_loss=0.09393, pruned_loss=0.0121, audio_tagging_loss=0.00912, over 14959.00 frames. ], tot_loss[loss=0.06616, simple_loss=0.08901, pruned_loss=0.0127, audio_tagging_loss=0.008954, over 3043362.55 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:33:43,861 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3041206.6666666665, ans=0.0
2023-11-24 23:33:56,671 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456200
2023-11-24 23:34:04,860 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3041340.0, ans=0.125
2023-11-24 23:34:20,875 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=3041406.6666666665, ans=0.09899494936611666
2023-11-24 23:34:25,514 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3041473.3333333335, ans=0.1
2023-11-24 23:34:29,374 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.75 vs. limit=15.0
2023-11-24 23:34:37,469 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11350, loss[loss=0.06627, simple_loss=0.08632, pruned_loss=0.0128, audio_tagging_loss=0.01032, over 15771.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08944, pruned_loss=0.01284, audio_tagging_loss=0.008788, over 3038955.66 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:34:47,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=3041540.0, ans=0.2
2023-11-24 23:34:58,578 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456250
2023-11-24 23:35:08,890 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3041673.3333333335, ans=0.1
2023-11-24 23:35:28,350 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.570e+01 8.600e+01 9.248e+01 9.972e+01 1.236e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 23:35:30,931 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=3041806.6666666665, ans=0.2
2023-11-24 23:35:39,411 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11400, loss[loss=0.06442, simple_loss=0.08133, pruned_loss=0.01422, audio_tagging_loss=0.009527, over 14437.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.09045, pruned_loss=0.01285, audio_tagging_loss=0.008678, over 3040652.72 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:35:44,999 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.48 vs. limit=15.0
2023-11-24 23:35:50,986 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3041940.0, ans=0.0
2023-11-24 23:35:52,339 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3041940.0, ans=0.1
2023-11-24 23:36:00,201 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456300
2023-11-24 23:36:20,697 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3042073.3333333335, ans=0.1
2023-11-24 23:36:22,082 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3042073.3333333335, ans=0.0
2023-11-24 23:36:28,391 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.91 vs. limit=10.0
2023-11-24 23:36:29,547 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.63 vs. limit=12.0
2023-11-24 23:36:39,135 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3042140.0, ans=0.1
2023-11-24 23:36:41,453 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11450, loss[loss=0.0638, simple_loss=0.08582, pruned_loss=0.01168, audio_tagging_loss=0.009215, over 14552.00 frames. ], tot_loss[loss=0.06643, simple_loss=0.09012, pruned_loss=0.01272, audio_tagging_loss=0.008651, over 3040349.94 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:36:49,901 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3042206.6666666665, ans=0.125
2023-11-24 23:37:02,338 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456350
2023-11-24 23:37:04,938 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3042340.0, ans=0.125
2023-11-24 23:37:09,184 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.05 vs. limit=12.0
2023-11-24 23:37:14,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3042340.0, ans=0.125
2023-11-24 23:37:16,055 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3042340.0, ans=0.125
2023-11-24 23:37:16,096 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3042340.0, ans=0.0
2023-11-24 23:37:17,124 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3042406.6666666665, ans=0.125
2023-11-24 23:37:22,652 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.11 vs. limit=10.0
2023-11-24 23:37:31,586 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.517e+01 9.262e+01 1.018e+02 1.314e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 23:37:35,534 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3042473.3333333335, ans=0.125
2023-11-24 23:37:42,258 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11500, loss[loss=0.0655, simple_loss=0.07905, pruned_loss=0.01247, audio_tagging_loss=0.01351, over 14459.00 frames. ], tot_loss[loss=0.06632, simple_loss=0.08988, pruned_loss=0.01266, audio_tagging_loss=0.008726, over 3049108.06 frames. ], batch size: 53, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:37:43,762 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=3042540.0, ans=0.125
2023-11-24 23:37:47,962 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3042540.0, ans=0.125
2023-11-24 23:37:55,717 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=3042606.6666666665, ans=0.125
2023-11-24 23:37:56,777 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:38:03,711 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456400
2023-11-24 23:38:37,512 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.10 vs. limit=6.0
2023-11-24 23:38:44,371 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11550, loss[loss=0.09011, simple_loss=0.1137, pruned_loss=0.0235, audio_tagging_loss=0.009761, over 14278.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.09036, pruned_loss=0.01275, audio_tagging_loss=0.008739, over 3057043.94 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:39:05,142 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456450
2023-11-24 23:39:10,102 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3043006.6666666665, ans=0.125
2023-11-24 23:39:19,197 WARNING [train_asr.py:1462] (2/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:39:24,907 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3043073.3333333335, ans=0.125
2023-11-24 23:39:32,493 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3043140.0, ans=0.125
2023-11-24 23:39:34,490 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.770e+01 9.475e+01 1.009e+02 1.231e+02, threshold=1.895e+02, percent-clipped=0.0
2023-11-24 23:39:35,895 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=3043140.0, ans=0.5
2023-11-24 23:39:42,558 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=3043140.0, ans=0.2
2023-11-24 23:39:45,425 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.63 vs. limit=22.5
2023-11-24 23:39:45,833 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11600, loss[loss=0.06462, simple_loss=0.08762, pruned_loss=0.01249, audio_tagging_loss=0.008321, over 16528.00 frames. ], tot_loss[loss=0.06657, simple_loss=0.09036, pruned_loss=0.01269, audio_tagging_loss=0.008699, over 3059722.49 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:39:58,197 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3043273.3333333335, ans=0.125
2023-11-24 23:40:06,274 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456500
2023-11-24 23:40:11,122 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3043340.0, ans=0.125
2023-11-24 23:40:23,703 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=3043406.6666666665, ans=0.0
2023-11-24 23:40:25,249 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.34 vs. limit=15.0
2023-11-24 23:40:46,563 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11650, loss[loss=0.06248, simple_loss=0.08541, pruned_loss=0.00929, audio_tagging_loss=0.01048, over 16714.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.09058, pruned_loss=0.01282, audio_tagging_loss=0.008682, over 3053467.36 frames. ], batch size: 66, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:40:48,045 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3043540.0, ans=0.125
2023-11-24 23:41:08,008 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456550
2023-11-24 23:41:14,683 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3043673.3333333335, ans=0.1
2023-11-24 23:41:36,902 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.669e+01 8.725e+01 9.251e+01 1.014e+02 1.338e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 23:41:48,070 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11700, loss[loss=0.06318, simple_loss=0.08857, pruned_loss=0.01122, audio_tagging_loss=0.007673, over 15001.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08958, pruned_loss=0.01284, audio_tagging_loss=0.008721, over 3051166.93 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:41:53,569 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3043873.3333333335, ans=0.1
2023-11-24 23:41:54,795 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=3043873.3333333335, ans=0.2
2023-11-24 23:42:09,636 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456600
2023-11-24 23:42:28,159 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.69 vs. limit=15.0
2023-11-24 23:42:30,285 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3044073.3333333335, ans=0.1
2023-11-24 23:42:39,808 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=3044140.0, ans=0.125
2023-11-24 23:42:43,248 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=3044140.0, ans=0.125
2023-11-24 23:42:48,686 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3044140.0, ans=0.0
2023-11-24 23:42:50,850 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11750, loss[loss=0.05689, simple_loss=0.06884, pruned_loss=0.01144, audio_tagging_loss=0.01103, over 15465.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.08962, pruned_loss=0.01289, audio_tagging_loss=0.008796, over 3053375.48 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:42:57,146 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3044206.6666666665, ans=0.125
2023-11-24 23:42:59,377 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3044206.6666666665, ans=0.0
2023-11-24 23:43:11,140 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456650
2023-11-24 23:43:12,451 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=3044273.3333333335, ans=0.1
2023-11-24 23:43:14,888 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=3044340.0, ans=0.0
2023-11-24 23:43:32,572 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=3044406.6666666665, ans=0.0
2023-11-24 23:43:39,167 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3044473.3333333335, ans=0.125
2023-11-24 23:43:40,164 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=3044473.3333333335, ans=0.035
2023-11-24 23:43:42,311 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.689e+01 9.260e+01 1.011e+02 1.469e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 23:43:51,928 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11800, loss[loss=0.07016, simple_loss=0.09846, pruned_loss=0.012, audio_tagging_loss=0.008924, over 15506.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.0899, pruned_loss=0.01288, audio_tagging_loss=0.00878, over 3047238.27 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:43:55,950 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=3044540.0, ans=0.125
2023-11-24 23:44:00,647 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=3044540.0, ans=0.0
2023-11-24 23:44:03,035 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3044606.6666666665, ans=0.0
2023-11-24 23:44:12,653 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456700
2023-11-24 23:44:14,915 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=13.92 vs. limit=22.5
2023-11-24 23:44:34,003 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3044740.0, ans=0.2
2023-11-24 23:44:37,386 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3044740.0, ans=0.0
2023-11-24 23:44:49,746 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=6.48 vs. limit=15.0
2023-11-24 23:44:52,991 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11850, loss[loss=0.08475, simple_loss=0.1167, pruned_loss=0.01739, audio_tagging_loss=0.009026, over 15350.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09039, pruned_loss=0.01291, audio_tagging_loss=0.008899, over 3042367.20 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:45:14,837 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456750
2023-11-24 23:45:44,601 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.737e+01 9.408e+01 9.939e+01 1.281e+02, threshold=1.882e+02, percent-clipped=0.0
2023-11-24 23:45:54,844 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11900, loss[loss=0.07165, simple_loss=0.1056, pruned_loss=0.01077, audio_tagging_loss=0.008098, over 17144.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09022, pruned_loss=0.01278, audio_tagging_loss=0.009051, over 3040972.46 frames. ], batch size: 64, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:46:11,012 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=3045273.3333333335, ans=0.0
2023-11-24 23:46:11,083 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3045273.3333333335, ans=0.125
2023-11-24 23:46:15,632 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456800
2023-11-24 23:46:22,143 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3045340.0, ans=0.1
2023-11-24 23:46:27,670 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-24 23:46:30,710 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3045406.6666666665, ans=0.125
2023-11-24 23:46:34,264 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3045406.6666666665, ans=0.0
2023-11-24 23:46:35,617 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=3045406.6666666665, ans=0.5
2023-11-24 23:46:56,509 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 11950, loss[loss=0.07413, simple_loss=0.09633, pruned_loss=0.01499, audio_tagging_loss=0.01098, over 14786.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09055, pruned_loss=0.01279, audio_tagging_loss=0.009117, over 3037565.89 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:47:08,459 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=3045606.6666666665, ans=0.2
2023-11-24 23:47:14,369 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=3045606.6666666665, ans=22.5
2023-11-24 23:47:17,208 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456850
2023-11-24 23:47:47,342 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.445e+01 8.455e+01 9.152e+01 9.873e+01 1.157e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 23:47:49,960 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3045806.6666666665, ans=0.0
2023-11-24 23:47:55,603 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3045873.3333333335, ans=0.125
2023-11-24 23:47:56,417 INFO [train_asr.py:1221] (2/4) Epoch 38, batch 12000, loss[loss=0.09118, simple_loss=0.1315, pruned_loss=0.01811, audio_tagging_loss=0.007337, over 16286.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09123, pruned_loss=0.01295, audio_tagging_loss=0.009152, over 3042095.41 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:47:56,417 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 23:48:40,308 INFO [train_asr.py:1253] (2/4) Epoch 38, validation: loss=0.05738, simple_loss=0.0508, pruned_loss=0.005195, audio_tagging_loss=0.02678, over 4681554.00 frames. 
2023-11-24 23:48:40,309 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 23:48:54,497 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.79 vs. limit=22.5
2023-11-24 23:48:59,665 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456900
2023-11-24 23:48:59,911 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3045940.0, ans=0.125
2023-11-24 23:49:37,958 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 0, loss[loss=0.06887, simple_loss=0.07771, pruned_loss=0.008438, audio_tagging_loss=0.02158, over 15299.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.07771, pruned_loss=0.008438, audio_tagging_loss=0.02158, over 15299.00 frames. ], batch size: 56, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:49:37,959 INFO [train_asr.py:1244] (2/4) Computing validation loss
2023-11-24 23:50:14,454 INFO [train_asr.py:1253] (2/4) Epoch 39, validation: loss=0.0578, simple_loss=0.05083, pruned_loss=0.005244, audio_tagging_loss=0.02714, over 4681554.00 frames. 
2023-11-24 23:50:14,455 INFO [train_asr.py:1254] (2/4) Maximum memory allocated so far is 26054MB
2023-11-24 23:50:14,752 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3046020.0, ans=0.1
2023-11-24 23:50:15,935 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=3046020.0, ans=0.0
2023-11-24 23:50:29,615 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-24 23:50:37,480 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3046153.3333333335, ans=0.0
2023-11-24 23:50:41,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3046153.3333333335, ans=0.125
2023-11-24 23:50:42,803 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3046153.3333333335, ans=0.125
2023-11-24 23:50:48,903 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3046153.3333333335, ans=0.125
2023-11-24 23:51:09,883 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 456950
2023-11-24 23:51:15,703 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 50, loss[loss=0.111, simple_loss=0.1414, pruned_loss=0.02859, audio_tagging_loss=0.01176, over 15310.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.092, pruned_loss=0.01275, audio_tagging_loss=0.01663, over 685981.29 frames. ], batch size: 54, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:51:39,932 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 8.015e+01 9.557e+01 1.022e+02 1.093e+02 1.810e+02, threshold=2.044e+02, percent-clipped=0.0
2023-11-24 23:51:43,025 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3046486.6666666665, ans=0.1
2023-11-24 23:51:59,839 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=3046553.3333333335, ans=0.07
2023-11-24 23:52:11,429 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457000
2023-11-24 23:52:18,509 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 100, loss[loss=0.07759, simple_loss=0.104, pruned_loss=0.01224, audio_tagging_loss=0.01334, over 16062.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09017, pruned_loss=0.0125, audio_tagging_loss=0.01626, over 1208058.76 frames. ], batch size: 61, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:52:20,405 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=24.25 vs. limit=22.5
2023-11-24 23:52:28,389 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=3046686.6666666665, ans=0.0
2023-11-24 23:52:35,580 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3046753.3333333335, ans=0.125
2023-11-24 23:53:06,595 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=14.74 vs. limit=22.5
2023-11-24 23:53:11,244 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=3046953.3333333335, ans=0.025
2023-11-24 23:53:14,570 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457050
2023-11-24 23:53:14,826 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3046953.3333333335, ans=0.0
2023-11-24 23:53:21,496 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 150, loss[loss=0.08634, simple_loss=0.1032, pruned_loss=0.02125, audio_tagging_loss=0.01346, over 14670.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.08999, pruned_loss=0.01246, audio_tagging_loss=0.01466, over 1614530.96 frames. ], batch size: 54, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:53:45,772 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.772e+01 8.928e+01 9.394e+01 1.010e+02 1.309e+02, threshold=1.879e+02, percent-clipped=0.0
2023-11-24 23:53:49,046 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=3047153.3333333335, ans=0.125
2023-11-24 23:53:56,887 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.31 vs. limit=22.5
2023-11-24 23:54:18,136 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457100
2023-11-24 23:54:24,057 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 200, loss[loss=0.06921, simple_loss=0.09658, pruned_loss=0.01252, audio_tagging_loss=0.008401, over 16290.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09049, pruned_loss=0.01265, audio_tagging_loss=0.01295, over 1935536.91 frames. ], batch size: 59, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:54:46,364 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.72 vs. limit=15.0
2023-11-24 23:54:46,454 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.73 vs. limit=15.0
2023-11-24 23:54:48,554 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3047486.6666666665, ans=0.125
2023-11-24 23:55:19,666 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457150
2023-11-24 23:55:22,089 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3047620.0, ans=0.125
2023-11-24 23:55:26,005 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 250, loss[loss=0.06822, simple_loss=0.09106, pruned_loss=0.01635, audio_tagging_loss=0.006346, over 14672.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.0902, pruned_loss=0.01255, audio_tagging_loss=0.01172, over 2180284.07 frames. ], batch size: 55, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:55:29,863 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3047686.6666666665, ans=0.125
2023-11-24 23:55:31,101 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3047686.6666666665, ans=0.125
2023-11-24 23:55:38,024 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3047753.3333333335, ans=0.2
2023-11-24 23:55:49,773 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=3047820.0, ans=0.05
2023-11-24 23:55:50,701 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.824e+01 9.475e+01 1.039e+02 1.300e+02, threshold=1.895e+02, percent-clipped=0.0
2023-11-24 23:56:09,462 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=3047886.6666666665, ans=0.02
2023-11-24 23:56:13,614 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=3047886.6666666665, ans=0.0
2023-11-24 23:56:21,556 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457200
2023-11-24 23:56:27,929 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.62 vs. limit=15.0
2023-11-24 23:56:28,291 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 300, loss[loss=0.05131, simple_loss=0.06082, pruned_loss=0.01062, audio_tagging_loss=0.01028, over 15012.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.08981, pruned_loss=0.01259, audio_tagging_loss=0.0108, over 2374051.94 frames. ], batch size: 58, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:56:30,290 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3048020.0, ans=0.1
2023-11-24 23:56:34,889 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3048020.0, ans=0.1
2023-11-24 23:56:53,331 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3048153.3333333335, ans=0.1
2023-11-24 23:56:54,626 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3048153.3333333335, ans=0.0
2023-11-24 23:57:01,108 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3048153.3333333335, ans=0.0
2023-11-24 23:57:09,333 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=3048220.0, ans=0.125
2023-11-24 23:57:19,157 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.91 vs. limit=15.0
2023-11-24 23:57:24,677 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457250
2023-11-24 23:57:28,225 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3048286.6666666665, ans=0.125
2023-11-24 23:57:30,464 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 350, loss[loss=0.05872, simple_loss=0.07368, pruned_loss=0.01135, audio_tagging_loss=0.01053, over 14794.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09021, pruned_loss=0.01257, audio_tagging_loss=0.01033, over 2520952.46 frames. ], batch size: 60, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:57:39,649 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3048353.3333333335, ans=0.0
2023-11-24 23:57:39,819 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.98 vs. limit=6.0
2023-11-24 23:57:56,160 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.715e+01 9.321e+01 9.899e+01 1.393e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 23:58:02,981 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3048486.6666666665, ans=0.125
2023-11-24 23:58:10,441 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3048553.3333333335, ans=0.1
2023-11-24 23:58:22,625 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=3048620.0, ans=0.05
2023-11-24 23:58:25,281 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.22 vs. limit=22.5
2023-11-24 23:58:25,983 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457300
2023-11-24 23:58:32,358 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 400, loss[loss=0.04804, simple_loss=0.05485, pruned_loss=0.009582, audio_tagging_loss=0.01103, over 14786.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09183, pruned_loss=0.01295, audio_tagging_loss=0.00988, over 2641419.22 frames. ], batch size: 60, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:58:45,848 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.30 vs. limit=22.5
2023-11-24 23:58:56,198 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=3048820.0, ans=0.0
2023-11-24 23:59:14,477 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=3048886.6666666665, ans=0.0
2023-11-24 23:59:18,007 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3048886.6666666665, ans=0.125
2023-11-24 23:59:27,945 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457350
2023-11-24 23:59:30,574 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3048953.3333333335, ans=0.125
2023-11-24 23:59:31,531 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=3048953.3333333335, ans=0.125
2023-11-24 23:59:31,723 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=3048953.3333333335, ans=0.0
2023-11-24 23:59:33,724 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 450, loss[loss=0.06591, simple_loss=0.09432, pruned_loss=0.01246, audio_tagging_loss=0.006289, over 15044.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09095, pruned_loss=0.01287, audio_tagging_loss=0.009559, over 2723781.80 frames. ], batch size: 55, lr: 1.75e-03, grad_scale: 32.0
2023-11-25 00:00:00,318 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 7.663e+01 8.552e+01 9.288e+01 9.905e+01 1.638e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-25 00:00:10,657 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=3049220.0, ans=0.125
2023-11-25 00:00:10,780 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3049220.0, ans=0.1
2023-11-25 00:00:30,632 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457400
2023-11-25 00:00:36,782 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 500, loss[loss=0.07534, simple_loss=0.101, pruned_loss=0.01564, audio_tagging_loss=0.009207, over 13470.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09026, pruned_loss=0.01287, audio_tagging_loss=0.00941, over 2800505.92 frames. ], batch size: 55, lr: 1.74e-03, grad_scale: 32.0
2023-11-25 00:00:40,609 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=3049353.3333333335, ans=0.07
2023-11-25 00:00:49,371 INFO [scaling.py:1118] (2/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-25 00:01:10,615 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3049486.6666666665, ans=0.0
2023-11-25 00:01:10,946 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.17 vs. limit=15.0
2023-11-25 00:01:14,076 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.23 vs. limit=15.0
2023-11-25 00:01:16,284 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0
2023-11-25 00:01:32,559 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457450
2023-11-25 00:01:36,828 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=3049620.0, ans=0.0
2023-11-25 00:01:39,086 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 550, loss[loss=0.05772, simple_loss=0.07379, pruned_loss=0.008517, audio_tagging_loss=0.01231, over 16123.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.08962, pruned_loss=0.01283, audio_tagging_loss=0.00941, over 2854282.35 frames. ], batch size: 60, lr: 1.74e-03, grad_scale: 32.0
2023-11-25 00:01:44,747 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3049686.6666666665, ans=0.0
2023-11-25 00:01:59,987 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=3049753.3333333335, ans=0.1
2023-11-25 00:02:05,698 INFO [optim.py:476] (2/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.616e+01 9.440e+01 1.005e+02 1.281e+02, threshold=1.888e+02, percent-clipped=0.0
2023-11-25 00:02:08,278 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3049820.0, ans=0.0
2023-11-25 00:02:35,470 INFO [model.py:792] (2/4) Freeze_encoder: False; Current batch idx: 457500
2023-11-25 00:02:35,732 INFO [scaling.py:213] (2/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3049953.3333333335, ans=0.0
2023-11-25 00:02:39,491 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.18 vs. limit=15.0
2023-11-25 00:02:41,287 INFO [train_asr.py:1221] (2/4) Epoch 39, batch 600, loss[loss=0.05479, simple_loss=0.07011, pruned_loss=0.009664, audio_tagging_loss=0.01007, over 15105.00 frames. ], tot_loss[loss=0.06622, simple_loss=0.08868, pruned_loss=0.01257, audio_tagging_loss=0.009304, over 2900703.91 frames. ], batch size: 58, lr: 1.74e-03, grad_scale: 32.0
2023-11-25 00:02:43,025 INFO [scaling.py:1022] (2/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.93 vs. limit=22.5
2023-11-25 00:02:46,463 INFO [checkpoint.py:75] (2/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/bad-model-2.pt